Background
One tedious but critical task for admins is provisioning databases, especially after a failure of a node or nodes. It’s not as simple as restoring a backup, because the backup then needs to be brought up to date, and the datasource then added back into the cluster. In addition to being time-consuming, a mistake at any step will corrupt the newly provisioned database and the admin has to start over from scratch.
To automate this process, the script tungsten_provision_slave was introduced in the early days of Tungsten Cluster. By using this tool, an admin could simply choose a healthy source database, and the script would perform the backup of the database, restore it onto the target, set the replication position so the target database would catch up, and then add the datasource back into the cluster as a replica. Using either mysqldump or xtrabackup, tungsten_provision_slave has reprovisioned countless replicas unattended by taking a backup of the source database, restoring it onto the target host, fixing all database permissions on the filesystem, removing outdated THL (Transaction History Log), and finally recovering the datasource. When using xtrabackup, the source database remains online for the duration of the script.
And Now - tprovision!
We decided the faithful tungsten_provision_slave script needed an update and it should satisfy the following requirements:
- Easy to maintain and update
- Easy to extend to new backup methods
- Support the latest Tungsten Cluster topologies, like Composite Active/Active (CAA)
And thus tprovision was written, from scratch, in Perl. Perl code is easy to maintain and compatible between versions and OS’s. The code now makes it easier to add new backup methods too. We’ve also retired the outdated name of the old script (Note that Continuent as a company has been leading the way to retire outdated names and terms in the MySQL Community).
tprovision is actually two scripts: tprovision, which is simply a shell wrapper, and tps.pl, which is the actual perl script that performs the provision. As of Tungsten Cluster 7.0.1, both mysqldump and xtrabackup are supported for the backup of the source database. tprovision by default will use mysqldump, however you can use xtrabackup by specifying -m xtrabackup on the command line.
tprovision is meant to be fast as well. Whether using mysqldump or xtrabackup, the backups from the source are streamed to the target – so there is no intermediate backup that has to be written to disk and then copied to the target. In addition to being faster, this also saves on disk space as well. When using xtrabackup, you also get the option to use xtrabackup’s multiple backup threads (using the -t parameter).
One extremely helpful feature of tprovision is being able to provision primary datasources. With this feature, if an entire site in a Composite Active/Active topology is corrupted, tprovision can provision a brand new primary from a replica (or primary) in another site. From there you can then provision local replicas. Use the -c (--create-primary) parameter to create a primary.
Requirements
Most of the requirements for tprovision follow the prerequisites of Tungsten Cluster. Depending on the backup method, you’ll need to be sure sudoers is configured as specified in the prerequisites section, and xtrabackup will require passwordless ssh to the source server for the tungsten user. Using screen is not a requirement, however for provisioning of large databases it is highly recommended in case the terminal session is disconnected.
Future Enhancements
Soon tprovision will support rsync as a backup method!
Using rsync to provision a database can be fast, especially for large databases. It is based on the idea that a failed or corrupted database is still mostly the same as the source database at the file level, and thus we just need to synchronize the differences from the source database. Of course, when copying a database at the file level, we need to quiesce the database at some point to get a consistent copy, which I will explain in an upcoming blog about using rsync to copy live databases.
Also planned is using an existing database backup as the source database, instead of a database in the cluster. By using an existing backup, we can avoid placing load on the source host. We can also extend this to provisioning of entire clusters from a single backup.
Closing Remarks
tprovision is the latest method of provisioning hosts. It was born in version 6.0.0, enhanced in 6.1.0, and finally given its new name in version 7.0.0. There is also additional interesting reading material on this topic:
- Documentation: https://docs.continuent.com/tungsten-clustering-7.0/cmdline-tools-tprovision.html
- Using Rsync to Provision a Database: https://www.continuent.com/resources/blog/how-use-rsync-provision-tungsten-cluster-node
Be sure to look out for a follow up blog on using rysnc to provision a database, and a follow up to this blog about tprovision when rsync is added as a backup method!
Comments
Add new comment