Introduction
Transaction History Log (THL) files are the core of Tungsten Replication, containing the actual MySQL write events that are replicated to the replicas.
These files can take up considerable amounts of disk space, making them of interest for housekeeping operations to limit the consumption and ultimately, the cost.
This blog post will walk you through THL management, along with the new command `tpm purge-thl` which helps automate the process when THL needs to be removed prior to the automatic rotation window.
The Brief
THL files are created on the write primary node by the Tungsten Replicator extractor reading the MySQL binary logs, then converting the extracted write events into the THL normalized format for consumption by the downstream Tungsten Replicator appliers.
By default, the log files are kept for 7 days.
If the disk space gets used up, existing THL files cannot be updated, and new THL files cannot be created, which would block replication.
Recognizing this fact, Tungsten provides easy ways to manage the growth of the THL files and associated disk space.
The Basics: Automated Log Retention
The first line of defense is the automatic deletion of THL after a set period of time, called the log retention period.
Configuring the THL Log Retention Period
To drop log files after a certain time period, set the tpm option `--thl-log-retention` to a time interval, defined as:
<integer>{d|h|m|s}
where the letters stand for Days, Hours, Minutes, or Seconds respectively.
For example, to rotate the THL files after five days instead of seven, use:
--thl-log-retention=5d
and for twelve hours, use:
--thl-log-retention=12h
Key Facts about THL Log Retention
Longer times retain the logs for a greater amount of time, increasing disk space usage while allowing access to the THL information for longer. Shorter logs reduce disk space usage while reducing the amount of log data available.
Please note that old THL files are only deleted when new data is written to the current files. If there has been no THL activity, the log files remain until new THL information is written.
How Long Should THL Logs Be Retained?
It is advised to keep the THL files for at least as long as you keep your MySQL binary log files.
If the THL log retention time is set to lower than the MySQL binary log expiry time, running `tpm install/update` will result in a warning like:
THL retention (5d) is set to less than the binary log expiry policy (10 days) (MySQLTHLBinlogsRetentionCheck)
Advanced: Manual THL Log File Management
In the event that disk space is filling up too quickly, it has always been possible to recover capacity by manually deleting THL files that are no longer needed.
“No longer needed” means the following things:
- on a replica: all of the events in that THL file have been applied to the target database;
- on the write primary: all of the events in that THL file have been applied to ALL of the downstream replica databases.
Prior to Version 7: The Long Way
Prior to version 7, a DBA would need to run the `trepctl status` command on each node (or the multi_trepctl command), and manually compare the applied sequence numbers to obtain the lowest one, then check in the THL files themselves for the exact file that contains the seqno, and then finally find the previous THL file in sequence just to be safe, then executing the thhl purge command with the appropriate parameters.
New Tool as of Version 7: tpm purge-thl
As of Tungsten Clustering v7.0.0, you may now recover disk space more easily with some help from the `tpm purge-thl` command.
The idea is to run the command on a replica, then execute the returned command on all nodes as the proper tungsten OS user. This will delete all THL files with events completely applied on all replicas.
To begin, simply execute the command on a replica:
shell> tpm purge-thl
/opt/continuent/tungsten/tungsten-replicator/bin/thl -service emea purge -high 71202
Please avoid running the tool on the write primary! If you run the command on a write primary, you will see a warning like this:
shell> tpm purge-thl
>>> WARNING: This script is running on a Primary node!
Be extra careful to ensure ALL downstream Replicas
have transferred and applied ALL needed THL
BEFORE running this purge-thl command.
/opt/continuent/tungsten/tungsten-replicator/bin/thl -service emea purge -high 71202
Wrap-Up
In this post we explored the details of THL log retention, including the new tool, `tpm purge-thl`, included with Tungsten version 7.0+.
For more information, please see our online documentation:
- https://docs.continuent.com/tungsten-clustering-7.0/files-install-thl.html#files-install-thl-retention
- https://docs.continuent.com/tungsten-clustering-7.0/cmdline-tools-tpm-configoptions-t.html#cmdline-tools-tpm-configoptions-thl-log-retention
- https://docs.continuent.com/tungsten-clustering-7.0/cmdline-tools-tpm-commands-purge-thl.html
Smooth sailing!
Comments
Add new comment