Introduction
In this blog post we explore the new command `tungsten_merge_logs` which merges log files under a directory tree into a single file.
There are many moving parts to a cluster and they are spread over multiple nodes (usually three).
When there is an issue, the logs are the key resource to find out what is going on.
The best practice is to gather the log files into one place and then read through them all.
This can be difficult with many files on multiple nodes.
For this reason, the `tungsten_merge_logs` tool was created.
The tungsten_merge_logs command is designed to aid troubleshooting by consolidating the various log files into one place (merged.log), ordered by time.
Log File Love
The three main log files are:
| Component | Log File | Path |
|---|---|---|
| Connector | connector.log |
$CONTINUENT_ROOT/tungsten/tungsten-connector/log |
| Manager | tmsvc.log |
$CONTINUENT_ROOT/tungsten/tungsten-manager/log |
| Replicator | trepsvc.log |
$CONTINUENT_ROOT/tungsten/tungsten-replicator/log |
There is also a shortcut link for each located inside $CONTINUENT_ROOT/service_logs:
tungsten@db1-demo:/opt/continuent/service_logs # ls -l
total 36
lrwxrwxrwx 1 tungsten tungsten 66 Oct 22 15:32 connector-user.log -> /opt/continuent/tungsten/tungsten-connector/log/connector-user.log
lrwxrwxrwx 1 tungsten tungsten 61 Oct 22 15:32 connector.log -> /opt/continuent/tungsten/tungsten-connector/log/connector.log
lrwxrwxrwx 1 tungsten tungsten 62 Oct 22 15:32 manager-user.log -> /opt/continuent/tungsten/tungsten-manager/log/manager-user.log
-rw-rw-r-- 1 tungsten tungsten 1621 Nov 22 20:46 mysqld_exporter.log
lrwxrwxrwx 1 tungsten tungsten 62 Oct 22 15:32 mysqldump.log -> /opt/continuent/tungsten/tungsten-replicator/log/mysqldump.log
-rw-rw-r-- 1 tungsten tungsten 3237 Nov 22 19:49 node_exporter.log
lrwxrwxrwx 1 tungsten tungsten 68 Oct 22 15:32 replicator-user.log -> /opt/continuent/tungsten/tungsten-replicator/log/replicator-user.log
lrwxrwxrwx 1 tungsten tungsten 55 Oct 22 15:32 tmsvc.log -> /opt/continuent/tungsten/tungsten-manager/log/tmsvc.log
lrwxrwxrwx 1 tungsten tungsten 60 Oct 22 15:32 trepsvc.log -> /opt/continuent/tungsten/tungsten-replicator/log/trepsvc.log
lrwxrwxrwx 1 tungsten tungsten 63 Oct 22 15:32 xtrabackup.log -> /opt/continuent/tungsten/tungsten-replicator/log/xtrabackup.log
Gathering the Logs
Getting the logs is a simple procedure - just one command will do it for you: `tpm diag –all`. For example, get the log from all six nodes:
tungsten@db1-demo:/home/tungsten/diags # tpm diag --all
db1-demo.continuent.com|Collecting full cluster diagnostics
Note: if ssh access to any of the cluster hosts is denied, use "--local" or "--hosts=<host1,host2,...>"
db1-demo.continuent.com|Collecting diag information for db1-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db2-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db3-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db4-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db5-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db6-demo.continuent.com
db1-demo.continuent.com|Diagnostic information written to /home/tungsten/diags/tungsten-diag-2021-11-24-17-23-31.tgz
In our example, a diagnostic tarball was created called `/home/tungsten/diags/tungsten-diag-2021-11-24-17-23-31.tgz`, which you can extract to gain access to the log files from every node. Let extract the tarball and have a look:
shell$ tar xvzf tungsten-diag-2021-11-24-17-23-31.tgz
shell$ cd tungsten-diag-2021-11-24-17-23-31
shell$ ls -l
total 24
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:32 db1-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db2-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db3-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db4-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db5-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db6-demo.continuent.com
As you can see, we have six sub-directories, one per database cluster node.
Each directory will have it’s own set of log files, which of course you can view individually, i.e.:
shell$ vi `find . -name "*svc.log"`
12 files to edit
Merging the Logs
Now, as nice as it is to have a vi session with 12 files to see, it would be much better to merge all the log files into a single file sorted by host and time so that it is easier to understand what is happening at any given point in time.
Doing this is very easy - just run the new `tungsten_merge_logs` command (available in v6.1.16 NYR and v7.0.0 Beta). For example, let’s gather all 12 log files into one single `merged.log` in the current directory:
shell$ pwd
/home/tungsten/diags/tungsten-diag-2021-11-24-17-23-31
shell$ tungsten_merge_logs -L 1
New merged log file ./merged.log created!
shell$ vi merged.log
Sky’s the Limit
You may have noticed that there was an argument provided: `-L 1`. This tells the tungsten_merge_logs script to only gather the latest log from eact component. Tungsten logs will be rotated, so that a sequence number will be appended to the file, like this:
shell$ find . -name "trepsvc.log*" -ls| grep db2
552970 10244 -rw-rw-r-- 1 tungsten tungsten 10485852 Oct 26 14:19 ./db2-demo.continuent.com/logs/trepsvc.log.9
563957 10248 -rw-rw-r-- 1 tungsten tungsten 10485944 Oct 26 14:26 ./db2-demo.continuent.com/logs/trepsvc.log.8
563960 10244 -rw-rw-r-- 1 tungsten tungsten 10485893 Oct 26 14:33 ./db2-demo.continuent.com/logs/trepsvc.log.7
552976 10244 -rw-rw-r-- 1 tungsten tungsten 10485893 Oct 26 14:39 ./db2-demo.continuent.com/logs/trepsvc.log.6
563956 10248 -rw-rw-r-- 1 tungsten tungsten 10485893 Oct 26 14:46 ./db2-demo.continuent.com/logs/trepsvc.log.5
552962 10244 -rw-rw-r-- 1 tungsten tungsten 10485893 Oct 26 14:53 ./db2-demo.continuent.com/logs/trepsvc.log.4
563961 10244 -rw-rw-r-- 1 tungsten tungsten 10485893 Oct 26 14:59 ./db2-demo.continuent.com/logs/trepsvc.log.3
562736 10244 -rw-rw-r-- 1 tungsten tungsten 10485893 Oct 26 15:24 ./db2-demo.continuent.com/logs/trepsvc.log.2
563952 10244 -rw-rw-r-- 1 tungsten tungsten 10485872 Oct 26 20:19 ./db2-demo.continuent.com/logs/trepsvc.log.1
552978 616 -rw-r----- 1 tungsten tungsten 630202 Nov 3 16:46 ./db2-demo.continuent.com/logs/trepsvc.log
Without the `-L 1` argument, tungsten_merge_logs will gather all logs, including the rotated ones.
More Information
Below is the help text you get when you run `tungsten_merge_logs -h`:
shell$ tungsten_merge_logs -h
Usage for tungsten_merge_logs:
tungsten_merge_logs [options]
>>> Introduction <<<
The tungsten_merge_logs command is designed to aid troubleshooting by consolidating the various
log files into one place ordered by time.
>>> Getting Started <<<
With no options specified, the tungsten_merge_logs script will
gather all log files in the current directory and below.
For example:
cd
tpm diag --all
tar xvzf tungsten-diag-2021-11-15-16-37-33.tgz
cd tungsten-diag-2021-11-15-16-37-33
tungsten_merge_logs
would result in something like the following:
New merged log file ./merged.log created!
>>> Operating Rules <<<
All logs files are gathered by default.
If you specify any of --connector, --replicator or --manager,
then only the log files for the specifed components will be gathered.
Using multiple options will aggregate the logs from the specified components.
>>> Examples <<<
tungsten_merge_logs
tungsten_merge_logs -h
tungsten_merge_logs -R
tungsten_merge_logs -C
tungsten_merge_logs -M
tungsten_merge_logs -M -R -L 1
>>> Command-Line Options <<<
[-L|--log-limit] Specify the quantity of log files to gather (default: unlimited).
When the wrapper rotates log files, it appends a period and an integer to the end of the log file name, when .1 is the newest and .2 is older and .3 older than that, etc.
This parameter will gather the base log file plus limit minus one rotated files.
Examples:
- a loglimit of 1 means gather the base file only, i.e. trepsvc.log
- a loglimit of 2 means gather the base file and the first backup file trepsvc.log.1
- a loglimit of 3 means gather the base file and the first two backup files, trepsvc.log.1 and trepsvc.log.2
[-C|--connector] Specify the connector
[-M|--manager] Specify the manager
[-R|--replicator] Specify the replicator
[-x|--extension {string}] Specify the file extension (default: log) Do NOT include the period.
[-h|--help]
[-q|--quiet]
[-v|--verbose]
[-d|--debug]
Summary
In this blog post we explored the new command `tungsten_merge_logs`, which merges log files under a directory tree into a single file `merged.log`.
If you are a Continuent customer, please submit a Zendesk ticket if you have any questions; otherwise, feel free to reach out to learn more about Tungsten Clustering.
Comments
Add new comment