MySQL / MariaDB Clustering: Seamless Primary Switch with Tungsten Clustering

Switch: A Single Command

One nice thing about Tungsten Clustering is that you need just a single command to move the primary role to another host in your cluster. Using the cluster control tool (cctrl) via the command-line, you just need to type "switch" to get the (complex) job done!

 /alpha > switch
SELECTED SLAVE: host2@alpha
PURGE REMAINING ACTIVE SESSIONS ON CURRENT MASTER 'host1@alpha'
PURGED A TOTAL OF 0 ACTIVE SESSIONS ON MASTER 'host1@alpha'
FLUSH TRANSACTIONS ON CURRENT MASTER 'host1@alpha'
PUT THE NEW MASTER 'host2@alpha' ONLINE
PUT THE PRIOR MASTER 'host1@alpha' ONLINE AS A SLAVE
RECONFIGURING SLAVE 'host3@alpha' TO POINT TO NEW MASTER 'host2@alpha'
SWITCH TO 'host2@alpha' WAS SUCCESSFUL

What's even nicer is that in most cases, your application won't even notice that a switch occurred. All you will get is a little lag in connectivity, while the new host is being reconfigured.

The magic resides in the fact that the Connector is able to "pause" new connections during the switch operation:

• First, the Connector will wait for ongoing connections to disconnect. After this configurable delay, connections are forcibly closed (but they will generally try to reconnect by themselves).
• At the same time, any new connection request will be paused until the new primary is ready to accept traffic. Here comes the little lag in your application!
• As soon as the new primary is ready (usually within seconds), traffic continues, application's happy, so are your customers.

From the Connector Point of View

Let’s have a closer look at the switch sequence in the connector logs (these have been reduced manually in order to ease reading in this article):

- Maintenance mode set to true
- Updating dataSource: host1@alpha(master:ONLINE) => host1@alpha(master:OFFLINE)
- Data service alpha - Preparing to close connections on data source: host1

This is where the connector pauses new connection requests, which will start “hanging”.

- Updating dataSource: host2@alpha(slave:ONLINE) => host2@alpha(slave:OFFLINE)
- Data service alpha - Preparing to close connections on data source: host2
- Updating dataSource: host1@alpha(master:OFFLINE) => host1@alpha(slave:OFFLINE)
- Updating dataSource: host2@alpha(slave:OFFLINE) => host2@alpha(master:OFFLINE)
- Updating dataSource: host2@alpha(master:OFFLINE) => host2@alpha(master:ONLINE)

The Connector sees a new online primary, it is now able to resume new connection requests.

- Updating dataSource: host1@alpha(slave:OFFLINE) => host1@alpha(slave:ONLINE)
- Maintenance mode set to false

The operation is now finished, and read-only connections can be made to the previous primary.

Of course, if your application was connected with read-only access to replicas other than the new primary, they will not even be affected by the operation!