Get Some Rest With the Tungsten API, Part 2

Introduction

In our last blog post on this topic we covered the basics of the new REST API available with Tungsten version 7.0.0.

In this post we will dive into the details of the API, including the payloads and advanced functionality.

The Brief

To begin with, we will cover the concept of the “payload.” Payloads are data structures in JSON format. The Tungsten API will accept a payload for some requests and reply with a payload in response to certain requests.

All payloads are documented:

• Data Structures
• Proxy Developer Docs
• Manager Developer Docs
• Replicator Developer Docs

For the bulk of this blog post, we are going to use the tapi wrapper command to display the power of the API’s advanced features, in addition to how and when to use payloads.

Prim and Proper Payloads

Tungsten API payloads all follow a similar structure:

shell> /usr/bin/curl -s 'http://127.0.0.1:8090/api/v2/ping'
{
  "payloadType": "PingPayload",
  "payloadVersion": "1",
  "payload": {
    "message": "Ping test",
    "date": "Thu Apr 28 17:57:23 UTC 2022",
    "hostName": "db4-demo.continuent.com",
    "pid": 20205,
    "jvmUptime": 7406653
  }
}

Each payload will contain three main keys:

• payloadType - a string to describe the payload contents
• payloadVersion - the version of the payload structure
• payload - the actual data as a hash, with the keys varying based on payloadType

Task Payloads: Tame Long-Running Jobs

Step 1: Obtain the task payload by executing an API call that returns a task payload:

Tungsten API Tasks allow certain API calls to return a Task Payload, which gives information about a long-running job (or task) initiated by that API call:

shell> /usr/bin/curl -s 'http://127.0.0.1:8090/api/v2/manager/control/service/east/recover’
~OR~
shell> tapi -M -r recover
{
   "taskId" : "eed36609-9f0b-45de-868f-5de518e92ff8",
   "operation" : "RecoverTask",
   "state" : "in_progress"
}

Each task payload will contain three main keys:

• taskId - a unique string identifier for the task
• operation - the description of the task in progress
• state - what is the current state of the task in terms of progress

Step 2: Get information about a specific task

To obtain information about a specific task, provide the taskId from the task payload in Step 1:

shell> /usr/bin/curl -s 'http://127.0.0.1:8090/api/v2/manager/task/ed36609-9f0b-45de-868f-5de518e92ff8’
~OR~
shell> tapi -M -r task eed36609-9f0b-45de-868f-5de518e92ff8
{
   "payloadType" : "TaskPayload",
   "payload" : {
      "taskResult" : "NO RECOVERY REQUIRED FOR PHYSICAL DATA SERVICE 'east'",
      "taskId" : "eed36609-9f0b-45de-868f-5de518e92ff8",
      "operation" : "RecoverTask",
      "state" : "complete"
   },
   "payloadVersion" : "1"
}

This new TaskPayload will contain the same keys as previously, plus possibly more, including the result of the job:

• taskResult - the description of the result of the task

Step 3: Get information about all running tasks

If you do not have a taskId, but still want to know about jobs that are either running or complete, just call the tasks endpoint to get a TasksPayload (note the plural Tasks):

shell> /usr/bin/curl -s 'http://127.0.0.1:8090/api/v2/manager/tasks’
~OR~
shell> tapi -M -r tasks
{
  "payloadType": "TasksPayload",
  "payloadVersion": "1",
  "payload": [
    {
      "taskId": "91829d7f-5b91-4f2e-8356-ca1cc55185b8",
      "state": "complete",
      "taskResult": "NO RECOVERY REQUIRED FOR PHYSICAL DATA SERVICE 'east'",
      "operation": "RecoverTask"
    },
    {
      "taskId": "db9e1d84-b94a-43d3-9172-e48ef19d3936",
      "state": "complete",
      "taskResult": "NO RECOVERY REQUIRED FOR PHYSICAL DATA SERVICE 'east'",
      "operation": "RecoverTask"
    }
  ]
}

Each TasksPayload will contain an object per job in the actual payload, with the same metadata as a TaskPayload.

Data Payloads: Feed the Beast

There may be API calls which need input data to function. You may pass the data into the API call as a JSON-formatted string payload.

In this example, we provide a simple payload via the tapi command to enable or disable THL encryption in the Replicator:
(Please note that the Replicator must be OFFLINE to change the THL Encryption setting using setEncryption)

shell> tapi -R --run getEncryption 
{
   "value" : false
}

shell> tapi -R --run setEncryption --payload '{ "value" : "true" }'
{
   "value" : "THL encryption is now turned on"
}

shell> tapi -R --run getEncryption 

   "value" : true
}

shell> tapi -R --run setEncryption --payload '{ "value" : "false" }'
{
   "value" : "THL encryption is now turned off"
}

shell> tapi -R --run getEncryption 
{
   "value" : false
}

Take a look at the actual payload - it is surrounded by single-quotes, and uses double-quotes around the keys and values, with colon separators.

shell> tapi -v -R --run setEncryption --false
shell> tapi -v -R --run setEncryption --true

Payload Types: The Complete List

There are many payload types defined. Each component has a unique set of payload types with specific requirements. You may find them here (search for payload in your browser to locate them easily):

• Connector Payloads: https://docs.continuent.com/apiv2docs/proxy/#syntax_json
• Manager Payloads: https://docs.continuent.com/apiv2docs/manager/#syntax_json
• Replicator Payloads: https://docs.continuent.com/apiv2docs/replicator/#syntax_json

Payloads: A Deep Dive - Replicator Offline and Offline-Deferred

What’s Available?

First, let’s see what API calls are available for the Replicator to go OFFLINE:

shell> tapi -R -L -F offline
~OR~
shell> tapi --replicator --listapi --filter offline
Mode         Unique Key           Type Payload                  Reqd API Call Path
replicator   offline              POST                          N    /replicator/offline
replicator   offlineDeferred      POST OfflineDeferredPayload   Y    /replicator/offline
replicator   serviceOffline       POST                          N    /replicator/service/%s/offline
replicator   serviceOfflineDeferred POST OfflineDeferredPayload   Y    /replicator/service/%s/offline

There are four API calls for the Replicator to go OFFLINE, two of which specify a service name, and two which are deferred and need a payload:

• offline - mimics `trepctl offline`, callpath /replicator/offline, no payload
• offlineDeferred - mimics `trepctl offline-deferred`, callpath /replicator/offline, uses payload OfflineDeferredPayload
• serviceOffline - mimics `trepctl -service {SERVICE} offline`, callpath /replicator/service/SERVICE/offline, no payload
• serviceOfflineDeferred - mimics `trepctl -service {SERVICE} offline-deferred`, callpath /replicator/service/SERVICE/offline, uses payload OfflineDeferredPayload

The Simplest Example

In the simplest use-case there is no payload, we just tell the replicator to go offline right away using the offline unique run key to tapi, and we get a taskId back:

shell> tapi -R —-run offline
{
   "taskId" : "1485a683-0e6a-4547-8bcd-1dc7fda60b50",
   "operation" : "OfflineTask",
   "state" : "in_progress"
}

shell> tapi -R -r tasks
[
   {
      "requestId" : "c21a172f-b68d-4141-a599-44c2a90ed7f3",
      "taskId" : "1485a683-0e6a-4547-8bcd-1dc7fda60b50",
      "operation" : "OfflineTask",
      "state" : "complete"
   }
]

Going Offline - Deferred

What if we wanted to defer the OFFLINE like trepctl offline-deferred?

Here is the syntax for the CLI:

trepctl offline-deferred [-at-seqno seqno] [-at-event event] [-at-heartbeat [name]] [-at-time YYYY-MM-DD_hh:mm:ss]

Finally, the power of the payload comes into play.

When using the API to take the Replicator offline, there is the option of adding a payload to the call to mimic the `trepctl offline-deferred` command.

Let’s examine an example payload, the OfflineDeferredPayload.

For each type of defer, there is a value needed. The payload is where to specify the defer type and value. For this example, the payload syntax is:

--payload ’{ "type" : "deferred", "deferredType" : "TYPE", "deferredValue" : "VALUE" }’

Where:

• TYPE is one of: at_seqno, at_event, at_heartbeat or at_time
• VALUE needs to match the TYPE, so:
  • If TYPE is at_seqno, then VALUE must be an integer sequence number (seqno)
  • if TYPE if at_event, then VALUE must be a MySQL event ID
  • if TYPE if at_heartbeat, then VALUE must be a string heartbeat name
  • if TYPE if at_time, then VALUE must be a timestamp (YYYY-MM-DD_hh:mm:ss)

The Nitty Gritty Example

OK, so to put it all together, let’s go OFFLINE at seqno 10 (remember maintenance mode to STAY offline):

tapi -R -r offlineDeferred --payload ’{ "type" : "deferred", "deferredType" : "at_seqno", "deferredValue" : "10" }’
{
   "input" : {
      "payloadType" : "OfflineDeferredPayload",
      "payload" : {
         "deferredType" : "at_seqno",
         "deferredValue" : "10",
         "type" : "deferred"
      },
      "payloadVersion" : "1"
   },
   "taskId" : "676e02d5-4802-4da8-bf15-94734be9668b",
   "operation" : "OfflineTask",
   "state" : "in_progress"
}

shell> tapi -R -r tasks
[
   {
      "requestId" : "f0133ba0-327e-4052-8e4d-fd9ab9650274",
      "input" : {
         "payloadType" : "OfflineDeferredPayload",
         "payload" : {
            "deferredType" : "at_seqno",
            "deferredValue" : "10",
            "type" : "deferred"
         },
         "payloadVersion" : "1"
      },
      "taskId" : "676e02d5-4802-4da8-bf15-94734be9668b",
      "operation" : "OfflineTask",
      "state" : "complete"
   }
]

Note that just because the task is complete, the Replicator may still be ONLINE! Why? The task of telling the replicator to go offline at seqno 10 has completed, and the Replicator is just waiting for the correct seqno to act, for example:

shell> trepctl status | egrep -i 'lastseqno|state'
appliedLastSeqno       : 9
state                  : ONLINE
timeInStateSeconds     : 725.97

shell> echo 'cluster heartbeat' | cctrl
Tungsten Clustering 7.0.1 build 20
east: session established, encryption=true, authentication=true
jgroups: encrypted
[LOGICAL] /east > cluster heartbeat
HEARTBEAT 'DEFAULT' INSERTED

[LOGICAL] /east > 
Exiting...

shell> trepctl status | egrep -i 'lastseqno|state'
appliedLastSeqno       : -1
state                  : OFFLINE:NORMAL
timeInStateSeconds     : 2.66

When Using API Clients Other Than tapi

{
   "payloadType" : "OfflineDeferredPayload",
   "payload" : {
      "type" : "deferred",
      "deferredType" : "TYPE",
      "deferredValue" : "VALUE"
   },
   "payloadVersion" : "1"
}

Payload Documentation

Remember that all payloads are fully defined in our online documentation. You may find them here (search for payload in your browser to locate them easily):

• Connector Payloads: https://docs.continuent.com/apiv2docs/proxy/#syntax_json
• Manager Payloads: https://docs.continuent.com/apiv2docs/manager/#syntax_json
• Replicator Payloads: https://docs.continuent.com/apiv2docs/replicator/#syntax_json

Bonus Command

As of version 7.0.1 the tapi command can display the payloads for each call. Not all payloads are implemented yet, so YMMV ;-}

shell> tapi --listpayloads
~OR~
shell> tapi --LP
Mode         Unique Key           Type Payload                  Reqd  Data
…
replicator   flush                POST FlushPayload             N    not defined yet
replicator   offlineDeferred      POST OfflineDeferredPayload   Y    --type {at_seqno|at_event|at_heartbeat|at_time} --value {value}  ~OR~  --payload '{ "type" : "deferred", "deferredType" : "at_heartbeat", "deferredValue" : "offline_hb" }'
…

shell> tapi --listpayloads -F offlineDeferred
Mode         Unique Key           Type Payload                  Reqd  Data
replicator   offlineDeferred      POST OfflineDeferredPayload   Y    --type {at_seqno|at_event|at_heartbeat|at_time} --value {value}  ~OR~  --payload '{ "type" : "deferred", "deferredType" : "at_heartbeat", "deferredValue" : "offline_hb" }'
…

Wrap-Up

In this post we explored the details of the Tungsten RESTful API used with Tungsten Clustering version 7.0+, including payloads and advanced functionality. We covered response payloads, tasks, and payloads specific to a specific API call.

The API is a vast pool of capability, and we have barely scratched the surface of what can be accomplished.

Smooth sailing!