21.6.8 Implementing Failover with NDB Cluster Replication
In the event that the primary Cluster replication process fails, it is possible to switch over to the secondary replication channel. The following procedure describes the steps required to accomplish this.
Obtain the time of the most recent global checkpoint (GCP). That is, you need to determine the most recent epoch from the
ndb_apply_statustable on the replica cluster, which can be found using the following query:
mysqlR'> SELECT @latest:=MAX(epoch) -> FROM mysql.ndb_apply_status;
In a circular replication topology, with a source and a replica running on each host, when you are using
ndb_log_apply_status=1, NDB Cluster epochs are written in the replicas' binary logs. This means that the
ndb_apply_statustable contains information for the replica on this host as well as for any other host which acts as a replica of the replication source server running on this host.
In this case, you need to determine the latest epoch on this replica to the exclusion of any epochs from any other replicas in this replica's binary log that were not listed in the
IGNORE_SERVER_IDSoptions of the
CHANGE MASTER TOstatement used to set up this replica. The reason for excluding such epochs is that rows in the
mysql.ndb_apply_statustable whose server IDs have a match in the
IGNORE_SERVER_IDSlist from the
CHANGE MASTER TOstatement used to prepare this replicas's source are also considered to be from local servers, in addition to those having the replica's own server ID. You can retrieve this list as
Replicate_Ignore_Server_Idsfrom the output of
SHOW SLAVE STATUS. We assume that you have obtained this list and are substituting it for
ignore_server_idsin the query shown here, which like the previous version of the query, selects the greatest epoch into a variable named
mysqlR'> SELECT @latest:=MAX(epoch) -> FROM mysql.ndb_apply_status -> WHERE server_id NOT IN (ignore_server_ids);
In some cases, it may be simpler or more efficient (or both) to use a list of the server IDs to be included and
server_id INin the
WHEREcondition of the preceding query.
You can use the following query to obtain the needed records from the
ndb_binlog_indextable on the source:
mysqlS'> SELECT -> @file:=SUBSTRING_INDEX(next_file, '/', -1), -> @pos:=next_position -> FROM mysql.ndb_binlog_index -> WHERE epoch >= @latest -> ORDER BY epoch ASC LIMIT 1;
These are the records saved on the source since the failure of the primary replication channel. We have employed a user variable
@latesthere to represent the value obtained in Step 1. Of course, it is not possible for one mysqld instance to access user variables set on another server instance directly. These values must be “plugged in” to the second query manually or by an application.
Now it is possible to synchronize the secondary channel by running the following query on the secondary replica server:
mysqlR'> CHANGE MASTER TO -> MASTER_LOG_FILE='@file', -> [email protected];
Again we have employed user variables (in this case
@pos) to represent the values obtained in Step 2 and applied in Step 3; in practice these values must be inserted manually or using an application that can access both of the servers involved.Note
@fileis a string value such as
'/var/log/mysql/replication-source-bin.00001', and so must be quoted when used in SQL or application code. However, the value represented by
@posmust not be quoted. Although MySQL normally attempts to convert strings to numbers, this case is an exception.
You can now initiate replication on the secondary channel by issuing the appropriate statement on the secondary replica mysqld:
mysqlR'> START SLAVE;
Once the secondary replication channel is active, you can investigate the failure of the primary and effect repairs. The precise actions required to do this will depend upon the reasons for which the primary channel failed.
The secondary replication channel is to be started only if and when the primary replication channel has failed. Running multiple replication channels simultaneously can result in unwanted duplicate records being created on the replicas.
If the failure is limited to a single server, it should in theory be possible to replicate from
R', or from