188.8.131.52 Defining NDB Cluster Data Nodes
[ndbd default] sections are used to configure the behavior of the cluster's data nodes.
There are many parameters which control buffer sizes, pool sizes, timeouts, and so forth. The only mandatory parameter is either one of
HostName; this must be defined in the local
NoOfReplicas should be defined in the
[ndbd default] section, as it is common to all Cluster data nodes. It is not strictly necessary to set
NoOfReplicas, but it is good practice to set it explicitly.
Most data node parameters are set in the
[ndbd default] section. Only those parameters explicitly stated as being able to set local values are permitted to be changed in the
[ndbd] section. Where present,
ExecuteOnComputer must be defined in the local
[ndbd] section, and not in any other section of
config.ini. In other words, settings for these parameters are specific to one data node.
For those parameters affecting memory usage or buffer sizes, it is possible to use
G as a suffix to indicate units of 1024, 1024×1024, or 1024×1024×1024. (For example,
100K means 100 × 1024 = 102400.)
Parameter names and values are case-insensitive, unless used in a MySQL Server
my.ini file, in which case they are case-sensitive.
Information about configuration parameters specific to NDB Cluster Disk Data tables can be found later in this section (see Disk Data Configuration Parameters).
All of these parameters also apply to ndbmtd (the multithreaded version of ndbd). Three additional data node configuration parameters—
NoOfFragmentLogParts—apply to ndbmtd only; these have no effect when used with ndbd. For more information, see Multi-Threading Configuration Parameters (ndbmtd). See also Section 21.4.3, “ndbmtd — The NDB Cluster Data Node Daemon (Multi-Threaded)”.
Table 21.27 This table provides type and value information for the NodeId data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units unsigned Default [none] Range 1 - 48 Restart Type IS
A unique node ID is used as the node's address for all cluster internal messages. For data nodes, this is an integer in the range 1 to 48 inclusive. Each node in the cluster must have a unique identifier.
NodeIdis the only supported parameter name to use when identifying data nodes. (
Idwas removed in NDB 7.5.0.)
Table 21.28 This table provides type and value information for the ExecuteOnComputer data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units name Default [none] Range ... Restart Type S Deprecated NDB 7.5.0
This refers to the
Idset for one of the computers defined in a
This parameter is deprecated as of NDB 7.5.0, and is subject to removal in a future release. Use the
Table 21.29 This table provides type and value information for the HostName data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units name or IP address Default localhost Range ... Restart Type N
Specifying this parameter defines the hostname of the computer on which the data node is to reside. To specify a hostname other than
localhost, either this parameter or
Table 21.30 This table provides type and value information for the ServerPort data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units unsigned Default [none] Range 1 - 64K Restart Type S
Each node in the cluster uses a port to connect to other nodes. By default, this port is allocated dynamically in such a way as to ensure that no two nodes on the same host computer receive the same port number, so it should normally not be necessary to specify a value for this parameter.
However, if you need to be able to open specific ports in a firewall to permit communication between data nodes and API nodes (including SQL nodes), you can set this parameter to the number of the desired port in an
[ndbd]section or (if you need to do this for multiple data nodes) the
[ndbd default]section of the
config.inifile, and then open the port having that number for incoming connections from SQL nodes, API nodes, or both.
Setting this parameter to
IP_ADDR_ANYso that connections can be made from anywhere (for autogenerated connections). The default is
Table 21.31 This table provides type and value information for the NodeGroup data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units unsigned Default [none] Range 0 - 65536 Restart Type IS
This parameter can be used to assign a data node to a specific node group. It is read only when the cluster is started for the first time, and cannot be used to reassign a data node to a different node group online. It is generally not desirable to use this parameter in the
[ndbd default]section of the
config.inifile, and care must be taken not to assign nodes to node groups in such a way that an invalid numbers of nodes are assigned to any node groups.
NodeGroupparameter is chiefly intended for use in adding a new node group to a running NDB Cluster without having to perform a rolling restart. For this purpose, you should set it to 65536 (the maximum value). You are not required to set a
NodeGroupvalue for all cluster data nodes, only for those nodes which are to be started and added to the cluster as a new node group at a later time. For more information, see Section 184.108.40.206, “Adding NDB Cluster Data Nodes Online: Detailed Example”.
Table 21.32 This table provides type and value information for the LocationDomainId data node configuration parameter
Property Value Version (or later) NDB 7.6.4 Type or units integer Default 0 Range 0 - 16 Restart Type S Added NDB 7.6.4
Assigns a data node to a specific availability domain (also known as an availability zone) within a cloud. By informing
NDBwhich nodes are in which availability domains, performance can be improved in a cloud environment in the following ways:
If requested data is not found on the same node, reads can be directed to another node in the same availability domain.
Communication between nodes in different availability domains are guaranteed to use
NDBtransporters' WAN support without any further manual intervention.
The transporter's group number can be based on which availability domain is used, such that also SQL and other API nodes communicate with local data nodes in the same availability domain whenever possible.
The arbitrator can be selected from an availability domain in which no data nodes are present, or, if no such availability domain can be found, from a third availability domain.
LocationDomainIdtakes an integer value between 0 and 16 inclusive, with 0 being the default; using 0 is the same as leaving the parameter unset.
Table 21.33 This table provides type and value information for the NoOfReplicas data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units integer Default 2 Range 1 - 2 Restart Type IS
This global parameter can be set only in the
[ndbd default]section, and defines the number of replicas for each table stored in the cluster. This parameter also specifies the size of node groups. A node group is a set of nodes all storing the same information.
Node groups are formed implicitly. The first node group is formed by the set of data nodes with the lowest node IDs, the next node group by the set of the next lowest node identities, and so on. By way of example, assume that we have 4 data nodes and that
NoOfReplicasis set to 2. The four data nodes have node IDs 2, 3, 4 and 5. Then the first node group is formed from nodes 2 and 3, and the second node group by nodes 4 and 5. It is important to configure the cluster in such a manner that nodes in the same node groups are not placed on the same computer because a single hardware failure would cause the entire cluster to fail.
If no node IDs are provided, the order of the data nodes will be the determining factor for the node group. Whether or not explicit assignments are made, they can be viewed in the output of the management client's
The default and recommended maximum value for
NoOfReplicasis 2. This is the recommended value for most production environments.Important
While it is theoretically possible for the value of this parameter to be 3 or 4, NDB Cluster 7.5 and NDB Cluster 7.6 do not support setting
NoOfReplicasto a value greater than 2 in production.Warning
NoOfReplicasto 1 means that there is only a single copy of all Cluster data; in this case, the loss of a single data node causes the cluster to fail because there are no additional copies of the data stored by that node.
The value for this parameter must divide evenly into the number of data nodes in the cluster. For example, if there are two data nodes, then
NoOfReplicasmust be equal to either 1 or 2, since 2/3 and 2/4 both yield fractional values; if there are four data nodes, then
NoOfReplicasmust be equal to 1, 2, or 4.
Table 21.34 This table provides type and value information for the DataDir data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units path Default . Range ... Restart Type IN
This parameter specifies the directory where trace files, log files, pid files and error logs are placed.
The default is the data node process working directory.
Table 21.35 This table provides type and value information for the FileSystemPath data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units path Default DataDir Range ... Restart Type IN
This parameter specifies the directory where all files created for metadata, REDO logs, UNDO logs (for Disk Data tables), and data files are placed. The default is the directory specified by
This directory must exist before the ndbd process is initiated.
The recommended directory hierarchy for NDB Cluster includes
/var/lib/mysql-cluster, under which a directory for the node's file system is created. The name of this subdirectory contains the node ID. For example, if the node ID is 2, this subdirectory is named
Table 21.36 This table provides type and value information for the BackupDataDir data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units path Default FileSystemPath Range ... Restart Type IN
This parameter specifies the directory in which backups are placed.Important
The string '
/BACKUP' is always appended to this value. For example, if you set the value of
/var/lib/cluster-data, then all backups are stored under
/var/lib/cluster-data/BACKUP. This also means that the effective default backup location is the directory named
BACKUPunder the location specified by the
[ndbd] parameters specifying the size of memory segments used to store the actual records and their indexes. In setting values for these, it is important to understand how
IndexMemory are used, as they usually need to be updated to reflect actual usage by the cluster.
IndexMemory is deprecated in NDB 7.6, and subject to removal in a future version of NDB Cluster. See the descriptions that follow for further information.
Table 21.37 This table provides type and value information for the DataMemory data node configuration parameter
This parameter defines the amount of space (in bytes) available for storing database records. The entire amount specified by this value is allocated in memory, so it is extremely important that the machine has sufficient physical memory to accommodate it.
The memory allocated by
DataMemoryis used to store both the actual records and indexes. There is a 16-byte overhead on each record; an additional amount for each record is incurred because it is stored in a 32KB page with 128 byte page overhead (see below). There is also a small amount wasted per page due to the fact that each record is stored in only one page.
For variable-size table attributes, the data is stored on separate data pages, allocated from
DataMemory. Variable-length records use a fixed-size part with an extra overhead of 4 bytes to reference the variable-size part. The variable-size part has 2 bytes overhead plus 2 bytes per attribute.
The maximum record size is 14000 bytes.
In NDB 7.5 (and earlier), the memory space defined by
DataMemoryis also used to store ordered indexes, which use about 10 bytes per record. Each table row is represented in the ordered index. A common error among users is to assume that all indexes are stored in the memory allocated by
IndexMemory, but this is not the case: Only primary key and unique hash indexes use this memory; ordered indexes use the memory allocated by
DataMemory. However, creating a primary key or unique hash index also creates an ordered index on the same keys, unless you specify
USING HASHin the index creation statement. This can be verified by running ndb_desc -d
In NDB 7.6, resources assigned to
DataMemoryare used for storing all data and indexes; any memory configured as
IndexMemoryis automatically added to that used by
DataMemoryto form a common resource pool.
Currently, NDB Cluster can use a maximum of 512 MB for hash indexes per partition, which means in some cases it is possible to get Table is full errors in MySQL client applications even when ndb_mgm -e "ALL REPORT MEMORYUSAGE" shows significant free
DataMemory. This can also pose a problem with data node restarts on nodes that are heavily loaded with data.
In NDB 7.5.4 and later, you can control the number of partitions per local data manager for a given table by setting the
PARTITION_BALANCEto one of the values
FOR_RA_BY_LDM_X_4, for 1, 2, 3, or 4 partitions per LDM, respectively, when creating the table (see Section 220.127.116.11, “Setting NDB_TABLE Options”).Note
In previous versions of NDB Cluster it was possible to create extra partitions for NDB Cluster tables and thus have more memory available for hash indexes by using the
CREATE TABLE. While still supported for backward compatibility, using
MAX_ROWSfor this purpose is deprecated beginning with NDB 7.5.4, where you should use
You can also use the
MinFreePctconfiguration parameter to help avoid problems with node restarts.
The memory space allocated by
DataMemoryconsists of 32KB pages, which are allocated to table fragments. Each table is normally partitioned into the same number of fragments as there are data nodes in the cluster. Thus, for each node, there are the same number of fragments as are set in
Once a page has been allocated, it is currently not possible to return it to the pool of free pages, except by deleting the table. (This also means that
DataMemorypages, once allocated to a given table, cannot be used by other tables.) Performing a data node recovery also compresses the partition because all records are inserted into empty partitions from other live nodes.
DataMemorymemory space also contains UNDO information: For each update, a copy of the unaltered record is allocated in the
DataMemory. There is also a reference to each copy in the ordered table indexes. Unique hash indexes are updated only when the unique index columns are updated, in which case a new entry in the index table is inserted and the old entry is deleted upon commit. For this reason, it is also necessary to allocate enough memory to handle the largest transactions performed by applications using the cluster. In any case, performing a few large transactions holds no advantage over using many smaller ones, for the following reasons:
Large transactions are not any faster than smaller ones
Large transactions increase the number of operations that are lost and must be repeated in event of transaction failure
Large transactions use more memory
In NDB 7.5 (and earlier), the default value for
DataMemoryis 80MB; beginning with NDB 7.6.2, this is 98MB. The minimum value is 1MB. There is no maximum size, but in reality the maximum size has to be adapted so that the process does not start swapping when the limit is reached. This limit is determined by the amount of physical RAM available on the machine and by the amount of memory that the operating system may commit to any one process. 32-bit operating systems are generally limited to 2-4GB per process; 64-bit operating systems can use more. For large databases, it may be preferable to use a 64-bit operating system for this reason.
Table 21.38 This table provides type and value information for the IndexMemory data node configuration parameter
In NDB 7.5 and earlier, this parameter controls the amount of storage used for hash indexes in NDB Cluster. Hash indexes are always used for primary key indexes, unique indexes, and unique constraints. When defining a primary key or a unique index, two indexes are created, one of which is a hash index used for all tuple accesses as well as lock handling. This index is also used to enforce unique constraints.
Beginning with NDB 7.6.2, the
IndexMemoryparameter is deprecated (and subject to future removal); any any memory assigned to
IndexMemoryis allocated instead to the same pool as
DataMemory, which becomes solely responsible for all resources needed for storing data and indexes in memory. In NDB 7.6.2 and later, the use of
IndexMemoryin the cluster configuration file triggers a warning from the management server.
You can estimate the size of a hash index using this formula:
size = ( (fragments * 32K) + (rows * 18) ) * replicas
fragmentsis the number of fragments,
replicasis the number of replicas (normally 2), and
rowsis the number of rows. If a table has one million rows, 8 fragments, and 2 replicas, the expected index memory usage is calculated as shown here:
((8 * 32K) + (1000000 * 18)) * 2 = ((8 * 32768) + (1000000 * 18)) * 2 = (262144 + 18000000) * 2 = 18262144 * 2 = 36524288 bytes = ~35MB
Index statistics for ordered indexes (when these are enabled) are stored in the
mysql.ndb_index_stat_sampletable. Since this table has a hash index, this adds to index memory usage. An upper bound to the number of rows for a given ordered index can be calculated as follows:
sample_size= key_size + ((key_attributes + 1) * 4) sample_rows = IndexStatSaveSize * ((0.01 * IndexStatSaveScale * log2(rows * sample_size)) + 1) / sample_size
In the preceding formula,
key_sizeis the size of the ordered index key in bytes,
key_attributesis the number ot attributes in the ordered index key, and
rowsis the number of rows in the base table.
Assume that table
t1has 1 million rows and an ordered index named
ix1on two four-byte integers. Assume in addition that
IndexStatSaveScaleare set to their default values (32K and 100, respectively). Using the previous 2 formulas, we can calculate as follows:
sample_size = 8 + ((1 + 2) * 4) = 20 bytes sample_rows = 32K * ((0.01 * 100 * log2(1000000*20)) + 1) / 20 = 32768 * ( (1 * ~16.811) +1) / 20 = 32768 * ~17.811 / 20 = ~29182 rows
The expected index memory usage is thus 2 * 18 * 29182 = ~1050550 bytes.
Prior to NDB 7.6.2, the default value for
IndexMemoryis 18MB and the minimum is 1 MB; in NDB 7.6.2 and later, the minimum and default vaue for this parameter is 0 (zero). This has implications for downgrades from NDB 7.6 to earlier versions of NDB Cluster; see Section 21.2.9, “Upgrading and Downgrading NDB Cluster”, for more information.
Table 21.39 This table provides type and value information for the StringMemory data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units % or bytes Default 25 Range 0 - 4294967039 (0xFFFFFEFF) Restart Type S
This parameter determines how much memory is allocated for strings such as table names, and is specified in an
[ndbd default]section of the
config.inifile. A value between
100inclusive is interpreted as a percent of the maximum default value, which is calculated based on a number of factors including the number of tables, maximum table name size, maximum size of
MaxNoOfTriggers, maximum column name size, and maximum default column value.
A value greater than
100is interpreted as a number of bytes.
The default value is 25—that is, 25 percent of the default maximum.
Under most circumstances, the default value should be sufficient, but when you have a great many
NDBtables (1000 or more), it is possible to get Error 773 Out of string memory, please modify StringMemory config parameter: Permanent error: Schema error, in which case you should increase this value.
25(25 percent) is not excessive, and should prevent this error from recurring in all but the most extreme conditions.
The following example illustrates how memory is used for a table. Consider this table definition:
CREATE TABLE example ( a INT NOT NULL, b INT NOT NULL, c INT NOT NULL, PRIMARY KEY(a), UNIQUE(b) ) ENGINE=NDBCLUSTER;
For each record, there are 12 bytes of data plus 12 bytes overhead. Having no nullable columns saves 4 bytes of overhead. In addition, we have two ordered indexes on columns
b consuming roughly 10 bytes each per record. There is a primary key hash index on the base table using roughly 29 bytes per record. The unique constraint is implemented by a separate table with
b as primary key and
a as a column. This other table consumes an additional 29 bytes of index memory per record in the
example table as well 8 bytes of record data plus 12 bytes of overhead.
Thus, for one million records, we need 58MB for index memory to handle the hash indexes for the primary key and the unique constraint. We also need 64MB for the records of the base table and the unique index table, plus the two ordered index tables.
You can see that hash indexes takes up a fair amount of memory space; however, they provide very fast access to the data in return. They are also used in NDB Cluster to handle uniqueness constraints.
Currently, the only partitioning algorithm is hashing and ordered indexes are local to each node. Thus, ordered indexes cannot be used to handle uniqueness constraints in the general case.
An important point for both
DataMemory is that the total database size is the sum of all data memory and all index memory for each node group. Each node group is used to store replicated information, so if there are four nodes with two replicas, there will be two node groups. Thus, the total data memory available is 2 ×
DataMemory for each data node.
It is highly recommended that
IndexMemory be set to the same values for all nodes. Data distribution is even over all nodes in the cluster, so the maximum amount of space available for any node can be no greater than that of the smallest node in the cluster.
DataMemory (and in NDB 7.5 and earlier
IndexMemory) can be changed, but decreasing it can be risky; doing so can easily lead to a node or even an entire NDB Cluster that is unable to restart due to there being insufficient memory space. Increases should be acceptable, but it is recommended that such upgrades are performed in the same manner as a software upgrade, beginning with an update of the configuration file, and then restarting the management server followed by restarting each data node in turn.
MinFreePct. A proportion (5% by default) of data node resources including
DataMemory (and in NDB 7.5 and earlier,
IndexMemory) is kept in reserve to insure that the data node does not exhaust its memory when performing a restart. This can be adjusted using the
MinFreePct data node configuration parameter (default 5).
Table 21.40 This table provides type and value information for the MinFreePct data node configuration parameter
|Version (or later)||NDB 7.5.0|
|Type or units||unsigned|
|Range||0 - 100|
Updates do not increase the amount of index memory used. Inserts take effect immediately; however, rows are not actually deleted until the transaction is committed.
Transaction parameters. The next few
[ndbd] parameters that we discuss are important because they affect the number of parallel transactions and the sizes of transactions that can be handled by the system.
MaxNoOfConcurrentTransactions sets the number of parallel transactions possible in a node.
MaxNoOfConcurrentOperations sets the number of records that can be in update phase or locked simultaneously.
Both of these parameters (especially
MaxNoOfConcurrentOperations) are likely targets for users setting specific values and not using the default value. The default value is set for systems using small transactions, to ensure that these do not use excessive memory.
MaxDMLOperationsPerTransaction sets the maximum number of DML operations that can be performed in a given transaction.
Table 21.41 This table provides type and value information for the MaxNoOfConcurrentTransactions data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units integer Default 4096 Range 32 - 4294967039 (0xFFFFFEFF) Restart Type N Deprecated Yes (in NDB 8.0)
Each cluster data node requires a transaction record for each active transaction in the cluster. The task of coordinating transactions is distributed among all of the data nodes. The total number of transaction records in the cluster is the number of transactions in any given node times the number of nodes in the cluster.
Transaction records are allocated to individual MySQL servers. Each connection to a MySQL server requires at least one transaction record, plus an additional transaction object per table accessed by that connection. This means that a reasonable minimum for the total number of transactions in the cluster can be expressed as
TotalNoOfConcurrentTransactions = (maximum number of tables accessed in any single transaction + 1) * number of SQL nodes
Suppose that there are 10 SQL nodes using the cluster. A single join involving 10 tables requires 11 transaction records; if there are 10 such joins in a transaction, then 10 * 11 = 110 transaction records are required for this transaction, per MySQL server, or 110 * 10 = 1100 transaction records total. Each data node can be expected to handle TotalNoOfConcurrentTransactions / number of data nodes. For an NDB Cluster having 4 data nodes, this would mean setting
MaxNoOfConcurrentTransactionson each data node to 1100 / 4 = 275. In addition, you should provide for failure recovery by ensuring that a single node group can accommodate all concurrent transactions; in other words, that each data node's MaxNoOfConcurrentTransactions is sufficient to cover a number of transactions equal to TotalNoOfConcurrentTransactions / number of node groups. If this cluster has a single node group, then
MaxNoOfConcurrentTransactionsshould be set to 1100 (the same as the total number of concurrent transactions for the entire cluster).
In addition, each transaction involves at least one operation; for this reason, the value set for
MaxNoOfConcurrentTransactionsshould always be no more than the value of
This parameter must be set to the same value for all cluster data nodes. This is due to the fact that, when a data node fails, the oldest surviving node re-creates the transaction state of all transactions that were ongoing in the failed node.
It is possible to change this value using a rolling restart, but the amount of traffic on the cluster must be such that no more transactions occur than the lower of the old and new levels while this is taking place.
The default value is 4096.
Table 21.42 This table provides type and value information for the MaxNoOfConcurrentOperations data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units integer Default 32K Range 32 - 4294967039 (0xFFFFFEFF) Restart Type N
It is a good idea to adjust the value of this parameter according to the size and number of transactions. When performing transactions which involve only a few operations and records, the default value for this parameter is usually sufficient. Performing large transactions involving many records usually requires that you increase its value.
Records are kept for each transaction updating cluster data, both in the transaction coordinator and in the nodes where the actual updates are performed. These records contain state information needed to find UNDO records for rollback, lock queues, and other purposes.
This parameter should be set at a minimum to the number of records to be updated simultaneously in transactions, divided by the number of cluster data nodes. For example, in a cluster which has four data nodes and which is expected to handle one million concurrent updates using transactions, you should set this value to 1000000 / 4 = 250000. To help provide resiliency against failures, it is suggested that you set this parameter to a value that is high enough to permit an individual data node to handle the load for its node group. In other words, you should set the value equal to
total number of concurrent operations / number of node groups. (In the case where there is a single node group, this is the same as the total number of concurrent operations for the entire cluster.)
Because each transaction always involves at least one operation, the value of
MaxNoOfConcurrentOperationsshould always be greater than or equal to the value of
Read queries which set locks also cause operation records to be created. Some extra space is allocated within individual nodes to accommodate cases where the distribution is not perfect over the nodes.
When queries make use of the unique hash index, there are actually two operation records used per record in the transaction. The first record represents the read in the index table and the second handles the operation on the base table.
The default value is 32768.
This parameter actually handles two values that can be configured separately. The first of these specifies how many operation records are to be placed with the transaction coordinator. The second part specifies how many operation records are to be local to the database.
A very large transaction performed on an eight-node cluster requires as many operation records in the transaction coordinator as there are reads, updates, and deletes involved in the transaction. However, the operation records of the are spread over all eight nodes. Thus, if it is necessary to configure the system for one very large transaction, it is a good idea to configure the two parts separately.
MaxNoOfConcurrentOperationswill always be used to calculate the number of operation records in the transaction coordinator portion of the node.
It is also important to have an idea of the memory requirements for operation records. These consume about 1KB per record.
Table 21.43 This table provides type and value information for the MaxNoOfLocalOperations data node configuration parameter
Property Value Version (or later) NDB 7.5.0 Type or units integer Default UNDEFINED Range 32 - 4294967039 (0xFFFFFEFF) Restart Type N Deprecated Yes (in N