Help Center > > User Guide> MRS Cluster Component Operation Guide> Using HBase> Configuring the HBase Replication Function

Configuring the HBase Replication Function

Updated at:Apr 28, 2020 GMT+08:00

Scenario

As a key feature to ensure high availability of the HBase cluster system, HBase cluster replication provides HBase with remote data replication in real time. It provides basic O&M tools such as tools used for the replication relationship maintenance, data reconstruction, data verification, and data synchronization progress view. To achieve real-time data replication, you can replicate data from the HBase cluster to another one.

Prerequisites

  • The active and standby clusters have been successfully installed and started (the cluster status is Running on the Active Clusters page), and you have the administrator rights of the clusters.
  • The network between the active and standby clusters is normal and ports can be used properly. For details, see the MapReduce Service Communication Matrix.
  • If the active cluster has historical data and the data needs to be synchronized to the standby cluster, cross-cluster replication has been configured for the active and standby clusters. For details, see Enabling the Cross-Cluster Copy Function.
  • The time of the active and standby clusters must be the same, and the NTP service in the active and standby clusters must use the same time source.
  • Mapping relationships between host names and service IP addresses have been configured in the /etc/hosts files in the active and standby clusters by appending "192.***.***.*** host1" to the hosts files.
  • The network bandwidth between the active and standby clusters is determined by service traffic and cannot be smaller than the maximum allowed service traffic.

Restrictions

  • Despite that HBase cluster replication provides the real-time data replication function, the data synchronization progress is determined by several factors, such as the service loads in the active cluster and the health status of processes in the standby cluster. Normally, the standby cluster should not take over services running in the active cluster. In extreme cases, system maintenance personnel and other decision makers determine whether the standby cluster takes over services according to the current data synchronization indicators.
  • Currently, the replication function supports only one active cluster and one standby cluster in HBase.
  • Typically, do not perform operations on data synchronization tables in the standby cluster, such as modifying table properties or deleting tables. If any misoperation on the standby cluster occurs, data synchronization between the active and standby clusters will fail and data of the corresponding table in the standby cluster will be lost.
  • If the replication function of HBase tables in the active cluster is enabled for data synchronization, after modifying the structure of a table in the active cluster, you need to manually modify the structure of the corresponding table in the standby cluster to ensure table structure consistency.

Procedure

Enable the replication function for the active cluster to synchronize data written by Put.

  1. On the MRS cluster details page, click Components.

    • If the Components tab is not displayed on the cluster details page, perform IAM user synchronization first. (On the Dashboard tab page of the cluster details page, click on the right side of IAM User Sync to synchronize IAM users.)
    • For MRS 1.8.10 or earlier, log in to MRS Manager. For details, see Accessing MRS Manager. Then, choose Services.

  2. Choose HBase > Service Configuration, set Type to All, and go to the HBase configuration page.
  3. Choose RegionServer > Replication, and check whether hbase.replication is set to true. If the value is false, set hbase.replication to true.

    In MRS 2.0.1 or later, this configuration has been removed. Therefore, skip step 3.

  4. (Optional) Set configuration items listed in Table 1. You can set parameters based on the description or use the default value.

    Table 1 Optional configuration items

    Navigation Path

    Configuration Item

    Default Value

    Description

    HMaster > Performance

    hbase.master.logcleaner.ttl

    600000

    Specifies the retention period of HLog. If it is set to 604800000 (unit: ms), the retention period of HLog is 7 days.

    hbase.master.cleaner.interval

    60000

    Specifies the deletion interval of HLog. The HLog that exceeds the configured period will be automatically deleted. You are advised to set it to the maximum value to save more HLogs.

    RegionServer > Replication

    replication.source.size.capacity

    16777216

    After data in the active cluster is synchronized to the standby cluster, the active cluster reads and sends data in HLog according to this parameter.

    replication.source.nb.capacity

    25000

    After data in the active cluster is synchronized to the standby cluster, the active cluster reads and sends data in HLog according to this parameter. This parameter is used with replication.source.size.capacity.

    replication.source.maxretriesmultiplier

    10

    Specifies the maximum number of retries for sending log data.

    replication.source.sleepforretries

    1000

    Specifies the sleeping period for log data sending failures. (unit: ms)

    hbase.regionserver.replication.handler.count

    6

    Specifies the number of threads of the RPC handler that the standby cluster uses to receive data.

Enable the replication function for the active cluster to synchronize data written by BulkLoad.

  1. Determine whether to enable the replication function to synchronize data written by BulkLoad.

    If you use the BulkLoad data import feature of HBase and need to synchronize data, you need to enable the replication function.

    If it needs to be enabled, go to 6.

    If it does not need to be enabled, go to 11.

  2. Choose Components > HBase > Service Configuration, set Type to All, and go to the HBase configuration page.
  3. On the HBase configuration interface of the active and standby clusters, search for hbase.replication.cluster.id and modify it. It specifies the HBase ID of the active and standby clusters. For example, the HBase ID of the active cluster is set to replication1 and the HBase ID of the standby cluster is set to replication2 for connecting the active cluster to the standby cluster. To save data overhead, the parameter value length is not recommended to exceed 30.
  4. On the HBase configuration interface of the standby cluster, search for hbase.replication.conf.dir and modify it. It specifies the HBase configurations of the active cluster client used by the standby cluster and is used for data replication when the BulkLoad data replication function is enabled. The parameter value is the path name, for example, /home.

    • When the BulkLoad data replication function is enabled, you need to manually place the HBase client configuration files (core-site.xml, hdfs-site.xml, and hbase-site.xml) in the active cluster on all RegionServer nodes in the standby cluster. The actual path for placing the configuration file is ${hbase.replication.conf.dir}/${hbase.replication.cluster.id}. For example, if hbase.replication.conf.dir of the standby cluster is set to /home and hbase.replication.cluster.id of the active cluster is set to replication1, the actual path for placing the configuration files in the standby cluster is /home/replication1. You also need to change the corresponding directory and file permissions by running the chown -R omm:wheel /home/replication1 command.
    • You can obtain the client configuration files from the client in the active cluster, for example, the /opt/client/HBase/hbase/conf path. For details about how to update the configuration files, see Updating a Client.
    • In versions earlier than MRS 2.0, you do not need to set this parameter and you can skip step 8.

  5. On the HBase configuration interface of the active cluster, search for hbase.replication.bulkload.enabled and change its value to true to enable the replication function to synchronize data written by BulkLoad.

Restart the HBase service and install the client.

  1. Click Save Configuration. In the displayed window, select Restart the affected services or instances and click OK to restart the HBase service.

    After the system displays Operation succeeded, click Finish. The service is successfully started.

  2. In the active and standby clusters, choose Services > HBase > Download Client to download the client. Follow instructions in Updating a Client to update the client configuration file.
  3. Use PuTTY to access the HBase shell of the active cluster as user hbase.

    1. On the active management node where the client has been updated, run the following command to switch to the client directory:

      cd /opt/client

    2. Run the following command to configure the environment variable:

      source bigdata_env

    3. If Kerberos authentication is enabled for the current cluster, run the following command to authenticate the user. If it is disabled, skip this step.

      kinit hbase

      After running kinit hbase, the system prompts you to enter a password. The default password of user hbase is Hbase@123.

    4. Run the following HBase client command:

      hbase shell

Synchronize the table data of the active cluster. (Skip this step if the active cluster has no data.)

  1. Check whether the standby cluster has historical data. If it has historical data and its data must be consistent with data in the active cluster, clear data in the standby cluster.

    1. On the HBase shell of the standby cluster, run the list command to view existing tables in the standby cluster.
    2. Run the disable 'tableName';drop 'tableName' command to delete data tables in the standby cluster.

  1. After the HBase replication function is configured and data synchronization is enabled, check whether tables and data exist in the active cluster and whether the historical data needs to be synchronized to the standby cluster.

    You can run the list command to view the existing tables in the active cluster and run the scan 'tableName' command to check whether there is historical data in the tables.

    • If tables exist and data needs to be synchronized, go to 15.
    • If data does not need to be synchronized, no further action is required.

  2. When the HBase replication function is configured, historical data in a table cannot be automatically synchronized. You need to replicate the historical data of the active cluster and then manually synchronize it to the standby cluster.

    The manual synchronization is single-table synchronization, which can be performed using Export, Distcp, and Import.

    Perform the following steps to manually synchronize a single table:

    1. Export table data from the active cluster.

      hbase org.apache.hadoop.hbase.mapreduce.Export -Dhbase.mapreduce.include.deleted.rows=true Table name Directory that saves source data

      Example: hbase org.apache.hadoop.hbase.mapreduce.Export -Dhbase.mapreduce.include.deleted.rows=true t1 /user/hbase/t1

    2. Copy the exported data to the standby cluster.

      hadoop distcp Directory that saves source data in the active cluster hdfs://ActiveNameNodeIP:9820/Directory that saves source data in the standby cluster

      ActiveNameNodeIP indicates the IP address of the active NameNode in the standby cluster.

      Example: hadoop distcp /user/hbase/t1 hdfs://192.168.40.2:9820/user/hbase/t1

    3. Import data to the standby cluster as the HBase table user of the standby cluster.

      hbase org.apache.hadoop.hbase.mapreduce.Import -Dimport.bulk.output=Directory that saves output in the standby cluster Table name Directory that saves source data in the standby cluster

      hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles Directory that saves output in the standby cluster Table name

      Example:

      hbase org.apache.hadoop.hbase.mapreduce.Import -Dimport.bulk.output=/user/hbase/output_t1 t1 /user/hbase/t1

      hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /user/hbase/output_t1 t1

Add the replication relationship between the active and standby clusters.

  1. Run the following command on the HBase shell to create a replication synchronization relationship for HBase between active and standby clusters.

    add_peer 'Standby cluster ID','CLUSTER_KEY => ZooKeeper address of the standby cluster',{HDFS_CONFS => true}

    • Standby cluster ID indicates an ID for the active cluster to recognize the standby cluster. It is recommended that the ID contain letters and digits.
    • ZooKeeper address of the standby cluster includes the service IP address of ZooKeeper, the port for monitoring client connections, and the HBase root directory of the standby cluster on ZooKeeper.
    • {HDFS_CONFS => true} indicates that the default HDFS configurations of the active cluster will be synchronized to the standby cluster. This parameter is used for HBase of the standby cluster to access HDFS of the active cluster. If the replication function for synchronizing data written by BulkLoad is disabled, you do not need to use this parameter.

      For example, to add a replication relationship between the active and standby clusters containing BulkLoad data, run the command as follows:

      Suppose the standby cluster ID is 1 and the ZooKeeper address of the standby cluster is 192.168.40.2,192.168.40.3,192.168.40.4:2181:/hbase.

      • For MRS 2.0.1 or later, run the following command: add_peer 'replication2',CLUSTER_KEY => '192.168.40.2,192.168.40.3,192.168.40.4:2181:/hbase',CONFIG => { "hbase.regionserver.kerberos.principal" => "<val>", "hbase.master.kerberos.principal" => "<val2>" }
        If hbase.master.kerberos.principal and hbase.regionserver.kerberos.principal parameters are the Kerberos users of HBase in the security cluster, you can search the hbase-site.xml file on the client to obtain the parameter values. For example, if the client is installed in the /opt/client directory of the Master node, you can run the grep "kerberos.principal" /opt/client/HBase/hbase/conf/hbase-site.xml -A1 command to obtain the principal of HBase, as shown in the following figure.
        Figure 1 Obtaining the principal of HBase
      • For versions earlier than MRS 2.0.1, run the following command: add_peer 'replication2',CLUSTER_KEY => '192.168.40.2,192.168.40.3,192.168.40.4:2181:/hbase'
      1. Choose Services > ZooKeeper > Instance to obtain the service IP address of ZooKeeper.
      2. Choose Services > ZooKeeper > Service Configuration and set Type to All. Search for clientPort, namely, the port for monitoring client connections.
      3. Run the list_peers command to check whether the replication relationship between the active and standby clusters is added. If the following information is displayed, the relationship is successfully added.
        hbase(main):003:0> list_peers
        PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL NAMESPACES TABLE_CFS BANDWIDTH SERIAL
        replication2 192.168.0.13,192.168.0.177,192.168.0.25:2181:/hbase ENABLED  true   0 false
        In versions earlier than MRS 2.0.1, if the following information is displayed after you run the list_peers command, the operation is successful.
        hbase(main):003:0> list_peers
        PEER_ID CLUSTER_KEY STATE TABLE_CFS
        replication2 192.168.0.13,192.168.0.177,192.168.0.25:2181:/hbase ENABLED

Specify the data writing status for the active and standby clusters.

  1. On the HBase shell of the active cluster, run the following command to retain the data writing status:

    set_clusterState_active

    If the following information is displayed, the command is successfully executed.

    hbase(main):001:0> set_clusterState_active
    => true

  2. On the HBase shell of the standby cluster, run the following command to retain the data read-only status:

    set_clusterState_standby

    If the following information is displayed, the command is successfully executed.

    hbase(main):001:0> set_clusterState_standby
    => true

Enable the HBase replication function to synchronize data.

  1. Check whether a name space exists in the HBase service instance of the standby cluster and whether the name space has the same name as the name space of the HBase table whose replication function will be enabled.

    You can run the list_namespace command on the HBase shell of the standby cluster to query a name space.
    • If the same name space exists, go to 20.
    • If the same name space does not exist, run the create_namespace 'ns1' command on the HBase shell of the standby cluster to create a same name space and then go to 20.

  2. On the HBase shell of the active cluster, run the following command to enable the real-time data replication function for the tables in the active cluster. This ensures that modified data in the active cluster can be synchronized to the standby cluster in real time.

    You can only synchronize data of one HTable at one time.

    enable_table_replication 'Table name'

    • If a table with the same name as the table where real-time synchronization is to be enabled does not exist in the standby cluster, the system will automatically create the table.
    • If a table with the same name as the table where real-time synchronization is to be enabled exists in the standby cluster, the structures of the two tables must be consistent.
    • If the encryption algorithm SMS4 or AES is configured for 'Table name', the function for synchronizing data from the active cluster to the standby cluster in real time cannot be enabled for the HBase table.
    • If the standby cluster is offline or has a table with the same name but different structure, the replication function will fail to be enabled.

      If the standby cluster is offline, start the standby cluster first.

      If the standby cluster has a table with the same name but different structure, modify the table structure to make it as the same as the table structure of the active cluster. Run the alter command on the HBase shell of the standby cluster as prompted.

  3. On the HBase shell of the active cluster, run the following command to enable the real-time replication function for the active cluster to synchronize the HBase permission table:

    enable_table_replication 'hbase:acl'

    If the standby cluster needs to read data when the permission of the HBase source data table in the active cluster is modified, modify the role permission of the standby cluster.

Check data synchronization status for the active and standby clusters.

  1. Run the following command on the HBase client to check the synchronized data of the active and standby clusters. After the replication function is enabled for data synchronization, you can also run this command to check whether the newly synchronized data is consistent.

    hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication --starttime=Start time --endtime=End time Column family name Standby cluster ID Table name

    • The start time must be earlier than the end time.
    • The values of starttime and endtime must be in the timestamp format. You need to run date -d "2015-09-30 00:00:00" +%s to change a common time format to a timestamp format. After command execution, a 10-digit number (accurate to second) will be returned. However, HBase identifies a 13-digit number (accurate to millisecond). Therefore, you need to supplement three zeros (000) to the command output.

    Switch over active and standby clusters.

    1. If the standby cluster needs to be switched over to an active one, configure an active-standby relationship again by referring to 1 to 12 and 16 to 21. Do not perform 13 to 15 to synchronize table data between active and standby clusters.

Related Commands

Table 2 HBase replication

Operation

Command

Description

Set up an active-standby relationship.

add_peer 'Standby cluster ID',CLUSTER_KEY =>'Standby cluster address'

Examples:

add_peer '1',CLUSTER_KEY => 'zk1,zk2,zk3:2181:/hbase'

add_peer '1',CLUSTER_KEY => 'zk1,zk2,zk3:2181:/hbase1'

In versions earlier than MRS 2.0.1, the command for setting up the relationship between the active cluster and the standby cluster is as follows:

add_peer 'Standby cluster ID, 'Standby cluster address'

Examples:

add_peer '1', 'zk1,zk2,zk3:2181:/hbase'

add_peer '1', 'zk1,zk2,zk3:2181:/hbase1'

Set up the relationship between the active cluster and the standby cluster. If the replication function for synchronizing data written by BulkLoad is enabled, run the following command: add_peer 'Standby cluster ID',CLUSTER_KEY => 'Standby cluster address'. Configure the hbase.replication.conf.dir parameter and manually copy the HBase client configuration files of the active cluster to all RegionServer nodes in the standby cluster. For details, see 5 to 11.

In versions earlier than MRS 2.0.1, if the replication function for synchronizing data written by BulkLoad is enabled, run the following command: add_peer 'Standby cluster ID','Standby cluster address',{HDFS_CONF => true}

Remove an active-standby relationship.

remove_peer 'Standby cluster ID'

Example:

remove_peer '1'

Remove standby cluster information from the active cluster.

Query an active-standby relationship.

list_peers

Query standby cluster information (mainly Zookeeper information) in the active cluster.

Enable the real-time user table synchronization function.

enable_table_replication 'Table name'

Example:

enable_table_replication 't1'

In the active cluster, synchronize existing user tables to the standby cluster.

Disable the real-time user table synchronization function.

disable_table_replication 'Table name'

Example:

disable_table_replication 't1'

In the active cluster, do not synchronize existing user tables to the standby cluster.

Verify data in the active and standby clusters.

bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication --starttime=Start time --endtime=End time Column family name Standby cluster ID Table name

Verify whether data of the specified table is the same between the active cluster and the standby cluster.

The description of the parameters in this command is as follows:

  • Start time: If start time is not specified, the default value 0 will be used.
  • End time: If end time is not specified, the time when the current operation is submitted will be used by default.
  • Table name: If a table name is not entered, all user tables for which the real-time synchronization function is enabled will be verified by default.

Switch data writing status.

set_clusterState_active

set_clusterState_standby

Specify whether data can be written to the cluster HBase tables.

Add or update the active cluster HDFS configurations saved in the peer cluster.

set_replication_hdfs_confs 'PeerId', {'key1' => 'value1', 'key2' => 'value2'}

Enable the replication function for data including BulkLoad data. When HDFS parameters are modified in the active cluster, the modification cannot be automatically synchronized to the standby cluster. You need to manually run the commands to synchronize the changes. The affected parameters are as follows:

  • fs.defaultFS
  • dfs.client.failover.proxy.provider.hacluster
  • dfs.client.failover.connection.retries.on.timeouts
  • dfs.client.failover.connection.retries

For example, if the value of fs.defaultFS is changed to hdfs://hacluster_sale, run the following command to synchronize HDFS configurations to the standby cluster whose ID is 1:

set_replication_hdfs_confs '1', {'fs.defaultFS' => 'hdfs://hacluster_sale'}

In MRS 2.0.1 or later, this command has been removed. If synchronization is required, manually copy the changed client configurations in the active cluster to the standby cluster. For details, see 8.

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel