Help Center > > User Guide> Data Backup and Restoration> Kafka Data

Kafka Data

Updated at: Nov 08, 2019 GMT+08:00

MirrorMaker is a powerful tool for Kafka data synchronization. It is used when data needs to be synchronized between two Kafka clusters or when data in the original Kafka cluster needs to be migrated to a new Kafka cluster. MirrorMaker is a built-in tool in Kafka. It actually integrates the functions of Kafka Consumer and Producer. MirrorMaker can read data from one Kafka cluster and write the data to another Kafka cluster to implement data synchronization between Kafka clusters.

This section describes how to use the MirroMaker tool provided by MRS to synchronize and migrate Kafka cluster data. Before migrating Kafka data, ensure that the two clusters can communicate with each other by following the instructions provided in Establishing a Data Transmission Channel.

Procedure

  1. Enable the Kerberos authentication for the clusters. For details, see Configuring Cross-Cluster Mutual Trust Relationships.
  2. If you plan to use the MirrorMaker tool in a source cluster, go to the details page of a destination cluster and choose Components. If you plan to use the MirrorMaker tool in a destination cluster, go to the details page of a source cluster and choose Components.

    For MRS 2.0.1 or earlier, if you plan to use the MirrorMaker tool in the source cluster, log in to MRS Manager of the destination cluster and choose Services. If you plan to use the MirrorMaker tool in the destination cluster, log in to MRS Manager of the destination cluster and choose Services.

  3. Choose Kafka > Service Configuration, and set Type to All.
  4. Click Customization and add the following rules on the displayed page:

    sasl.kerberos.principal.to.local.rules = RULE:[1:$1@$0](.*@XXXYYYZZZ.COM)s/@.*//,RULE:[2:$1@$0](.*@ XXXYYYZZZ.COM)s/@.*//,DEFAULT

    In the preceding rule, XXXYYYZZZ.COM indicates the domain name of the cluster (source cluster) where data resides. The domain name must contain uppercase letters.

    Figure 1 Adding rules

  5. Click Save Configuration and select Restart the affected services or instances. Click Yes to restart the Kafka service.

    For a security cluster with the Kerberos authentication enabled, perform 1 to 5. For a normal cluster with the Kerberos authentication disabled, skip 1 to 5 and go to 6.

  6. In the cluster that uses the MirrorMaker tool, go to the cluster details page and choose Components.

    For MRS 2.0.1 or earlier, log in to MRS Manager and choose Services.

  7. Choose Kafka > Service Configuration, set Type to All, and set Role to MirrorMaker.

    Figure 2 Set the MirrorMaker-related parameters.

    Parameter description:

    • The bootstrap.servers parameter in the source and dest tags indicates the broker node list and port information of the source and destination Kafka clusters respectively.
    • Set parameter security.protocol in the source and dest tags based on the actual configurations of the source and destination Kafka clusters.
    • If the source Kafka cluster or destination Kafka cluster is a security cluster, you need to set kerberos.domain.name and sasl.kerberos.service.name in the source and dest tags. If the local host is used, you do not need to set kerberos.domain.name. If the local host is not used, set kerberos.domain.name and sasl.kerberos.service.name based on the site requirements. The default value of sasl.kerberos.service.name is kafka.
    • Set whitelist in the mirror tag, that is, the name of the topic to be synchronized.

  8. Click Save Configuration and select Restart the affected services or instances. Click Yes to restart the MirrorMaker instance.

    After the MirrorMaker instance is restarted, the data migration task is started. You can use KafkaManager or command line tool to monitor the data synchronization progress of MirrorMaker.

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel