Getting Started with Kafka
MapReduce Service (MRS) provides enterprise-level big data clusters on the cloud. Tenants can fully control clusters and easily run big data components such as Hadoop, Spark, HBase, Kafka, and Storm.
This document uses a cluster with Kerberos authentication disabled as an example to describe how to generate and consume messages in a Kafka topic from scratch. If Kerberos authentication is enabled in the cluster, you need to authenticate the user when performing Step 5 and change the port number when performing Step 6.
Buy a cluster. -> Log in to the Master node using VNC. -> Use the Kafka client to create a topic. -> Manage messages in the Kafka topic.
Step 1: Buy a Cluster
1. Log in to the public cloud management console.
2. Choose Service List > EI Enterprise Intelligence > MapReduce Service.
The MRS management console is displayed.
3. Click Buy Cluster to switch to the Buy Cluster page.
Buy a Cluster
Step 2: Configure the Cluster
1. Select a billing mode. In this example, Billing Mode is set to Pay-per-use.
2. Configure basic information of the cluster. The figure on the right is for your reference.
3. Configure the password and advanced settings of the cluster. The figure on the right is for your reference.
4. After the configuration is complete, click Next.
Click here to view more parameter description.
Configure the Cluster-01
Configure the Cluster-02
Configure the Cluster-03
Step 3: Confirm Your Cluster Configurations
After confirming the information, click Submit. The system automatically creates the cluster for you.
It takes some time to create the MRS cluster. After you submit the cluster creation request, the initial status of the cluster is Starting. After the cluster has been created, its status changes to Running.
Confirm Your Cluster Configurations
Step 4: Log In to the Master Node Using VNC
1. In the MRS management console, choose Clusters > Active Clusters. In the cluster list, click mrs_test to switch to the cluster details page. On the Nodes tab page, locate the ECS, for which Type is Master2, and click the node name to switch to the ECS details page.
2. Click Remote Login to remotely log in to the Master2 node by using user root and the password specified during cluster creation.
Log In to the Master Node Using VNC-01
Log In to the Master Node Using VNC-02
Step 5: Use the Kafka Client to Create a Topic
1. Configure environment variables.
2. If Kerberos authentication is enabled for the cluster, run the kinit admin command to authenticate the user and enter the password of user admin set during cluster creation as prompted. If Kerberos authentication is disabled for the cluster, skip this step.
3. In the MRS management console, choose Clusters > Active Clusters. In the cluster list, click mrs_test to switch to the cluster details page. Click View next to Cluster Manager to switch to MRS Manager.
4. On MRS Manager, choose Services > ZooKeeper > Instance to view the IP addresses of the ZooKeeper instances.
Record the IP address of any ZooKeeper instance, for example, 192.168.0.237.
5. Run the following command to create a Kafka topic:
kafka-topics.sh --create --zookeeper <IP address of the node where the ZooKeeper instance is located:2181/kafka> --partitions 2 --replication-factor 2 --topic <Topic name>
The figure on the right shows how to create a topic named test.
Configure Environment Variables
Create a Topic
Step 6: Manage Messages in the Topic
1. On MRS Manager, choose Services > Kafka > Instance to view the IP addresses of the Kafka instances.
Record the IP address of any Kafka instance, for example, 192.168.0.237.
2. Produce messages in topic test.
Run the following command first:
kafka-console-producer.sh --broker-list <IP address of the node where the Kafka instance is located:9092> --topic <Topic name> --producer.config /opt/client/Kafka/kafka/config/producer.properties
And then, input specified information as the messages produced by the producer and then press Enter to send the messages. To stop producing messages, press Ctrl+C to exit.
3. Consume messages in topic test.
kafka-console-consumer.sh --topic <Topic name> --bootstrap-server <IP address of the node where the Kafka instance is located:9092> --new-consumer --consumer.config /opt/client/Kafka/kafka/config/consumer.properties
Note: If Kerberos authentication is enabled in the cluster, change the port number 9092 to 21007 when running the preceding two commands. For details, see List of Open Source Component Ports.
Manage Messages in the Topic-01
Manage Messages in the Topic-02