Help Center > > Getting Started> Getting Started with Kafka
Getting Started with Kafka

Getting Started with Kafka

Updated at: May 16, 2019 GMT+08:00
  • MapReduce Service (MRS) provides enterprise-level big data clusters on the cloud. Tenants can fully control clusters and easily run big data components such as Hadoop, Spark, HBase, Kafka, and Storm.

  • This document uses a cluster with Kerberos authentication disabled as an example to describe how to generate and consume messages in a Kafka topic from scratch. If Kerberos authentication is enabled in the cluster, you need to authenticate the user when performing Step 5 and change the port number when performing Step 6.

  • Buy a cluster. -> Log in to the Master node using VNC. -> Use the Kafka client to create a topic. -> Manage messages in the Kafka topic.

Step 1: Buy a Cluster

1. Log in to the public cloud management console.
2. Choose Service List > EI Enterprise Intelligence > MapReduce Service.
The MRS management console is displayed.
3. Click Buy Cluster to switch to the Buy Cluster page.

1

MapReduce Service

Apply for a VPC.

2

Buy a Cluster

Apply for an ECS.

View Image

Step 2: Configure the Cluster

1. Select a billing mode. In this example, Billing Mode is set to Pay-per-use.
2. Configure basic information of the cluster. The figure on the right is for your reference.

3. Configure the password and advanced settings of the cluster. The figure on the right is for your reference.

4. After the configuration is complete, click Next.
Click here to view more parameter description.

1

Configure the Cluster-01

Select the charging mode.

2

Configure the Cluster-02

选择实例配置

3

Configure the Cluster-03

选择实例配置

View Image

Step 3: Confirm Your Cluster Configurations

After confirming the information, click Submit. The system automatically creates the cluster for you.
It takes some time to create the MRS cluster. After you submit the cluster creation request, the initial status of the cluster is Starting. After the cluster has been created, its status changes to Running.

1

Confirm Your Cluster Configurations

Obtain the instance's connection address.

View Image

Step 4: Log In to the Master Node Using VNC

1. In the MRS management console, choose Clusters > Active Clusters. In the cluster list, click mrs_test to switch to the cluster details page. On the Nodes tab page, locate the ECS, for which Type is Master2, and click the node name to switch to the ECS details page.
2. Click Remote Login to remotely log in to the Master2 node by using user root and the password specified during cluster creation.

1

Log In to the Master Node Using VNC-01

Obtain the instance's connection address.

2

Log In to the Master Node Using VNC-02

Obtain the instance's connection address.

View Image

Step 5: Use the Kafka Client to Create a Topic

1. Configure environment variables.
source /opt/client/bigdata_env
2. If Kerberos authentication is enabled for the cluster, run the kinit admin command to authenticate the user and enter the password of user admin set during cluster creation as prompted. If Kerberos authentication is disabled for the cluster, skip this step.

3. In the MRS management console, choose Clusters > Active Clusters. In the cluster list, click mrs_test to switch to the cluster details page. Click View next to Cluster Manager to switch to MRS Manager.
4. On MRS Manager, choose Services > ZooKeeper > Instance to view the IP addresses of the ZooKeeper instances.
Record the IP address of any ZooKeeper instance, for example, 192.168.0.237.
5. Run the following command to create a Kafka topic:
kafka-topics.sh --create --zookeeper <IP address of the node where the ZooKeeper instance is located:2181/kafka> --partitions 2 --replication-factor 2 --topic <Topic name>
The figure on the right shows how to create a topic named test.

1

Configure Environment Variables

Obtain the instance's connection address.

2

Create a Topic

Obtain the instance's connection address.

View Image

Step 6: Manage Messages in the Topic

1. On MRS Manager, choose Services > Kafka > Instance to view the IP addresses of the Kafka instances.
Record the IP address of any Kafka instance, for example, 192.168.0.237.
2. Produce messages in topic test
Run the following command first:
kafka-console-producer.sh --broker-list <IP address of the node where the Kafka instance is located:9092> --topic <Topic name> --producer.config /opt/client/Kafka/kafka/config/producer.properties
And then, input specified information as the messages produced by the producer and then press Enter to send the messages. To stop producing messages, press Ctrl+C to exit.
3. Consume messages in topic test.
kafka-console-consumer.sh --topic <Topic name> --bootstrap-server <IP address of the node where the Kafka instance is located:9092> --new-consumer --consumer.config /opt/client/Kafka/kafka/config/consumer.properties

Note: If Kerberos authentication is enabled in the cluster, change the port number 9092 to 21007 when running the preceding two commands. For details, see List of Open Source Component Ports.

 

1

Manage Messages in the Topic-01

Obtain the instance's connection address.

2

Manage Messages in the Topic-02

Obtain the instance's connection address.

View Image

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel