Amazon Managed Streaming for Apache Kafka - Hands On Demo

Step 1: Create an Amazon MSK cluster

 

  1. Sign in to the AWS Management Console, and open the Amazon MSK console.

  2. Choose Create cluster.

  3. For Creation method, leave the Quick create option selected. The Quick create option lets you create a cluster with default settings.

  4. For Cluster name, enter a descriptive name for your cluster. 

  5. For General cluster properties, choose Provisioned as the Cluster type.

  6. From the table under All cluster settings, copy the values of the following settings and save them because you need them later in this tutorial:

    • VPC

    • Subnets

    • Security groups associated with VPC

  7. Choose Create cluster.

  8. Check the cluster Status on the Cluster summary page. The status changes from Creating to Active as Amazon MSK provisions the cluster. When the status is Active, you can connect to the cluster. 

MSK-Tutorial-Cluster

Step 2: Create an IAM role

 

  1. Open the IAM console.

  2. On the navigation pane, choose Policies.

  3. Choose Create Policy.

  4. Choose the JSON tab, then replace the JSON in the editor window with the following JSON.

    Replace region with the code of the AWS region where you created your cluster. Replace Account-ID with your account ID. Replace MSK-Tutorial-Cluster with the name of your cluster.

  5. Choose Next: Tags.

  6. Choose Next: Review.

  7. For the policy name, enter a descriptive name, such as msk-tutorial-policy.

  8. Choose Create policy.

msk-tutorial-policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:651623850282:cluster/MSK-Tutorial-Cluster/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:651623850282:topic/MSK-Tutorial-Cluster/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:651623850282:group/MSK-Tutorial-Cluster/*"
            ]
        }
    ]
}
To create an IAM role and attach the policy to it

  1. On the navigation pane, choose Roles.

  2. Choose Create role.

  3. Under Common use cases, choose EC2, then choose Next: Permissions.

  4. In the search box, enter the name of the policy that you previously created for this tutorial. Then select the box to the left of the policy.

  5. Choose Next: Tags.

  6. Choose Next: Review.

  7. For the role name, enter a descriptive name, such as msk-tutorial-role.

  8. Choose Create role.

msk-tutorial-role

Step 3: Create a client machine

 

  1. Open the Amazon EC2 console.

  2. Choose Launch instances.

  3. Enter a Name for your client machine, such as MSK-Tutorial-Client.

  4. Leave Amazon Linux 2 AMI (HVM) - Kernel 5.10, SSD Volume Type selected for Amazon Machine Image (AMI) type.

  5. Leave the t2.micro instance type selected.

  6. Under Key pair (login), choose Create a new key pair. Enter MSK-Key-Pair for Key pair name, and then choose Download Key Pair. Alternatively, you can use an existing key pair.

  7. Expand the Advanced details section and choose the IAM role that you created in previous step.

  8. Choose Launch instance.

  9. Choose View Instances. Then, in the Security Groups column, choose the security group that is associated with your new instance. Copy the ID of the security group, and save it for later.

  10. Open the Amazon VPC console.

  11. In the navigation pane, choose Security Groups. Find the security group whose ID you saved in previous step.

  12. In the Inbound Rules tab, choose Edit inbound rules.

  13. Choose Add rule.

  14. In the new rule, choose All traffic in the Type column. In the second field in the Source column, select the security group of your client machine. This is the group whose name you saved after you launched the client machine instance.

  15. Choose Save rules. Now the cluster's security group can accept traffic that comes from the client machine's security group.

MSK-Key-Pair
MSK-Tutorial-Client

Step 4: Create a topic

 

To find the version of your MSK cluster

  1. Go to the region where you have created your cluster for example https://us-east-1.console.aws.amazon.com/msk/

  2. Select the MSK cluster.

  3. Note the version of Apache Kafka used on the cluster.

  4. Replace instances of Amazon MSK version numbers in this tutorial with the version obtained in last step.

To create a topic on the client machine

  1. Open the Amazon EC2 console.

  2. In the navigation pane, choose Instances. Then select the check box beside the name of the client machine that you created in Create a client machine.

  3. Choose Actions, and then choose Connect. Follow the instructions in the console to connect to your client machine.

  4. Run below commands.

# INSTALL JAVA
sudo yum -y install java-11
# Run the following command in the directory where you downloaded the TAR file in the previous step.
tar -xzf kafka_2.13-${MSKVERSION}.tgz
# PUT VERSION BELOW FIRST
export MSKVERSION=
wget https://archive.apache.org/dist/kafka/${MSKVERSION}/kafka_2.13-${MSKVERSION}.tgz

Go to the kafka_2.13-${MSKVERSION}/libs directory, then run the following command to download the Amazon MSK IAM JAR file. The Amazon MSK IAM JAR makes it possible for the client machine to access the cluster.

cd ~/kafka_2.13-${MSKVERSION}/libs
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
cd ~/kafka_2.13-${MSKVERSION}/bin
cat << EOF > client.properties
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
EOF

Go to the kafka_2.13-${MSKVERSION}/bin directory and run below command:.
This sets up the connectivity from client to kafka cluster.

  1. Open the Amazon MSK console.

  2. Wait for the status of your cluster to become Active. This might take several minutes. After the status becomes Active, choose the cluster name. This takes you to a page containing the cluster summary.

  3. Choose View client information.

  4. Copy the connection string for the private endpoint.

    You will get three endpoints for each of the brokers. You only need one broker endpoint for the following step.

  5. Run the following command, replacing BootstrapServerString with one of the broker endpoints that you obtained in the previous step.

  6. If the command succeeds, you see the following message: Created topic MSK-Tutorial-Topic.

export MSKVERSION=2.8.1
export KAFKA_HOME=~/kafka_2.13-${MSKVERSION}
export BootstrapServerString=
${KAFKA_HOME}/bin/kafka-topics.sh \
  --create \
  --bootstrap-server $BootstrapServerString \
  --command-config client.properties \
  --replication-factor 3 \
  --partitions 1 \
  --topic MSK-Tutorial-Topic

Step 5: Produce and consume data

 

Run the following command to start a console producer.

${KAFKA_HOME}/bin/kafka-console-producer.sh \
  --broker-list $BootstrapServerString \
  --producer.config client.properties \
  --topic MSK-Tutorial-Topic
  1. Enter any message that you want, and press Enter. Repeat this step two or three times. Every time you enter a line and press Enter, that line is sent to your Apache Kafka cluster as a separate message.
  2. Keep the connection to the client machine open, and then open a second, separate connection to that machine in a new window.
cd ${KAFKA_HOME}/bin
${KAFKA_HOME}/bin/kafka-console-consumer.sh \
  --bootstrap-server $BootstrapServerString \
  --consumer.config client.properties \
  --topic MSK-Tutorial-Topic \
  --from-beginning
export BootstrapServerString=
export MSKVERSION=2.8.1
export KAFKA_HOME=~/kafka_2.13-${MSKVERSION}

Step 6: Use Amazon CloudWatch to view Amazon MSK metrics

 

  1. Open the CloudWatch console.

  2. In the navigation pane, choose Metrics.

  3. Choose the All metrics tab, and then choose AWS/Kafka.

  4. To view broker-level metrics, choose Broker ID, Cluster Name. For cluster-level metrics, choose Cluster Name.

Step 7: Delete the AWS resources created for this tutorial

 

  1. Open the Amazon MSK console.

  2. Choose the name of your cluster. For example, MSK-Tutorial-Cluster.

  3. Choose Actions, then choose Delete.

  4. Open the Amazon EC2 console.

  5. Choose the instance that you created for your client machine, for example, MSK-Tutorial-Client.

  6. Choose Instance state, then choose Terminate instance.

To delete the IAM policy and role

  1. Open the IAM console.

  2. On the navigation pane, choose Roles.

  3. In the search box, enter the name of the IAM role that you created for this tutorial.

  4. Choose the role. Then choose Delete role, and confirm the deletion.

  5. On the navigation pane, choose Policies.

  6. In the search box, enter the name of the policy that you created for this tutorial.

  7. Choose the policy to open its summary page. On the policy's Summary page, choose Delete policy.

  8. Choose Delete.

Thanks

For

Watching