Amazon Managed Streaming for Apache Kafka

Hands-On Demo

In this demo, we will: 

  1. Create an Amazon MSK cluster.
  2. Create IAM Policy and Role for the EC2 Instance to assume.
  3. Create an EC2 instance to interact with the MSK cluster.
  4. Create a Kafka topic.
  5. Produce and consume messages using the Kafka client.
  6. Test the setup.
  7. Clean up resources.

Agenda

Demo Overview

Creata an Amazon MSK Cluster

msk-demo-cluster

Custom create

Cluster type

Brokers

Storage

Configuration

Networking

Security groups

Access control methods

Encryption

Monitoring and tags

Open monitoring with Prometheus

Cluster tags

Review and create

Create IAM Policy and IAM Role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:651623850282:cluster/msk-demo-cluster/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:651623850282:topic/msk-demo-cluster/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:651623850282:group/msk-demo-cluster/*"
            ]
        }
    ]
}

Create policy

msk-iam-policy

Create role

msk-iam-policy

Add permissions

msk-iam-role

Name, review, and create

Create an EC2 Instance

MSK-Demo-Client

Launch an instance

Instance type

Create key pair

msk-demo-key-pair

Key pair (login)

Network settings

Configure storage

Attach IAM Role to EC2 IAM Instance Profile

Launch instance

Allow Traffic from EC2 Instance Security Group to MSK Security Group

Launch-Wizard-1 Security Group  ---> Default Security Group

Final Setting Look Like

Create a Topic in Amazon MSK Cluster

EC2 Instance Connect

sudo yum -y install java-11

wget https://archive.apache.org/dist/kafka/3.5.1/kafka_2.13-3.5.1.tgz

tar -xzf kafka_2.13-3.5.1.tgz

cd kafka_2.13-3.5.1/libs/

wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar

ls aws-msk-iam-auth-1.1.1-all.jar

cd ../bin

Install Java and Kafka

pwd

export PLAIN="
./kafka-topics.sh --create \
--bootstrap-server $PLAIN \
--replication-factor 2 \
--partitions 1 \
--topic msk-demo-topic

Plaintext Connection

export SSL="
./kafka-topics.sh --create \
--bootstrap-server $SSL \
--command-config client.properties \
--replication-factor 2 \
--partitions 1 \
--topic msk-demo-topic-2
cat << EOF > client.properties
security.protocol=SSL
ssl.truststore.location=/usr/lib/jvm/java-11-amazon-corretto/lib/security/cacerts
ssl.truststore.password=changeit
EOF
cat client.properties

SSL Based Connection

Open a Kafka

Consumer Client

cd kafka_2.13-3.5.1/bin

export SSL="
./kafka-console-consumer.sh \
--bootstrap-server $SSL \
--consumer.config client.properties \
--topic msk-demo-topic --from-beginning

Consumer

Producer  

export SSL="
./kafka-console-producer.sh \
--broker-list $SSL \
--producer.config client.properties \
--topic msk-demo-topic
hi, how are you

i am good

what a lovely way to exchange messages

i will continue to send messages

and this will be picked up when the consumer comes online

great

Shutdown the consumer for testing

Clean Up

Delete msk-demo-cluster

Terminate (delete) instance

Delete IAM Policy

Delete IAM Role

Delete Key Pair

Remove Inbound Rule from Default Security Group

Delete launch-wizard-1 Security Group

🙏

Thanks

for

Watching

Amazon Managed Streaming for Apache Kafka - Hands-On Demo

By Deepak Dubey

Amazon Managed Streaming for Apache Kafka - Hands-On Demo

Amazon Managed Streaming for Apache Kafka - Hands-On Demo

  • 75

More from Deepak Dubey