Amazon Managed Streaming for Apache Kafka
Hands-On Demo

In this demo, we will:
- Create an Amazon MSK cluster.
- Create IAM Policy and Role for the EC2 Instance to assume.
- Create an EC2 instance to interact with the MSK cluster.
- Create a Kafka topic.
- Produce and consume messages using the Kafka client.
- Test the setup.
- Clean up resources.
Agenda

Demo Overview
Creata an Amazon MSK Cluster


msk-demo-cluster
Custom create

Cluster type

Brokers

Storage

Configuration



Networking

Security groups

Access control methods


Encryption

Monitoring and tags

Open monitoring with Prometheus

Cluster tags

Review and create





Create IAM Policy and IAM Role
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kafka-cluster:Connect",
"kafka-cluster:AlterCluster",
"kafka-cluster:DescribeCluster"
],
"Resource": [
"arn:aws:kafka:us-east-1:651623850282:cluster/msk-demo-cluster/*"
]
},
{
"Effect": "Allow",
"Action": [
"kafka-cluster:*Topic*",
"kafka-cluster:WriteData",
"kafka-cluster:ReadData"
],
"Resource": [
"arn:aws:kafka:us-east-1:651623850282:topic/msk-demo-cluster/*"
]
},
{
"Effect": "Allow",
"Action": [
"kafka-cluster:AlterGroup",
"kafka-cluster:DescribeGroup"
],
"Resource": [
"arn:aws:kafka:us-east-1:651623850282:group/msk-demo-cluster/*"
]
}
]
}

Create policy

msk-iam-policy

Create role

msk-iam-policy
Add permissions

msk-iam-role
Name, review, and create


Create an EC2 Instance

MSK-Demo-Client
Launch an instance



Instance type

Create key pair
msk-demo-key-pair

Key pair (login)

Network settings

Configure storage

Attach IAM Role to EC2 IAM Instance Profile

Launch instance

Allow Traffic from EC2 Instance Security Group to MSK Security Group
Launch-Wizard-1 Security Group ---> Default Security Group
Final Setting Look Like




Create a Topic in Amazon MSK Cluster

EC2 Instance Connect
sudo yum -y install java-11
wget https://archive.apache.org/dist/kafka/3.5.1/kafka_2.13-3.5.1.tgz
tar -xzf kafka_2.13-3.5.1.tgz
cd kafka_2.13-3.5.1/libs/
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
ls aws-msk-iam-auth-1.1.1-all.jar
cd ../bin
Install Java and Kafka
pwd
export PLAIN="
./kafka-topics.sh --create \
--bootstrap-server $PLAIN \
--replication-factor 2 \
--partitions 1 \
--topic msk-demo-topic
Plaintext Connection
export SSL="
./kafka-topics.sh --create \
--bootstrap-server $SSL \
--command-config client.properties \
--replication-factor 2 \
--partitions 1 \
--topic msk-demo-topic-2
cat << EOF > client.properties
security.protocol=SSL
ssl.truststore.location=/usr/lib/jvm/java-11-amazon-corretto/lib/security/cacerts
ssl.truststore.password=changeit
EOF
cat client.properties
SSL Based Connection
Open a Kafka
Consumer Client
cd kafka_2.13-3.5.1/bin
export SSL="
./kafka-console-consumer.sh \
--bootstrap-server $SSL \
--consumer.config client.properties \
--topic msk-demo-topic --from-beginning
Consumer
Producer
export SSL="
./kafka-console-producer.sh \
--broker-list $SSL \
--producer.config client.properties \
--topic msk-demo-topic
hi, how are you
i am good
what a lovely way to exchange messages
i will continue to send messages
and this will be picked up when the consumer comes online
great
Shutdown the consumer for testing
Clean Up

Delete msk-demo-cluster


Terminate (delete) instance


Delete IAM Policy


Delete IAM Role


Delete Key Pair
Remove Inbound Rule from Default Security Group


Delete launch-wizard-1 Security Group

🙏
Thanks
for
Watching
Amazon Managed Streaming for Apache Kafka - Hands-On Demo
By Deepak Dubey
Amazon Managed Streaming for Apache Kafka - Hands-On Demo
Amazon Managed Streaming for Apache Kafka - Hands-On Demo
- 196