CS2031:

Desarrollo Basado en Plataformas

Semana 16: Kafka

Contenido

• Data Problem
   
• Publish/Subscribe Messaging
   
• Kafka: Overview
   
• Demo
  • Installing Kafka: Docker/AWS

Data Problem

Data Problem

data

data

data

If you have 4 source systems, and 4 target systems, you need to write 16 integrations.

Data Problem

If you have 4 source systems, and 4 target systems, you need to write 16 integrations.

What's the problem?

Data Problem

Problem:

Protocol - How the data is transported (TCP, HTTP, REST, FTP, JDBC, etc)

Data format - How the data is parsed (Binary, CSV, JSON, Avro, etc)

Data Schema & Evolution - How to data is shaped and may change

Connections - Each source system will have an increased load from the connections

If you have 4 source systems, and 4 target systems, you need to write 16 integrations.

Data Solution

broker

Data Solution

broker

Use cases: Messaging System, Activity tracking, Gather metrics from many different locations, Application Logs gathering, etc

Publish/Subscribe Messaging

"Pub/Sub messasing is a pattern that is characterized by the sender (publisher) of a piece of data (message) not specifically directing it to a receiver.

The publisher classifies the message, and the receiver (subscriber) subscribes to receive certain classes of messages"

Kafka: Overview

Kafka Cluster

Kafka: Overview

Topics:
- Partitions

- Replications

References

• Kafka
  https://kafka.apache.org/
   
• Kafka course
  https://www.conduktor.io/kafka/