Apache Pulsar and the Legacy of Telegraphy
Jowanza Joseph
@jowanza
Agenda
- About Me
- History Lesson
- Pulsar Internals
- Pulsar Usages
- The Future
- Questions?*
- Software Engineer at Pluralsight
- Live in Salt Lake City area
- Cyclist/Model Rocketry
History
Problems
- Data Storage
- Data Transmission
- Multi-casting
- Encryption
- Delivery Guarantees
Problems
- Data Storage
- Data Transmission
- Multi-casting
- Encryption
- Delivery Guarantees
Data Storage
Immutable Log
Tiered Storage
Problems
Data Storage- Data Transmission
- Multi-casting
- Encryption
- Delivery Guarantees
Data Transmission
{"namespace": "com.bigdatums.avro",
"type": "record",
"name": "BdPerson",
"fields": [
{"name": "id", "type": "int"},
{"name": "username", "type": "string"},
]
}Schema Evolution
{"namespace": "com.bigdatums.avro",
"type": "record",
"name": "BdPerson",
"fields": [
{"name": "id", "type": "int"},
{"name": "username", "type": ["string", "null"], "default": "null"}
}Problems
Data StorageData Transmission- Multi-casting
- Encryption
- Delivery Guarantees
Multi-Casting
Problems
Data StorageData Transmission- Multi-casting
- Encryption
- Delivery Guarantees
Encryption
Problems
Data StorageData TransmissionMulti-castingEncryption- Delivery Guarantees
Delivery Guarantees
No Guarantees
At Most Once
At Least Once
Exactly Once
Problems
Data StorageData TransmissionMulti-castingEncryptionDelivery Guarantees
Stream Processing
Pulsar SQL
Pulsar Functions
package main
import (
"context"
"fmt"
"log"
"github.com/apache/pulsar/pulsar-client-go/pulsar"
)
func main() {
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "pulsar://localhost:6650",
})
if err != nil {
log.Fatal(err)
return
}
defer client.Close()
producer, err := client.CreateProducer(pulsar.ProducerOptions{
Topic: "my-topic",
})
defer producer.Close()
ctx := context.Background()
for i := 0; i < 10; i++ {
if err := producer.Send(ctx, pulsar.ProducerMessage{
Payload: []byte(fmt.Sprintf("hello-%d", i)),
}); err != nil {
log.Fatal(err)
}
}
}Future
Complex Logic
Try Pulsar
- Read the Pulsar Docs
- Try Pulsar Locally
- Free Trial of Streamlio
Questions?
@jowanza
me@jowanza.com
Apache Pulsar and the Legacy of Telegraphy
By Jowanza Joseph
