Storm in the Wild
What is it?
- Realtime Computation System
- Distributed
- Fault tolerant
- Can be used with any language
Lambda Architecture
Speed Layer
|
Serving Layer
|
Batch Layer
My Life Before Storm
Things I spent my time doing...
- Standing up Message Queues and Services
- Fiddling with config between queues and services
- Writing endless monitors for queues and services
- Writing endless automated or manual recovery processes when queues/services failed
- Devising elaborate plans for message tracking through the system
- Building great integration environments to actually tests all of this in
- Creating deployment plans
- (sometimes writing the actual business logic I needed .....)
Things Storm Does For Me
- Manages messaging between bolts
- Manages bolt health
- Manages deployments
- Tracks messages, can guarantee reliable processing
- Easy to run locally
Let's Contrive an Example!
Demo!
Enough Talking Phil, I demand Code Now
https://github.com/philipsdoctor/storm-demo-talk
Speed
Storm's inherent parallelism means it can process very high throughputs of messages with very low latency. Storm was benchmarked at processing one million 100 byte messages per second per node on the following hardware:
-
Processor: 2x Intel ES645@2.4Ghz
-
Memory: 24 GB
Other Things to do with Storm...
- DRPC
- Trident (Marceline)
Storm in the Wild
By Philip Doctor
