resource and cluster management
YARN
YARN basics
TEZ
Stinger.next
SPARK
SPARK: DAG & RDD
dataflow topologies
storm
NOT all systems ARE yarn-ready
- Are mature distibuted architectures
- Have failover & service discovery mechanisms
- Have delivery and computation guarantees
- Desgned to run on bare metal
- Don't need YARN actually
MESOS
mesos
incubating
FUTURE is here
- New architectural abstractions
- Less deployment and houskeeping
- Programming against unified API
- Failover and service discovery out-of-the-box
- True cross-platform systems
- Any system = distributed system
- Paradigm shift
thank you
be awesome
and
build great software
Massive Parallel Processing Trends
By Anton Kirillov
Massive Parallel Processing Trends
Early draft of overview of contemporary technologies for data processing, computational resources and cluster management: YARN, Mesos, Spark, Storm
