resource and cluster management

YARN



YARN basics










TEZ



Stinger.next



SPARK



SPARK: DAG & RDD

dataflow topologies

storm

NOT all systems ARE yarn-ready


  • Are mature distibuted architectures
  • Have failover & service discovery mechanisms
  • Have delivery and computation guarantees
  • Desgned to run on bare metal
  • Don't need YARN actually


So far so good, but what about hardware failures? 


MESOS

mesos



incubating

FUTURE is here


  • New architectural abstractions
  • Less deployment and houskeeping
  • Programming against unified API
  • Failover and service discovery out-of-the-box
  • True cross-platform systems
  • Any system = distributed system
  • Paradigm shift

thank you




be awesome 

and 

build great software

Massive Parallel Processing Trends

By Anton Kirillov

Massive Parallel Processing Trends

Early draft of overview of contemporary technologies for data processing, computational resources and cluster management: YARN, Mesos, Spark, Storm

  • 988
Loading comments...

More from Anton Kirillov