MesosCon 2016

Report Back: June 1-2, Denver, CO

Summary

  • At a Glance
  • What is Apache Mesos? Why is it useful?
  • Presentations
  • Conference Themes
  • Conversations
  • Q+A

MesosCon: At a Glance

  • # of presentations
    • 1​54 submissions, 48 selected
  • # of attendees
    • ​+800
  • Companies and organizations present
    • Twitter, Apple, Airbnb, Verizon Labs, Netflix, Uber, Confluent, Mesosphere, Databricks, CERN, IBM, EMC, Bloomberg, Cisco, Intel, DigitalOcean, SignalFX, Huawei, Microsoft, Adobe, PayPal, Tendril, Yammer, Oracle, Yellow Pages, Activision, etc.

What is Apache Mesos?

Mesos is a DevOps tool for treating CPU, memory, and storage across data centers as a single managed pool of resources shared among applications

More technically speaking: Mesos is an open source distributed systems kernel that raises the abstract-ion level away from individual machines to clusters of machines as a whole and provides isolated frame-works for heterogeneous applications to run across the cluster. It is written in C/C++ and built on stout, libprocess, leveldb, Linux cgroups and namespaces.

Mesos Architecture

mesos-master

mesos-agent 1

framework 2

mesos-agent 2

framework 2

Allocation Module

App 1

App 2

framework scheduler

Job 1

Job 2

framework scheduler

Executor

Task

Task

Executor

Task

Task

1. Resource Availability

2. Resource Offer

3. Task Info

4. Resource Allocation

Why is Mesos useful?

  • Scalability - Industry proven to easily scale to 10,000s of nodes.
  • High Availability - Fault-tolerant replicated masters and agents using ZooKeeper for multi-master leader election. Non-disruptive upgrades.
  • Container Deployment - Native support for launching Docker and AppC images
  • Pluggable Isolation - First class isolation support for CPU, memory, disk, ports, GPU and modules for custom resource isolation
  • Two-Level Scheduling - Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them
  • Cross Platform - Runs on Linux, Mac OSX, and Windows. Cloud provider agnostic. Cloud native or on-premises. Frameworks can be written in C/C++, Python, or Java.
  • Increased Utilitization* - Run fewer machines for the same work load, near-optimal data locality, and co-locate disparate applications on same node

Increased Utilization

Hindman, et al. "Mesos: A Platform for Fine-Grained Resource Sharing in the Datacenter". (2010)

Increased Utilization

Hindman, et al. "Mesos: A Platform for Fine-Grained Resource Sharing in the Datacenter". (2010)

Conference Themes

Storage & Analytics

Cluster Management

Microservices

Containerization

Presentations

(Just the ones I attended)

Wed, June 1

  • Keynotes
  • Contributing to Apache Mesos: Where to Begin (Mesosphere)
  • Apache Mesos on Windows (Microsoft)
  • Lessons fro Mesos Clusters (Netflix)
  • Running Cassandra on Apache Mesos Across Multiple Datacenters (Uber)
  • A Seamless Monitoring Systems for Apache Mesos Clusters (Bloomberg)
  • CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos (CloudBees)
  • Spark on Apache Mesos: The State of the Art (Databricks)

Thurs, June 2

  • Keynotes
  • It's Complicated, Okay: or Let's Talk Openly about Apache Mesos', Neighbors, Friends, and Rivals (Mesosphere)
  • GPU Support in Apache Mesos (Mesosphere)
  • Resource Allocation in Apache Mesos: Present and Future (Mesosophere)
  • A Declarative Approach to Building Stateful Frameworks (Mesosophere)
  • How to Stand Up a 600 Node Bare Metal Mesos Cluster in Two Weeks (Verizon Labs)
  • Lessons Learned from Running a Heterogeneous Workload on Apache Mesos (Yellow Pages)
  • Persistence Primitives in Action 2.0 (ArrangoDB, Mesosphere)

Conversations

  • Invited Prof. Ken Birman to visit and give a talk at SolarCity (need to follow up with him)
  • Creating a DevOps-centric culture within your organization
  • Stateless vs Stateful Containers
  • Running the JVM inside a container
  • Golang GC vs Java GC
  • Neat tools for C/C++ development
  • Hybrid data centers spanning Linux and Windows host machines
  • Met this one guy from SF working at Sysdig who complained that lunch should've been served at 11a instead of 12noon -- his team is building some cool Mesos monitoring tools apparently

Q + A