Mesos(phere)

Sfeir Info Days, March 2016

Christoph Meier, Sfeir Benelux

Change Log:

2016-02-09: initial draft
2016-03-04: add mesosphere

Apache Mesos

http://mesos.apache.org

Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.

Written in C++ (10k lines)
Runs on Linux (kernel >= 3.10)
API for C++, Java, Python

→ Create apps (frameworks) running on Mesos

scheduling
isolation (LXC, Docker)
distributed
fault tolerant (Zookeeper)

Summary

solid foundation for distributed apps
resource offers empower frameworks
frameworks can share resources (data locality)

Chronos

http://mesos.github.io/chronos

cron framework for Mesos
schedule jobs
native support for Docker
asynchronous callbacks
jobs dependency chains

Marathon

https://mesosphere.github.io/marathon

init framework for Mesos
run services & keep em alive
native support for Docker
run apps from resources
custom scripts
health checks
resource constraints
dependencies
...

{
    "id": "simple-docker", 
    "container": {
      "docker": {
        "image": "busybox"
      }
    },
    "cmd": "echo hello from docker",
    "cpus": 0.2,
    "mem": 32.0,
    "instances": 2
}

{
  "id": "/docker/registry",
  "instances": 1,
  "cpus": 0.5,
  "mem": 1024.0,
  "disk": 128,
  "container": {
    "docker": {
      "type": "DOCKER",
      "image": "registry:latest",
      "network": "BRIDGE",
      "parameters": [],
      "portMappings": [
        { "containerPort": 5000, "hostPort": 0, "protocol": "tcp", "servicePort": 5000 }
      ]
    },
    "volumes": [
      {
        "hostPath": "/local/path/to/store/packages",
        "containerPath": "/storage",
        "mode": "RW"
      }
    ]
  },
  "env": {
    "SETTINGS_FLAVOR": "local",
    "STORAGE_PATH": "/storage"
  },
  "ports": [ 0 ]
}

{
  "id": "toggle",
  "cmd": "python toggle.py $PORT0",
  "cpus": 0.2,
  "disk": 0.0,
  "healthChecks": [
    {
      "protocol": "HTTP",
      "path": "/health",
      "portIndex": 0,
      "gracePeriodSeconds": 5,
      "intervalSeconds": 10,
      "timeoutSeconds": 10,
      "maxConsecutiveFailures": 3
    }
  ],
  "instances": 2,
  "mem": 32.0,
  "ports": [0],
  "uris": ["http://downloads.mesosphere.com/misc/toggle.tgz"]
}

Summary

meta-framework to run almost anything on Mesos without having to implement your own framework

Aurora

http://aurora.apache.org

alternative to chronos/marathon
Apache Project (powered by Twitter)
Python DSL
more or less same features

import os

hello_world_process = Process(name = 'hello_world', cmdline = 'echo hello world')

hello_world_task = Task(
  resources = Resources(cpu = 0.1, ram = 16 * MB, disk = 16 * MB),
  processes = [hello_world_process])

hello_world_job = Job(
  cluster = 'cluster1',
  role = os.getenv('USER'),
  task = hello_world_task)

jobs = [hello_world_job]

# copy hello_world.py into the local sandbox
install = Process(
  name = 'fetch_package',
  cmdline = 'cp %s . && echo %s && chmod +x hello_world.py' % (pkg_path, pkg_checksum))

# run the script
hello_world = Process(
  name = 'hello_world',
  cmdline = 'python -u hello_world.py')

# describe the task
hello_world_task = SequentialTask(
  processes = [install, hello_world],
  resources = Resources(cpu = 1, ram = 1*MB, disk=8*MB))

jobs = [
  Service(cluster = 'devcluster',
          environment = 'devel',
          role = 'www-data',
          name = 'hello_world',
          task = hello_world_task)
]