The future of our infrastructure from 3000 feet view

Why do we need to talk about it at all?

Infrastructure affects SLDC and CI/CD pipeline

Manually managing bare metal is hard & time consuming

USA Patriot Act & paranoia

How to manage mixed cluster?


AWS for Test env
Bare Metal for Prod env

How can we have partially on-premise setup?

Potential solutions

Open/Cloud- Stack
Docker Swarm
Apache Mesos & DCOS
Kubernetes
RedHat OpenShift

 

CloudStack & OpenStack

Open/Cloud- Stack

 

OpenStack and CloudStack have many things in common incl. VMs

 

VMs give you enough flexibility by pretty much letting you to emulate whatever machine and run whatever OS on it.


But you pay high price for it - performance penalties.

Also it doesn't allow you to utilize resources fully.

 

Too many software components to be installed and managed*

 

* OpenStack: Nova, Neutron, Cinder, Keystone, Glance, Swift, Horizon, Ceilometer, Magnum

** Apache CloudStack doing better in this sense

Docker Swarm

Docker Swarm

Provides capabilities to run docker cluster.

100% docker API compatible.

Docker Machine can be used to provision docker instances.

Docker Compose can be used to orchestrate several images/nodes.

Relies on external tools for service discovery: zookeeper, etcd, consul.

Provides failover.

Provides capabilities to define constraints.

Pretty low level.

Requires many "manual" steps and custom scripts to be automated.

Tooling around Docker Swarm is still missing.

No web UI out of the box but can be solved by project Shipyard.

CLI

Apache Mesos

Mesos

Mesos is datacenter kernel OS.

Built using proven technologies and in production since 4 years.

Provides ways to manage cluster.

Provides self healing.
CLI.

IaaS

Metal

OnPrem

Mesos

PaaS

SaaS

Provision and manage machines

Build & run frameworks using resources

Deploy and manage applications and services

Provide customer services

Mesos

Mesos consists of master(s), slaves, scheduler, executors, resources and frameworks.

Master

Standby Master

Standby Master

Zookeeper quorum

Task

Slave

Executor

Task

Slave

Executor

...

Scheduler

Mesos

Node

Node

Node

Node

Mesos

Marathon

DCOS

Aurora

K8s

Frameworks

...

Applications

App

App

Mesos

Mesos

Marathon

Mesos

Marathon is a scheduler which provides REST API and Web UI.

Does orchestration and health checks.

Also does service discovery via Marathon-LB (HAProxy)
or Mesos-DNS.

Zookeeper, etcd or consul can be used as well.

Mesos provides resources and labels and Marathon applies constraints to pin apps to instances or colocate them (if needed).

Marathon

Mesos

Marathon

Mesos

Aurora

Marathon alternative.

Main differences are that Aurora has DSL to define state-machine-like orchestration (i.e. deploy 10 instances and in case only 7 was deployed - rollback)

Mesos

DCOS

DCOS being developed by Mesosphere.

Same company behind Marathon.

Provides better UI to Mesos, CLI, package centre, extra features.
Has enterprise version.

Mesos

Default UI

Mesos

DCOS UI

Google Kubernetes (K8s)

K8s

Does pretty much the same as Mesos and implements same idea.

Does not require external scheduler (i.e. Marathon)

Manages cluster of Pods - Mesos Slave aka Node

Provides self-heal

Provides labels

Each Pod gets own IP

Service - LB endpoint for a Pod so it can be accessed from outside.

Very powerful CLI 

Feature rich deployment, rollback and orchestrations strategies.

Container agnostic and can run rkt (Rocket) by CoreOS using Tectonic.

K8s can run on Mesos.

K8s

RedHat OpenShift

OpenShift

Where does Ansible and Terrafrom fits?

Cluster still needs machines so with Terraform we will be able to provision them.

 

And of course any of listed solutions should be installed so thats where Ansible kicks in.

And don't forget about app configuration.

So what to pick?

Bound to docker and love a lot of control? 

Docker Swarm

 

Want to have cluster and have PaaS focus?

Apache Mesos

 

Doesn't care about container tech and want to focus on app orchestration?

Kubernetes

Opened questions

For all of these solutions we would need to have Docker Registry to run somewhere since we wanna pack our applications in containers.

 

How do we wanna configure out apps?
Should we build self contained apps with backed in configuration? 
Or should we have configuration pulled from a server on container start?

 

How do we develop locally? 

Should we run whole cluster?

Should we just run boot2docker and docker containers?

 

Which solution to pick?

Open discussion starts now!

Thank you.

The future of our infrastructure from 3,000 feet view

By Mykhailo Lieibenson

The future of our infrastructure from 3,000 feet view

Some brainstorming ideas for the future of our infrastructure. Input for open discussion.

  • 1,080