Mesos + Singularity:
PaaS automation for mortals
Gregory Chomatas @gchomatas
PaaS team
120 meters: My shortest travel to a Conference
Thales of Miletus - 624 BC
Those who can, do, the others philosophise... Really?
Miletus
Optionality* with Mesos
Invest in a Mesos-powered PaaS and keep doing what you love most; building your product
* Optionality is the property of asymmetric upside (preferably unlimited) with corresponding limited downside (preferably tiny)
Blogging
SEO
Social Media
CMS
Lead Management
Landing Pages
Calls-to-Action
Marketing Automation
Analytics
CRM Sync
The underlying culture & structure
12-factor apps
.net monolith to microservices
Small, autonomous teams with end-to-end ownership - no ops
High productivity
~100 engineers
800+ components that can be updated/scaled independently
QA: ~400 small to medium AWS machines
PROD: ~750 medium to large AWS machines
Source: Martin Fowler
http://martinfowler.com/bliki/MicroservicePremium.html
1. Develop locally
2. Provision QA aws instance
3. Deploy via local Python script
4. Provision PROD aws instance
5. Deploy via local Python script
6. Repeat 4 & 5 to scale
…
10. Repeat 4&5 at 4am to replace hw
But for how long...
Statically partitioning the cluster is inefficient
The cost of flexibility & asynchronicity
High operational overhead
Poor utilisation & elasticity
Higher rate of failures
Redress the balance with a Mesos-based PaaS
Abstract away machines
Homogenous environment
Scale out in seconds
Centralized deployables registry
Sept 2013: Our First Mesos Cluster
To Boldly go...to Singularity
Singularity: do more with a single scheduler
Great UI & HTTP API
Native Docker Support
Health Checks
Load Balancing API
Log Maintenance
Oct 2013: Start building Singularity
Singularity: do even more...
Security / artifact signature verification
Agent & Rack maintenance
Webhooks
Auto-rollback
Email Notifications
Executor cleanup
Singularity Components
The PaaS Stack
BUILD
DEPLOY
RUN
Jenkins
Orion
Singularity
The Build / Deploy cycle
buildpack runner
S3
The Deployer - Dry run
The Deployer - Deploying
Singularity
Deployable view
Singularity Task view
Singularity file tailing
Singularity Health Checks & Resources
Singularity All Deployables view
Singularity
Cluster Status
Singularity
Cluster Maintenance view
Migration to Mesos - Timeline
Manual Server Provisioning
Server provisioning UI usage
1800+ deployables
~300 deploys / day
10 minutes from git push to production
I want persistent IMAP connections to 200k+ inboxes (Jul 2015)
Stateful Services
Single Process services
Hard coded stationary hosts
Cgroups memory isolation
User resistance
Migration Issues
All eggs in one basket
Mesos / Framework issues (pingback port)
Failures (Zookeeper, Mesos, Singularity)
Cluster Maintenance
Missing features
Operational Issues
Phased rollout of new Kernel, Instance types
Rolling upgrade of instance basic SW with puppet vars
Maintenance
Rolling upgrade of master/agent process with ansible
Local testing on docker cluster
Roll out at infra-QA then product-QA and last to Production cluster
Deploy tools deploy themselves but maintain command line alternative with fabric
Optionality with Mesos@HubSpot
Singularity
Ghidorah - Load Balancers in Mesos
Massive Builds in Mesos
Baragon - Tasks Load Balancer Manager
Mesos Spark Cluster
The sweet spot
Source: Mark Leslie (http://firstround.com/review/The-Arc-of-Company-Life-and-How-to-Prolong-It/)
Invest early in a deploy & build infrastructure
Dedicate 1-2 engineers to experiment on a Mesos powered PaaS
Try Singularity today!
Useful links
Mesos + Singularity: PaaS automation & Sustainable Development Velocity for mortals
By Gregory Chomatas
Mesos + Singularity: PaaS automation & Sustainable Development Velocity for mortals
Singularity is an open source, battle-tested framework for Mesos Clusters. It combines distributed job scheduling and lifecycle automation of resilient microservices. It is ideal for building PaaS infrastructures that can help high growth companies to sustain development velocity by significantly reducing the operational complexity and deployment friction. In this presentation, we share our experiences in developing the HubSpot PaaS / Deploy pipeline and in moving all our QA and production environment to Mesos. Our key goals are: a) to demonstrate that Mesos clusters provide a mature, safe and proven solution for deploying highly available, complex software architectures comprised of thousands of components, b) to prove that Mesos clusters are not only for the big boys and invite medium size companies to follow HubSpot example and start reaping the benefits of Mesos.
