adam meghji

Co-Founder and CTO at @universe

Entrepreneur & hacker.

Passionate about DevOps & APIs.

http://djmarmalade.com.

Always inspired!

the universe

tech stack

OVERVIEW

WEB APPS, MOBILE CLIENTS

PUBLIC-FACING MICROSERVICES

INTERNAL MICROSERVICES

DATABASES, PERSISTENCE

CHALLENGES for devs

Different languages:

Ruby, Node, Python

Different API frameworks:

Rails, Sinatra, Express, Flask

Different JS build tools:

Ember v1, Ember CLI, Webpack

THE INEVITABLE QUESTIONS ..

"How can I run the server?"

"How can I run the test suite?"

"How can I setup a blank DB schema?"

"How can I populate my DB with seed data?"

makefiles!

THE INEVITABLE QUESTIONS ..

"How can I run the server?"

make run

"How can I run the test suite?"

make test

"How can I setup a blank DB schema?"

make install

"How can I populate my DB with seed data?"

make data

Makefile

"How can I run the server?"

rails s
npm start
ember server
python web.py
!?!?!?!?

make run

Makefile

"How can I run the test suite?"

rspec spec
npm test
ember test
sniffer
!?!?!?!?

make test

MAKEFILE template

MAKEFILE: NODEJS (MINIMAL)

MAKEFILE: NODEJS (full)

How can I run
the full stack?

FOREMAN

"How can I run the full stack?"

PROCFILE

Example:

foreman start

BUILD PIPELINE

Developer writes code and sends it to Github
CircleCI tests the commit
2nd Developer peer reviews the code, and merges to (master|staging) when ready
2nd Developer issues a deployment via ChatOps: "@uniiverse web production deploy"
Hubot kicks off a deploy on build server
Build Server shells out to custom deploy script: "./launch.rb deploy web production"

DEPLOY SCRIPT

BUILD: Assemble code & artifacts
UPLOAD: Copy build to S3
SYNC: Distribute to EC2 Instances
MIGRATE: Execute data migrations

[1/4]: BUILD

pulls git HEAD
"bundle install --path vendor/bundle" if ruby
"npm install node_modules/" if node
"rake assets:precompile" if rails
<etc. for any supported target language>

[2/4]: UPLOAD

tar -czvf artifacts created in BUILD step
Upload to S3
```
aws s3 cp app.tgz s3://... 
```

[3/4]: SYNC

identify EC2 instances tagged Name=web_production
broadcast a command to initiate the deploy:
```
chef-client -j /etc/chef/first-boot.json
```

[4/4]: MIGRATE

identify 1 random instance tagged Name=web_production

run migration scripts:

docker exec web_production rake db:migrate

platform

INFRASTRUCTURE

EMbracing devops

How can we empower developers to

collaborate on platform infrastructure?

How can we safely issue cloud configuration changes?

How to audit changes as we iterate and evolve?

How do we curate an amazing DevOps culture?

THE OLD WAY ...

console.aws.amazon.com + 1,000,000 mouse clicks

THE NEW WAY ...

terraform!

terraform makefile

TERRAFORM Base.tf

terraform app cluster

USER-data.sh

TL;DR - bootstrap chef & execute role[sidekiq]

APP-specific AMI

ubuntu 14.04 LTS via AWS AM
cookbooks rendered to custom AMI via Packer

[base] include_recipe 'apt'
[base] include_recipe 'apt::unattended-upgrades'
[base] include_recipe 'fail2ban'
[base] include_recipe 'unii-chef'
(deregister from chef.io on shutdown)
[base] include_recipe 'unii-papertrail'
[base] include_recipe 'unii-diamond::install'

[docker] include_recipe 'aufs'
[docker] include_recipe 'awscli'

APP-specific AMI

configure .dockercfg private repo authentication
pull docker uniiverse/web-production
(first time, thus entire image)

write upstart configuration:
/etc/init/web_production.conf

/usr/bin/docker run --rm --name web_production -v /home/ubuntu/deploy/web/production:/home/app/web -w /home/app/web -P -p 80:80 uniiverse/web-production 2>&1 | logger -t api_production

write monit & docker memory health check
(restart if > 75% MEM)

AUTOSCALING SEQUENCE

RE-APPLY COOKBOOK TO DEPLOY

DOCKER

CONTAINERS

Docker containers used to encapsulate an app's runtime environment & all dependencies

For example:

Ruby + bundler
NodeJS + npm
ImageMagick + libraries
supervisord + nginx
private keys + certs
(secrets, passbook, etc)

strategy #1:

thin containers

THIN CONTAINERS

"Container runs a fully-baked app server ENVIRONMENT"

DOES NOT include application code.

DOES NOT include application gems, node_modules, etc.

Instead, app code & vendorized libs reside on host instance's FS,

and exposed to container's process via shared volumes.

Containers are much lighter, and infrequently built.

strategy #2:

container "binaries"

DOCKERFILE: nodejs (MINIMAL)

DOCKERFILE: RAILS (MINIMAL)

DOCKERFILE: RAILS (FULL)

supervisord.conf

METRICS

collector: diamond

Diamond is a python daemon that collects system metrics and publishes them to Graphite. It is capable of collecting cpu, memory, network, i/o, load and disk metrics.

Additionally, it features an API for implementing custom collectors for gathering metrics from almost any source.

CHARTING: GRAFANA

Gorgeous metric viz, dashboards & editors for Graphite, InfluxDB & OpenTSDB

MONITORING:

Pingdom

LOG AGGREGATION:

Papertrail

helpful

devops tools

./ssh.sh

quick CLI to SSH into any instance:

./ssh.sh <app> <environment> [command]

./ssh.sh web staging

./ssh.sh web staging free -m

Resolves EC2 instances by Name tag

Solves the problem of server discovery for remote access

Uses the Jump Server to access the instance

./attach.sh

quick CLI to SSH into any app container:

./attach.sh <app> <environment> [command]

./attach.sh web staging

./attach.sh web staging bundle exec rails console staging

./attach.sh web staging bundle exec rake cache:clear

Uses ./ssh.sh and docker exec

Simplifies connecting to the running container.

Perfect for opening an interactive console, etc.

THE

OUTCOME?

BENEFITS

A consistent way to ship microservices in Containers despite underlying language, framework, or dependencies

Services are multi-region, can auto-scale with traffic, and auto-heal during failure

Developers have 1 way of shipping code, with gory details neatly abstracted. No longer daunting to add a new microservice.

Helpful tools and scripts can be written once and reused everywhere.

AREA OF OPTIMIZATION #1

OPTIMIZE COST:

Requires lots of ELB instances

(2 environments * N microservices)

ONE SOLUTION?

1 ELB & 1 ASG of HAProxy machines

incl. subdomain routing, health detection, etc.

AREA OF OPTIMIZATION #2

OPTIMIZE UTILIZATION:

EC2 instances are single-purpose

(only run 1 docker container)

ONE SOLUTION?

Marathon: execute long-running tasks via Mesos & REST API

All instance CPU & RAM is pooled, and tasks (i.e. containers) are evenly distributed by resource utilization

Kubernetes?

happy hacking :)

https://universe.com

@AdamMeghji

adam@universe.com

DEVOPS OVERVIEW

Universe DevOps Overview

More from adammeghji