omaha-server

High Fidelity, High Velocity Deployments in the Cloud

  • Python
  • Django
  • PostgreSQL
  • Redis

Technology Stack

Workflow

git commit & git push
  • Run tests often
  • Preferably on every commit
  • In a clean environment
  • Pull requests testing
  • Status Images

Hosted Continuous Integration

Minimal Config

language: python

python:
  - "2.7"

virtualenv:
  system_site_packages: true

env:
  global:
    - HOST_NAME: 'travis-ci'
    - SECRET_KEY: 'SECRET_KEY'

services:
  - redis-server

before_install:
  - sudo apt-get install -qq python-lxml python-psycopg2

install: "pip install -r requirements-test.txt --use-mirrors"

script: paver test

after_success:
  - coveralls --verbose

What is Docker

  • At very basic level, think lightweight VM
  • Docker is an open platform for developing, shipping, and running applications.
  • Run any application securely isolated in a container.
  • Open Source engine to commoditize LXC
  • Written in Go, runs as a daemon, comes with a CLI
  • Full REST API
  • Allows to build images in standard, reproducible way
  • Allows to share images

Running a Container

$ docker run -d --name omaha -p 9090:9090 crystalnix/omaha-server
  • Pulls the api image: Docker checks for the presence of the omaha-server image and, if it doesn't exist locally on the host, then Docker downloads it from docker registry. If the image already exists, then Docker uses it for the new container.

  • Creates a new container: Once Docker has the image, it uses it to create a container.

  • Allocates a filesystem and mounts a read-write layer: The container is created in the file system and a read-write layer is added to the image.

  • Allocates a network / bridge interface: Creates a network interface that allows the Docker container to talk to the local host.

  • Sets up an IP address: Finds and attaches an available IP address from a pool.

  • Executes a process that you specify: Runs omaha-server application

  • Captures and provides application output: Connects and logs standard input, outputs and errors for you to see how your application is running.

Managing a Container

$ docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED
a254c3d91bab        omahaserver_web:latest   "paver docker_run pa   Up 5 minutes

STATUS              PORTS                    NAMES
Up 5 minutes        0.0.0.0:9090->80/tcp     omaha

$ docker stop omaha
$ docker start omaha
$ docker restart omaha
$ docker logs omaha

---> pavement.docker_run
---> pavement.migrate
./manage.py migrate --noinput
Operations to perform:
  Synchronize unmigrated apps: django_filters, djangobower, cacheops, django_select2, suit, django_extensions, django_tables2, suit_redactor, rest_framework, storages, versionfield, absolute
  Apply all migrations: crash, omaha, sessions, admin, auth, sparkle, contenttypes
Synchronizing apps without migrations:
  Creating tables...
  Installing custom SQL...
  Installing indexes...
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
---> pavement.loaddata
./manage.py loaddata fixtures/initial_data.json
Installed 5 object(s) from 1 fixture(s)
---> pavement.create_admin
./createadmin.py
---> pavement.collectstatic
./manage.py collectstatic --noinput

0 static files copied to '/srv/omaha/omaha_server/static', 781 unmodified.
---> pavement.mount_s3
s3fs aws_storage_bucket_name /srv/omaha_s3 -ouse_cache=/tmp
/usr/bin/supervisord
/usr/lib/python2.7/dist-packages/supervisor/options.py:295: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
2015-01-19 05:38:24,740 CRIT Supervisor running as root (no user in config file)
2015-01-19 05:38:24,741 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2015-01-19 05:38:24,767 INFO RPC interface 'supervisor' initialized
2015-01-19 05:38:24,767 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-01-19 05:38:24,768 INFO supervisord started with pid 39
2015-01-19 05:38:25,774 INFO spawned: 'celery' with pid 42
2015-01-19 05:38:25,776 INFO spawned: 'nginx-app' with pid 43
2015-01-19 05:38:25,779 INFO spawned: 'omaha' with pid 44
2015-01-19 05:38:26,821 INFO success: nginx-app entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-01-19 05:38:26,822 INFO success: omaha entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-01-19 05:38:36,307 INFO success: celery entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)

Docker Images

  • Docker images are the build component of Docker.
  • A Docker image is a read-only template.
  • Images are used to create Docker containers.
  • Docker provides a simple way to build new images or update existing images, or you can download Docker images that other people have already created.
  • Each image consists of a series of layers. Docker makes use of union file systems to combine these layers into a single image.
  • When you change a Docker image – for example, update an application to a new version – a new layer gets built.

Docker Registries

  • Docker registries are the distribution component of Docker.
  • Docker registries hold images.
  • These are public or private stores from which you upload or download images.
  • The public Docker registry is called Docker Hub. It provides a huge collection of existing images for your use.

Docker Containers

  • Docker containers are the run component of Docker.
  • A Docker container holds everything that is needed for an application to run.
  • Each container is created from a Docker image.
  • The Docker image is read-only. When Docker runs a container from an image, it adds a read-write layer on top of the image in which your application can then run.
  • Docker containers can be run, started, stopped, moved, and deleted.
FROM ubuntu:14.04.1

RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y python-pip python-lxml python-psycopg2 uwsgi supervisor
RUN apt-get install -y uwsgi-plugin-python
RUN apt-get install -y nginx

# install s3fs

RUN apt-get install -y build-essential libfuse-dev libfuse-dev libfuse-dev mime-support
RUN apt-get clean

RUN wget https://github.com/s3fs-fuse/s3fs-fuse/archive/v1.78.tar.gz -O /usr/src/v1.78.tar.gz
RUN tar xvz -C /usr/src -f /usr/src/v1.78.tar.gz
RUN cd /usr/src/s3fs-fuse-1.78 && ./autogen.sh 
RUN ./configure --prefix=/usr && make && make install
RUN mkdir /srv/omaha_s3

ADD . /srv/omaha

# setup all the configfiles
RUN rm /etc/nginx/sites-enabled/default
RUN rm /etc/nginx/nginx.conf
RUN ln -s /srv/omaha/conf/nginx.conf /etc/nginx/
RUN ln -s /srv/omaha/conf/nginx-app.conf /etc/nginx/sites-enabled/
RUN ln -s /srv/omaha/conf/supervisord.conf /etc/supervisor/conf.d/

WORKDIR /srv/omaha
RUN pip install paver --use-mirrors
RUN pip install -r requirements.txt --use-mirrors

EXPOSE 80
ENTRYPOINT ["paver", "docker_run"]
  • Fast, isolated development environments using Docker
  • Quick and easy to start
  • Manages a collection of containers
db:
  image: postgres
redis:
  image: redis
web:
  image: crystalnix/omaha-server
  command: paver docker_run
  working_dir: /srv/omaha
  privileged: true
  links:
   - db
   - redis
  ports:
   - "9090:80"
  environment:
    DB_HOST: db
    DB_PORT: 5432
    DB_USER: postgres
    DB_NAME: postgres
    DB_PASSWORD: ''
    HOST_NAME: '*'
    SECRET_KEY: 'SECRET_KEY'
    DJANGO_SETTINGS_MODULE: 'omaha_server.settings'
    REDIS_HOST: redis
    REDIS_PORT: 6379

Up PostgreSQL + Redis + omaha-server

  • Python
  • Django
  • PostgreSQL
  • Redis

AWS Elastic Beanstalk

[...] quickly deploy and manage applications in the AWS cloud without worrying about the infrastructure that runs those applications. [...] reduces management complexity without restricting choice or control. 

You simply upload your application, and AWS Elastic Beanstalk automatically handles the details of capacity provisioning, load balancing, scaling, and application health monitoring.

Omaha-server

Deployment without Docker

Deployment with Docker

One Command to Deploy

$ ebs-deploy deploy -e omaha-server-dev

ebs-deploy is a command line tool for managing application deployments on Amazon's Beanstalk.

  • Zero downtime deployment
  • Simple config file
  • Update an environment(s)
aws:
    access_key: 'AWS Access Key'
    secret_key: 'AWS Secret Key'
    region: 'us-east-1'
    bucket: 'the bucket that beanstalk apps will be stored in'
    bucket_path: 'omaha-server'
app:
    versions_to_keep: 10
    app_name: 'omaha-server'
    description: 'Omaha Server'
    all_environments:
        solution_stack_name: '64bit Amazon Linux 2014.09 v1.0.9 running Docker 1.2.0'
        tier_name: 'WebServer'
        tier_type: 'Standard'
        tier_version: '1.0'
        option_settings:
            'aws:autoscaling:launchconfiguration':
                InstanceType: 't2.small'
                SecurityGroups: 'omaha_server_dev'
                EC2KeyName: 'Key Name'
            'aws:autoscaling:asg':
                MinSize: 1
                MaxSize: 10
            'aws:autoscaling:trigger':
                BreachDuration: 300
                MeasureName: 'CPUUtilization'
                Unit: 'Percent'
                LowerThreshold: 20
                UpperThreshold: 70
                UpperBreachScaleIncrement: 1
            'aws:elasticbeanstalk:application':
                Application Healthcheck URL: '/admin/login/'
            includes:
                - 'Dockerrun.aws.json'

AWS & Beanstalk

  • Auto scaling
  • Auto balance 
  • Multi regions 
  • Integrado com RDS
  • Monitoring 
  • Full API Managable 

Auto Scaling

Auto Scaling is a web service that enables you to automatically launch or terminate Amazon Elastic Compute Cloud (Amazon EC2) instances based on user-defined policies, health status checks, and schedules.

AWS CloudWatch /Monitoring

The End

Made with Slides.com