MERY on Docker

[2015/11/10] Fashion Tech meetup #1

Yu Yamanaka (@yuurelx)

DevOps engineer at peroli, Inc.

No.1 curation platform for women in Japan !

MERY is running on Docker

of course in production

How many Docker used?

How do you think about Docker?

Awesome!

(but  have a concern about using in production...)

lower performance?

not stabilized yet?

many constraints?

You are right!!

No silver bullet.

This presentation is

our try-and-error story

about migrating to Docker architecture

Topics

  1. Before
  2. After
  3. Troubles

Topics

  1. Before
  2. After
  3. Troubles

Deploy codes & reload server

Clone repository

class ListsController < ApplicationController
  require_login = [:new, :create, :update, :destroy, :click]

  before_filter :noindex, only: [:search, :click, :favorites]
  before_filter :noindex_after_second_page, only: [:index]

  SHOW_NUMBER_OF_ITEMS_PER_PAGE_IN_PC = 50
  SHOW_NUMBER_OF_ITEMS_PER_PAGE_IN_SP = 20
  SHOW_NUMBER_OF_LINKED_LISTS = 5

  def index
    if request.smart_phone?
      side_ranks
    else
      @ranks = DailyRank.weekly_list_rank_from_cache(5) unless ...
    end
    @big_lists = BigList.top_lists unless fragment_exist?(...)
    @top_lists = TopList.including_lists_by_page_and_per_variables(...)

Build app

$ bundle exec cap deploy

Old deployment flow

Create AMI

Manual

It took about an hour per a module...

Issues

  • We cannot delivery features to users everyday
  • We have risks of human error
  • Who ensure sameness between AMI and recipe?

We compared some solutions, and finally chose Docker

Topics

  1. Before
  2. After
  3. Troubles

Concepts

One click deployment

Only merging a PR!

Pull deployment model

DockerHub

Launched by autoscaling job

EC2

EC2

Existing instances

Pull application images from the registry!

Micro Blue-Green deployment

No downtime & Easy rollback

Architecture

Autoscaling flow

Rundeck

Modern

Job Scheduler

Future

  • Staging environments for each branch
  • Using docker images to build dev env (on Mac)
  • Deploy with some orchestration tool
     (Amazon ECS, Kubernetes)

Topics

  1. Before
  2. After
  3. Troubles

Performance issue

Native vs Docker

 (bridge net. mode)

Apache Bench

container

100%

110%

About 10% slower...

1000 reqs

(by 65 sessions)

DB

Native vs Docker

 (host net.)

Apache Bench

container

100%

100%

No overhead!

DB

Sudden death of bridge networking

(From: "http://www.agilegroup.co.jp/technote/docker-network-in-bridge.html")

A bug of Docker?

We decided not to use bridge networking because could not find the cause....

Stabilization of middleware version

Launched by autoscaling job

EC2

EC2

Existing instances

haproxy 1.x

haproxy 1.y

We decided to create an base AMI with required middleware by Packer.

Handling of credentials

DockerHub

source codes without credencials

encrypted credentials with Amazon KMS

(GitHub, Docker, ...)

build an image with fetched credentials

Handling of credentials

...

# Prepare to clone Git repositories
COPY id_rsa /root/.ssh/id_rsa
RUN touch /root/.ssh/known_hosts && \
    ssh-keyscan github.com > /root/.ssh/known_hosts && \
    chmod -R go-rwx /root/.ssh && \

    # build app 

    rm -f id_rsa /root/.ssh/*

...

(A snippet of the Dockefile)

production:
  branch: release
  commands:
    - sudo pip install awscli
    - aws s3 cp s3://path/to/id_rsa ./id_rsa
    - aws s3 cp s3://path/to/dockercfg $HOME/.dockercfg
    - docker build -t example/mery_api:circleci_$CIRCLE_BUILD_NUM .
    - docker push example/mery_api:circleci_$CIRCLE_BUILD_NUM
    - # kick the rundeck job ...

(A snippet of the circle.yml)

Longer CI time

Each has a cache mechanism. 

At default those are incompatible, but you can make that better.

VS

https://circleci.com/docs/docker#caching-docker-layers

1. Before
2. After
3. Troubles

=> You learned why we chose Docker.

=> You learned how we utilize Docker for our infrastructures.

=> You knew some troubles and solutions about Docker in production.

Conclusion

・Migrating to Docker infrastructures is not easy

・But those are running healthy also today

・Since Docker and its ecosystem has great features, they might help your issues

Let's imagine usages of Docker

in your production!

Thank you for your attention!

Yu Yamanaka (@yuurelx)

DevOps engineer at peroli, Inc.

Made with Slides.com