Blue-Green Rails

Turn-key ECS with CloudFormation

Codes?

github.com/laser/blue-green-rails

All About Me Me Me

  • From rural Wa(r)shington
  • RPCV Honduras
  • Senior Developer/Reluctant Lead at Carbon Five
  • Love bullet points

Concepts

Docker

  • Containerization technology from Docker, Inc.
  • Abstracts over OS stuff (file system, syscalls, etc.)
  • Multiple containers share single kernel
  • Package your application up with system stuff it depends on
  • Run it on any Docker host

Amazon EC2 Container Service (ECS)

  • Container orchestration service from Amazon which supports Docker containers
  • Deploys, scales, monitors your running containers on AWS

Amazon CloudFormation

  • Provides a language to describe and provision all your AWS infrastructure via a YAML file
  • Interfaces with all the major Amazon Web Services, e.g. RDS, ELB, EC2
  • It's like Terraform

Blue/Green Deploys

  • Green app is the V1
  • Blue app (V2) spins up
  • Load balancer transfer traffic to V2
  • Blue app becomes green

Motivations

Desire

  • I want app sources and system dependencies packaged together

Rationale

  • Self-contained unit of deployment
  • I want dev and prod environments to be as similar as possible

How do I get what I want?

Desire

  • Infrastructure as (declarative) code

Rationale

  • Repeatability
  • Auditability
  • I don't want to click around in some web UI

How do I get what I want?

Desire

  • Quick horizontal and vertical scaling in cluster

Rationale

  • Pay for only as much horsepower as the business needs

Desire

  • Zero-downtime deploys

Rationale

  • No outages while deploying
  • Easy rollback if (when) I screw something up

How do I get what I want?

In This Talk We Will:

  1. Generate and dockerize a Rails app
  2. Push generated Docker image to ECR
  3. Deploy image to newly-provisioned ECS cluster
  4. Discuss some strategies for applying migrations to shared (RDS) database

ECS Cluster of Our Dreams

Dockerizing Rails

Generating a Rails App

$ rvm use ruby-2.5.0@global --create \
  && gem install --no-rdoc --no-ri bundler \
  && gem install --no-rdoc --no-ri rails -v "5.1.2"

Generating a Rails App

$ rvm use ruby-2.5.0@global --create \
  && gem install --no-rdoc --no-ri bundler \
  && gem install --no-rdoc --no-ri rails -v "5.1.2"

Using /Users/erinswenson-healey/.rvm/gems/ruby-2.5.0 with gemset global
Successfully installed bundler-1.16.1
1 gem installed
Successfully installed rails-5.1.2
1 gem installed

Generating a Rails App

$ rails new websvc -d postgresql --skip-yarn --skip-spring \
  && cd websvc \
  && bundle \
  && rails generate scaffold Post title:string content:text

Generating a Rails App

$ rails new websvc -d postgresql --skip-yarn --skip-spring \
  && cd websvc \
  && bundle \
  && rails generate scaffold Post title:string content:text

      create
      create  README.md
      create  Rakefile
      create  config.ru
      create  .gitignore

Generating a Rails App

$ rails new websvc -d postgresql --skip-yarn --skip-spring \
  && cd websvc \
  && bundle \
  && rails generate scaffold Post title:string content:text

      create
      create  README.md
      create  Rakefile
      create  config.ru
      create  .gitignore
    
      [...]

      Using turbolinks 5.1.0
      Using uglifier 4.1.5
      Using web-console 3.5.1
      Bundle complete! 14 Gemfile dependencies, 68 gems now installed.
      Use `bundle info [gemname]` to see where a bundled gem is installed.

Generating a Rails App

$ rails new websvc -d postgresql --skip-yarn --skip-spring \
  && cd websvc \
  && bundle \
  && rails generate scaffold Post title:string content:text

      create
      create  README.md
      create  Rakefile
      create  config.ru
      create  .gitignore
    
      [...]

      Using turbolinks 5.1.0
      Using uglifier 4.1.5
      Using web-console 3.5.1
      Bundle complete! 14 Gemfile dependencies, 68 gems now installed.
      Use `bundle info [gemname]` to see where a bundled gem is installed.

      [...]

      invoke    scss
      create      app/assets/stylesheets/posts.scss
      invoke  scss
      create    app/assets/stylesheets/scaffolds.scss

Create Dockerfile

# ./websvc/Dockerfile

FROM ruby:2.5.0-alpine

RUN apk add --update postgresql-dev alpine-sdk nodejs tzdata

# install gems
COPY Gemfile* /opt/bundle/
WORKDIR /opt/bundle
RUN bundle update && bundle install

# copy app sources
COPY . /opt/app
WORKDIR /opt/app

# define flexible entrypoint
# usage: docker run -it foo:latest "rails migrate && rails server"
ENTRYPOINT ["/bin/ash", "-c"]

Create Dockerfile

# ./websvc/Dockerfile

FROM ruby:2.5.0-alpine

RUN apk add --update postgresql-dev alpine-sdk nodejs tzdata

# install gems
COPY Gemfile* /opt/bundle/
WORKDIR /opt/bundle
RUN bundle update && bundle install

# copy app sources
COPY . /opt/app
WORKDIR /opt/app

# define flexible entrypoint
# usage: docker run -it foo:latest "rails migrate && rails server"
ENTRYPOINT ["/bin/ash", "-c"]

Create Dockerfile

# ./websvc/Dockerfile

FROM ruby:2.5.0-alpine

RUN apk add --update postgresql-dev alpine-sdk nodejs tzdata

# install gems
COPY Gemfile* /opt/bundle/
WORKDIR /opt/bundle
RUN bundle update && bundle install

# copy app sources
COPY . /opt/app
WORKDIR /opt/app

# define flexible entrypoint
# usage: docker run -it foo:latest "rails migrate && rails server"
ENTRYPOINT ["/bin/ash", "-c"]

Create Dockerfile

# ./websvc/Dockerfile

FROM ruby:2.5.0-alpine

RUN apk add --update postgresql-dev alpine-sdk nodejs tzdata

# install gems
COPY Gemfile* /opt/bundle/
WORKDIR /opt/bundle
RUN bundle update && bundle install

# copy app sources
COPY . /opt/app
WORKDIR /opt/app

# define flexible entrypoint
# usage: docker run -it foo:latest "rails migrate && rails server"
ENTRYPOINT ["/bin/ash", "-c"]

Create Dockerfile

# ./websvc/Dockerfile

FROM ruby:2.5.0-alpine

RUN apk add --update postgresql-dev alpine-sdk nodejs tzdata

# install gems
COPY Gemfile* /opt/bundle/
WORKDIR /opt/bundle
RUN bundle update && bundle install

# copy app sources
COPY . /opt/app
WORKDIR /opt/app

# define flexible entrypoint
# usage: docker run -it foo:latest "rails migrate && rails server"
ENTRYPOINT ["/bin/ash", "-c"]

Create Dockerfile

# ./websvc/Dockerfile

FROM ruby:2.5.0-alpine

RUN apk add --update postgresql-dev alpine-sdk nodejs tzdata

# install gems
COPY Gemfile* /opt/bundle/
WORKDIR /opt/bundle
RUN bundle update && bundle install

# copy app sources
COPY . /opt/app
WORKDIR /opt/app

# define flexible entrypoint
# usage: docker run -it foo:latest "rails migrate && rails server"
ENTRYPOINT ["/bin/ash", "-c"]

Building the Image

$ docker build . -t demo:latest

Building the Image

$ docker build . -t demo:latest

Building the Image

$ docker build . -t demo:latest

Sending build context to Docker daemon  140.3kB
Step 1/8 : from ruby:2.5.0-alpine
 ---> 308418a1844f
Step 2/8 : RUN apk add --update postgresql-dev alpine-sdk nodejs tzdata
 ---> Using cache
 ---> e7e55a707ca1
Step 3/8 : COPY Gemfile* /opt/bundle/
 ---> Using cache
 ---> 0942e5df00cf
Step 4/8 : WORKDIR /opt/bundle
 ---> Using cache
 ---> cd6ad4f315fc

Building the Image

$ docker build . -t demo:latest

Sending build context to Docker daemon  140.3kB
Step 1/8 : from ruby:2.5.0-alpine
 ---> 308418a1844f
Step 2/8 : RUN apk add --update postgresql-dev alpine-sdk nodejs tzdata
 ---> Using cache
 ---> e7e55a707ca1
Step 3/8 : COPY Gemfile* /opt/bundle/
 ---> Using cache
 ---> 0942e5df00cf
Step 4/8 : WORKDIR /opt/bundle
 ---> Using cache
 ---> cd6ad4f315fc
Step 5/8 : RUN bundle install
 ---> Using cache
 ---> fd49623c082c
Step 6/8 : COPY . /opt/app
 ---> a1734eb5074d
Step 7/8 : WORKDIR /opt/app
Removing intermediate container 92bf3ce950fa
 ---> ff996b3b426f
Step 8/8 : ENTRYPOINT ["/bin/ash", "-c"]
 ---> Running in 4258e14bcab6
Removing intermediate container 4258e14bcab6
 ---> a89cc7eda27f

Building the Image

$ docker build . -t demo:latest

Sending build context to Docker daemon  140.3kB
Step 1/8 : from ruby:2.5.0-alpine
 ---> 308418a1844f
Step 2/8 : RUN apk add --update postgresql-dev alpine-sdk nodejs tzdata
 ---> Using cache
 ---> e7e55a707ca1
Step 3/8 : COPY Gemfile* /opt/bundle/
 ---> Using cache
 ---> 0942e5df00cf
Step 4/8 : WORKDIR /opt/bundle
 ---> Using cache
 ---> cd6ad4f315fc
Step 5/8 : RUN bundle install
 ---> Using cache
 ---> fd49623c082c
Step 6/8 : COPY . /opt/app
 ---> a1734eb5074d
Step 7/8 : WORKDIR /opt/app
Removing intermediate container 92bf3ce950fa
 ---> ff996b3b426f
Step 8/8 : ENTRYPOINT ["/bin/ash", "-c"]
 ---> Running in 4258e14bcab6
Removing intermediate container 4258e14bcab6
 ---> a89cc7eda27f
Successfully built a89cc7eda27f
Successfully tagged demo:latest

Replace Database Config.

# ./config/database.yml

default: &default
  adapter: postgresql
  encoding: unicode

development:
  <<: *default
  url: <%= ENV['DATABASE_URL'] %>

test:
  <<: *default
  url: <%= ENV['DATABASE_URL'] %>_test

production:
  <<: *default
  url: <%= ENV['DATABASE_URL'] %>

Create a (Local) Database

$ createuser --createdb foouser \
    && createdb --owner foouser foodb
$ docker run -it \
    -v $(pwd):/opt/app \
    -p 3333:3000 \
    -e RAILS_ENV=development \
    -e DATABASE_URL=postgresql://foouser@docker.for.mac.host.internal/foodb \
    demo:latest "rails db:migrate && rails server -b 0.0.0.0"

Running the Container

$ docker run -it \
    -v $(pwd):/opt/app \
    -p 3333:3000 \
    -e RAILS_ENV=development \
    -e DATABASE_URL=postgresql://foouser@docker.for.mac.host.internal/foodb \
    demo:latest "rails db:migrate && rails server -b 0.0.0.0"

Running the Container

$ docker run -it \
    -v $(pwd):/opt/app \
    -p 3333:3000 \
    -e RAILS_ENV=development \
    -e DATABASE_URL=postgresql://foouser@docker.for.mac.host.internal/foodb \
    demo:latest "rails db:migrate && rails server -b 0.0.0.0"

Running the Container

$ docker run -it \
    -v $(pwd):/opt/app \
    -p 3333:3000 \
    -e RAILS_ENV=development \
    -e DATABASE_URL=postgresql://foouser@docker.for.mac.host.internal/foodb \
    demo:latest "rails db:migrate && rails server -b 0.0.0.0"

Running the Container

$ docker run -it \
    -v $(pwd):/opt/app \
    -p 3333:3000 \
    -e RAILS_ENV=development \
    -e DATABASE_URL=postgresql://foouser@docker.for.mac.host.internal/foodb \
    demo:latest "rails db:migrate && rails server -b 0.0.0.0"

Running the Container

Running the Container

$ docker run -it \
    -v $(pwd):/opt/app \
    -p 3333:3000 \
    -e RAILS_ENV=development \
    -e DATABASE_URL=postgresql://foouser@docker.for.mac.host.internal/foodb \
    demo:latest "rails db:migrate && rails server -b 0.0.0.0"

=> Booting Puma
=> Rails 5.1.4 application starting in development
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 3.11.2 (ruby 2.5.0-p0), codename: Love Song
* Min threads: 5, max threads: 5
* Environment: development
* Listening on tcp://0.0.0.0:3000
Use Ctrl-C to stop

Our Docker Container

Open locally-running app

In This Talk We Will:

  1. Generate and dockerize a Rails app
  2. Push generated Docker image to ECR
  3. Deploy image to newly-provisioned ECS cluster
  4. Discuss some strategies for applying database migrations

Create ECR Repository

# image-repository.yml

Resources:
  Repository:
    Type: AWS::ECR::Repository
Outputs:
  RepositoryUri:
    Value:
      Fn::Join:
      - ""
      - - !Ref AWS::AccountId
        - ".dkr.ecr."
        - !Ref AWS::Region
        - ".amazonaws.com/"
        - !Ref Repository

Create ECR Repository

$ aws cloudformation deploy \
  --stack-name demorepo \
  --template-file ./image-repository.yml

Create ECR Repository

$ aws cloudformation deploy \
  --stack-name demorepo \
  --template-file ./image-repository.yml

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - demorepo

Logging in to ECR

$ eval $(aws ecr get-login --no-include-email)

Logging in to ECR

$ eval $(aws ecr get-login --no-include-email)

WARNING! Using --password via the CLI is insecure. Use 
--password-stdin.
Login Succeeded

Tagging and Pushing

$ docker tag demo:latest ${REPOSITORY_URI}:latest \
  && docker push ${REPOSITORY_URI}:latest

Tagging and Pushing

$ docker tag demo:latest ${REPOSITORY_URI}:latest \
  && docker push ${REPOSITORY_URI}:latest

The push refers to repository [166875342547.dkr.ecr.us-east-1.amazonaws.com/demor-repos-1as4v]
9354d709b983: Pushed
d768dd910be3: Pushing [============>                                      ]  24.28MB/98.1MB
488a4eca2037: Pushed
602c1e0aeed9: Pushing [=====>                                             ]   26.3MB/222.3MB
838e7becd078: Pushed
61cb6c204d39: Pushing [=======================>                           ]  26.95MB/56.56MB
e9bcacee1741: Pushed
cd7100a72410: Pushing [==================================================>]  4.403MB

Tagging and Pushing

$ docker tag demo:latest ${REPOSITORY_URI}:latest \
  && docker push ${REPOSITORY_URI}:latest

The push refers to repository [166875342547.dkr.ecr.us-east-1.amazonaws.com/demor-repos-1as4v]
9354d709b983: Pushed
d768dd910be3: Pushed
488a4eca2037: Pushed
602c1e0aeed9: Pushed
838e7becd078: Pushed
61cb6c204d39: Pushed
e9bcacee1741: Pushed
cd7100a72410: Pushed
latest: digest: sha256:d08ab6bbda1aa90bd76a035a8b262b66b21d2c1430cb27a745914fb88cce size: 1995

In This Talk We Will:

  1. Generate and dockerize a Rails app
  2. Push generated Docker image to ECR
  3. Deploy image to newly-provisioned ECS cluster
  4. Discuss some strategies for applying database migrations

ECS Cluster of Our Dreams

Creating RDS DBInstance

# database.yml

Resources:
  Database:
    Type: AWS::RDS::DBInstance
    Properties:
      AllocatedStorage: 5
      BackupRetentionPeriod: 0
      DBInstanceClass: db.t2.micro
      DBName: foodb
      Engine: postgres
      EngineVersion: 9.6.5
      MasterUsername: foouser
      MasterUserPassword: foopassword
      PubliclyAccessible: true
Outputs:
  DatabaseUrl:
    Description: A database connection string
    Value:
      Fn::Join:
      - ""
      - - "postgresql://foouser:foopassword@"
        - !GetAtt Database.Endpoint.Address
        - ":"
        - !GetAtt Database.Endpoint.Port
        - "/foodb"

Creating RDS DBInstance

$ aws cloudformation deploy \
  --stack-name demodb \
  --template-file ./database.yml

Creating RDS DBInstance

$ aws cloudformation deploy \
  --stack-name demodb \
  --template-file ./database.yml

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - demorepo

ECS Cluster of Our Dreams

ECS Cluster of Our Dreams

Cluster Stack

# cluster.yml

Parameters: [...]
Resources:
  WideOpenSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties: [...]
  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties: [...]
  LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties: [...]
  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties: [...]
  Cluster:
    Type: AWS::ECS::Cluster
    Properties: [...]
  Service:
    Type: AWS::ECS::Service
    DependsOn: LoadBalancerListener
    Properties: [...]
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties: [...]
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties: [...]
  TaskExecutionRole:
    Type: AWS::IAM::Role
    Properties: [...]
Outputs:
  LoadBalancerUrl:
    Description: The URL of the NLB
    Value: !GetAtt LoadBalancer.DNSName

Cluster Stack

# cluster.yml

Parameters: [...]
Resources:
  WideOpenSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties: [...]
  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties: [...]
  LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties: [...]
  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties: [...]
  Cluster:
    Type: AWS::ECS::Cluster
    Properties: [...]
  Service:
    Type: AWS::ECS::Service
    DependsOn: LoadBalancerListener
    Properties: [...]
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties: [...]
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties: [...]
  TaskExecutionRole:
    Type: AWS::IAM::Role
    Properties: [...]
Outputs:
  LoadBalancerUrl:
    Description: The URL of the NLB
    Value: !GetAtt LoadBalancer.DNSName

ECS TaskDefinition YAML

# cluster.yml

Parameters:
  DatabaseUrl: { Type: String }
  DockerImage: { Type: String }
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      ContainerDefinitions:
      - Name: !Sub ${AWS::StackName}-container
        Image: !Ref DockerImage
        Command:
        - !Sub "rails db:migrate && rails assets:precompile && rails server -b 0.0.0.0"
        Environment:
        - { Name: RAILS_SERVE_STATIC_FILES, Value: true }
        - { Name: RAILS_LOG_TO_STDOUT, Value: true }
        - { Name: RAILS_ENV, Value: production }
        - { Name: DATABASE_URL, Value: !Ref DatabaseUrl }
        - { Name: SECRET_KEY_BASE, Value: [...] }
        PortMappings:
        - ContainerPort: 3000
        Essential: true
        LogConfiguration: [...]
      Family: !Sub ${AWS::StackName}-task-family
      NetworkMode: awsvpc
      RequiresCompatibilities:
      - FARGATE
      [...]

ECS TaskDefinition YAML

# cluster.yml

Parameters:
  DatabaseUrl: { Type: String }
  DockerImage: { Type: String }
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      ContainerDefinitions:
      - Name: !Sub ${AWS::StackName}-container
        Image: !Ref DockerImage
        Command:
        - !Sub "rails db:migrate && rails assets:precompile && rails server -b 0.0.0.0"
        Environment:
        - { Name: RAILS_SERVE_STATIC_FILES, Value: true }
        - { Name: RAILS_LOG_TO_STDOUT, Value: true }
        - { Name: RAILS_ENV, Value: production }
        - { Name: DATABASE_URL, Value: !Ref DatabaseUrl }
        - { Name: SECRET_KEY_BASE, Value: [...] }
        PortMappings:
        - ContainerPort: 3000
        Essential: true
        LogConfiguration: [...]
      Family: !Sub ${AWS::StackName}-task-family
      NetworkMode: awsvpc
      RequiresCompatibilities:
      - FARGATE
      [...]

ECS TaskDefinition YAML

# cluster.yml

Parameters:
  DatabaseUrl: { Type: String }
  DockerImage: { Type: String }
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      ContainerDefinitions:
      - Name: !Sub ${AWS::StackName}-container
        Image: !Ref DockerImage
        Command:
        - !Sub "rails db:migrate && rails assets:precompile && rails server -b 0.0.0.0"
        Environment:
        - { Name: RAILS_SERVE_STATIC_FILES, Value: true }
        - { Name: RAILS_LOG_TO_STDOUT, Value: true }
        - { Name: RAILS_ENV, Value: production }
        - { Name: DATABASE_URL, Value: !Ref DatabaseUrl }
        - { Name: SECRET_KEY_BASE, Value: [...] }
        PortMappings:
        - ContainerPort: 3000
        Essential: true
        LogConfiguration: [...]
      Family: !Sub ${AWS::StackName}-task-family
      NetworkMode: awsvpc
      RequiresCompatibilities:
      - FARGATE
      [...]

ECS TaskDefinition YAML

# cluster.yml

Parameters:
  DatabaseUrl: { Type: String }
  DockerImage: { Type: String }
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      ContainerDefinitions:
      - Name: !Sub ${AWS::StackName}-container
        Image: !Ref DockerImage
        Command:
        - !Sub "rails db:migrate && rails assets:precompile && rails server -b 0.0.0.0"
        Environment:
        - { Name: RAILS_SERVE_STATIC_FILES, Value: true }
        - { Name: RAILS_LOG_TO_STDOUT, Value: true }
        - { Name: RAILS_ENV, Value: production }
        - { Name: DATABASE_URL, Value: !Ref DatabaseUrl }
        - { Name: SECRET_KEY_BASE, Value: [...] }
        PortMappings:
        - ContainerPort: 3000
        Essential: true
        LogConfiguration: [...]
      Family: !Sub ${AWS::StackName}-task-family
      NetworkMode: awsvpc
      RequiresCompatibilities:
      - FARGATE
      [...]

ECS TaskDefinition YAML

# cluster.yml

Parameters:
  DatabaseUrl: { Type: String }
  DockerImage: { Type: String }
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      ContainerDefinitions:
      - Name: !Sub ${AWS::StackName}-container
        Image: !Ref DockerImage
        Command:
        - !Sub "rails db:migrate && rails assets:precompile && rails server -b 0.0.0.0"
        Environment:
        - { Name: RAILS_SERVE_STATIC_FILES, Value: true }
        - { Name: RAILS_LOG_TO_STDOUT, Value: true }
        - { Name: RAILS_ENV, Value: production }
        - { Name: DATABASE_URL, Value: !Ref DatabaseUrl }
        - { Name: SECRET_KEY_BASE, Value: [...] }
        PortMappings:
        - ContainerPort: 3000
        Essential: true
        LogConfiguration: [...]
      Family: !Sub ${AWS::StackName}-task-family
      NetworkMode: awsvpc
      RequiresCompatibilities:
      - FARGATE
      [...]

ECS TaskDefinition YAML

# cluster.yml

Parameters:
  DatabaseUrl: { Type: String }
  DockerImage: { Type: String }
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      ContainerDefinitions:
      - Name: !Sub ${AWS::StackName}-container
        Image: !Ref DockerImage
        Command:
        - !Sub "rails db:migrate && rails assets:precompile && rails server -b 0.0.0.0"
        Environment:
        - { Name: RAILS_SERVE_STATIC_FILES, Value: true }
        - { Name: RAILS_LOG_TO_STDOUT, Value: true }
        - { Name: RAILS_ENV, Value: production }
        - { Name: DATABASE_URL, Value: !Ref DatabaseUrl }
        - { Name: SECRET_KEY_BASE, Value: [...] }
        PortMappings:
        - ContainerPort: 3000
        Essential: true
        LogConfiguration: [...]
      Family: !Sub ${AWS::StackName}-task-family
      NetworkMode: awsvpc
      RequiresCompatibilities:
      - FARGATE
      [...]

ECS TaskDefinition YAML

# cluster.yml

Parameters:
  DatabaseUrl: { Type: String }
  DockerImage: { Type: String }
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      ContainerDefinitions:
      - Name: !Sub ${AWS::StackName}-container
        Image: !Ref DockerImage
        Command:
        - !Sub "rails db:migrate && rails assets:precompile && rails server -b 0.0.0.0"
        Environment:
        - { Name: RAILS_SERVE_STATIC_FILES, Value: true }
        - { Name: RAILS_LOG_TO_STDOUT, Value: true }
        - { Name: RAILS_ENV, Value: production }
        - { Name: DATABASE_URL, Value: !Ref DatabaseUrl }
        - { Name: SECRET_KEY_BASE, Value: [...] }
        PortMappings:
        - ContainerPort: 3000
        Essential: true
        LogConfiguration: [...]
      Family: !Sub ${AWS::StackName}-task-family
      NetworkMode: awsvpc
      RequiresCompatibilities:
      - FARGATE
      [...]

ECS Service YAML

# cluster.yml

Resources:
  Service:
    Type: AWS::ECS::Service
    DependsOn: LoadBalancerListener
    Properties:
      Cluster: !Ref Cluster
      LaunchType: FARGATE
      DesiredCount: 4
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100
      TaskDefinition: !Ref TaskDefinition
      NetworkConfiguration:
        AwsvpcConfiguration:
          Subnets: !Ref PublicSubnets
          AssignPublicIp: ENABLED
          SecurityGroups:
          - !Ref WideOpenSecurityGroup
      LoadBalancers:
      - ContainerName: !Sub ${AWS::StackName}-container
        ContainerPort: 3000
        TargetGroupArn: !Ref TargetGroup

ECS Service YAML

# cluster.yml

Resources:
  Service:
    Type: AWS::ECS::Service
    DependsOn: LoadBalancerListener
    Properties:
      Cluster: !Ref Cluster
      LaunchType: FARGATE
      DesiredCount: 4
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100
      TaskDefinition: !Ref TaskDefinition
      NetworkConfiguration:
        AwsvpcConfiguration:
          Subnets: !Ref PublicSubnets
          AssignPublicIp: ENABLED
          SecurityGroups:
          - !Ref WideOpenSecurityGroup
      LoadBalancers:
      - ContainerName: !Sub ${AWS::StackName}-container
        ContainerPort: 3000
        TargetGroupArn: !Ref TargetGroup

ECS Service YAML

# cluster.yml

Resources:
  Service:
    Type: AWS::ECS::Service
    DependsOn: LoadBalancerListener
    Properties:
      Cluster: !Ref Cluster
      LaunchType: FARGATE
      DesiredCount: 4
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100
      TaskDefinition: !Ref TaskDefinition
      NetworkConfiguration:
        AwsvpcConfiguration:
          Subnets: !Ref PublicSubnets
          AssignPublicIp: ENABLED
          SecurityGroups:
          - !Ref WideOpenSecurityGroup
      LoadBalancers:
      - ContainerName: !Sub ${AWS::StackName}-container
        ContainerPort: 3000
        TargetGroupArn: !Ref TargetGroup

ECS Service YAML

# cluster.yml

Resources:
  Service:
    Type: AWS::ECS::Service
    DependsOn: LoadBalancerListener
    Properties:
      Cluster: !Ref Cluster
      LaunchType: FARGATE
      DesiredCount: 4
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100
      TaskDefinition: !Ref TaskDefinition
      NetworkConfiguration:
        AwsvpcConfiguration:
          Subnets: !Ref PublicSubnets
          AssignPublicIp: ENABLED
          SecurityGroups:
          - !Ref WideOpenSecurityGroup
      LoadBalancers:
      - ContainerName: !Sub ${AWS::StackName}-container
        ContainerPort: 3000
        TargetGroupArn: !Ref TargetGroup

ECS Cluster YAML

# cluster.yml

Resources:
  Cluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Sub ${AWS::StackName}-cluster

Deploying the ECS Stack

$ aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --stack-name democluster \
  --template-file ./cluster.yml \
  --parameter-overrides \
    RailsSecretBase=$(rails secret) \
    DatabaseUrl=${DATABASE_URL} \
    VpcId=${VPC_ID} \
    PublicSubnets=${PUBLIC_SUBNETS} \
    DockerImage=${REPOSITORY_URI}:latest

Deploying the ECS Stack

$ aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --stack-name democluster \
  --template-file ./cluster.yml \
  --parameter-overrides \
    RailsSecretBase=$(rails secret) \
    DatabaseUrl=${DATABASE_URL} \
    VpcId=${VPC_ID} \
    PublicSubnets=${PUBLIC_SUBNETS} \
    DockerImage=${REPOSITORY_URI}:latest

Deploying the ECS Stack

$ aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --stack-name democluster \
  --template-file ./cluster.yml \
  --parameter-overrides \
    RailsSecretBase=$(rails secret) \
    DatabaseUrl=${DATABASE_URL} \
    VpcId=${VPC_ID} \
    PublicSubnets=${PUBLIC_SUBNETS} \
    DockerImage=${REPOSITORY_URI}:latest

Deploying the ECS Stack

$ aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --stack-name democluster \
  --template-file ./cluster.yml \
  --parameter-overrides \
    RailsSecretBase=$(rails secret) \
    DatabaseUrl=${DATABASE_URL} \
    VpcId=${VPC_ID} \
    PublicSubnets=${PUBLIC_SUBNETS} \
    DockerImage=${REPOSITORY_URI}:latest

Deploying the ECS Stack

$ aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --stack-name democluster \
  --template-file ./cluster.yml \
  --parameter-overrides \
    RailsSecretBase=$(rails secret) \
    DatabaseUrl=${DATABASE_URL} \
    VpcId=${VPC_ID} \
    PublicSubnets=${PUBLIC_SUBNETS} \
    DockerImage=${REPOSITORY_URI}:latest

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - demorepo

Deploying the ECS Stack

$ aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --stack-name democluster \
  --template-file ./cluster.yml \
  --parameter-overrides \
    RailsSecretBase=$(rails secret) \
    DatabaseUrl=${DATABASE_URL} \
    VpcId=${VPC_ID} \
    PublicSubnets=${PUBLIC_SUBNETS} \
    DockerImage=${REPOSITORY_URI}:latest

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - demorepo

export LB_URL=$($(aws cloudformation describe-stacks \
  --stack-name democluster \
  | jq -r '(.Stacks[0].Outputs[] \
    | select(.OutputKey == "LoadBalancerUrl")).OutputValue'))

Deploying the ECS Stack

$ aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --stack-name democluster \
  --template-file ./cluster.yml \
  --parameter-overrides \
    RailsSecretBase=$(rails secret) \
    DatabaseUrl=${DATABASE_URL} \
    VpcId=${VPC_ID} \
    PublicSubnets=${PUBLIC_SUBNETS} \
    DockerImage=${REPOSITORY_URI}:latest

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - demorepo

export LB_URL=$($(aws cloudformation describe-stacks \
  --stack-name democluster \
  | jq -r '(.Stacks[0].Outputs[] \
    | select(.OutputKey == "LoadBalancerUrl")).OutputValue'))

open http://${LB_URL}/posts
Open app

Visualizing a Blue/Green Deploy

Initial State

Service:
  # ...
  DeploymentConfiguration:
    MaximumPercent: 200
    MinimumHealthyPercent: 100
  Properties:
    DesiredCount: 4
    # ...

Service Update Initiated

Service:
  # ...
  DeploymentConfiguration:
    MaximumPercent: 200
    MinimumHealthyPercent: 100
  Properties:
    DesiredCount: 4
    # ...

ELB Drain/Reroute

Service:
  # ...
  DeploymentConfiguration:
    MaximumPercent: 200
    MinimumHealthyPercent: 100
  Properties:
    DesiredCount: 4
    # ...

Shut Down Old Tasks

Service:
  # ...
  DeploymentConfiguration:
    MaximumPercent: 200
    MinimumHealthyPercent: 100
  Properties:
    DesiredCount: 4
    # ...

Service Update Complete

Service:
  # ...
  DeploymentConfiguration:
    MaximumPercent: 200
    MinimumHealthyPercent: 100
  Properties:
    DesiredCount: 4
    # ...

Making a Change

<p id="notice"><%= notice %></p>
<h1>Posts</h1>
<table>
  <thead>
    <tr>
      <th>Title</th>
      <th>Content</th>
      <th colspan="3"></th>
    </tr>
  </thead>
  <tbody>
    <% @posts.each do |post| %>
      <tr>
        <td><%= post.title %></td>
        <td><%= post.content %></td>
        <td><%= link_to 'Show', post %></td>
        <td><%= link_to 'Edit', edit_post_path(post) %></td>
        <td><%= link_to 'Destroy', post, method: :delete, data: { confirm: 'Are you sure?' } %></td>
      </tr>
    <% end %>
  </tbody>
</table>
<br>
<%= link_to 'New Post', new_post_path %>

Making a Change

<p id="notice"><%= notice %></p>
<h1>MEOW MEOW MEOW MEOW MEOW</h1>
<table>
  <thead>
    <tr>
      <th>Title</th>
      <th>Content</th>
      <th colspan="3"></th>
    </tr>
  </thead>
  <tbody>
    <% @posts.each do |post| %>
      <tr>
        <td><%= post.title %></td>
        <td><%= post.content %></td>
        <td><%= link_to 'Show', post %></td>
        <td><%= link_to 'Edit', edit_post_path(post) %></td>
        <td><%= link_to 'Destroy', post, method: :delete, data: { confirm: 'Are you sure?' } %></td>
      </tr>
    <% end %>
  </tbody>
</table>
<br>
<%= link_to 'New Post', new_post_path %>

Commit, Build, Tag, Push

$ docker build . -t demo:latest \
  && docker tag demo:latest ${REPOSITORY_URI}:latest \
  && docker push ${REPOSITORY_URI}:latest

Commit, Build, Tag, Push

$ docker build . -t demo:latest \
  && docker tag demo:latest ${REPOSITORY_URI}:latest \
  && docker push ${REPOSITORY_URI}:latest

Sending build context to Docker daemon  137.2kB
Step 1/8 : FROM ruby:2.5.0-alpine
 ---> 308418a1844f
Step 2/8 : RUN apk add --update   alpine-sdk   nodejs   postgresql-dev   tzdata
 ---> Using cache
 ---> 8ed21fdcffa9
[...]
Successfully built 33d23bb7b314

Commit, Build, Tag, Push

$ docker build . -t demo:latest \
  && docker tag demo:latest ${REPOSITORY_URI}:latest \
  && docker push ${REPOSITORY_URI}:latest

Sending build context to Docker daemon  137.2kB
Step 1/8 : FROM ruby:2.5.0-alpine
 ---> 308418a1844f
Step 2/8 : RUN apk add --update   alpine-sdk   nodejs   postgresql-dev   tzdata
 ---> Using cache
 ---> 8ed21fdcffa9
[...]
Successfully built 33d23bb7b314

Successfully tagged demo:latest

Commit, Build, Tag, Push

$ docker build . -t demo:latest \
  && docker tag demo:latest ${REPOSITORY_URI}:latest \
  && docker push ${REPOSITORY_URI}:latest

Sending build context to Docker daemon  137.2kB
Step 1/8 : FROM ruby:2.5.0-alpine
 ---> 308418a1844f
Step 2/8 : RUN apk add --update   alpine-sdk   nodejs   postgresql-dev   tzdata
 ---> Using cache
 ---> 8ed21fdcffa9
[...]
Successfully built 33d23bb7b314

Successfully tagged demo:latest

The push refers to repository [166875342547.dkr.ecr.us-east-1.amazonaws.com/demor-repos-1as4v]
ed2e9d4c2ebf: Pushed
3736f3faece1: Layer already exists
960106cd0a97: Layer already exists
34f06064af2b: Layer already exists
838e7becd078: Layer already exists
61cb6c204d39: Layer already exists
e9bcacee1741: Layer already exists
cd7100a72410: Layer already exists
latest: digest: sha256:ee0e5e23725d8050b6499e4bd6bd783cbf8b2436629489e2f7a322e3d63 size: 1995

Force Service Redeploy

$ aws ecs update-service \
  --service democluster-service \
  --cluster democluster-cluster \
  --force-new-deployment

Force Service Redeploy

$ while true; do \
      echo $(date)-$(curl -s ${POSTS_URL} | grep \<h1); \
      sleep 2; \
  done;

Wed Jan 31 16:16:37 PST 2018-<h1>Posts</h1>
Wed Jan 31 16:16:40 PST 2018-<h1>Posts</h1>
Wed Jan 31 16:16:42 PST 2018-<h1>Posts</h1>
Wed Jan 31 16:16:44 PST 2018-<h1>Posts</h1>
Wed Jan 31 16:16:50 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:16:52 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:16:55 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:16:57 PST 2018-<h1>Posts</h1>
Wed Jan 31 16:16:59 PST 2018-<h1>Posts</h1>
Wed Jan 31 16:17:02 PST 2018-<h1>Posts</h1>
Wed Jan 31 16:17:04 PST 2018-<h1>Posts</h1>
Wed Jan 31 16:17:06 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:17:08 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:17:11 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:17:13 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:17:16 PST 2018-<h1>Posts</h1>
Wed Jan 31 16:17:18 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:17:20 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:17:26 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:17:28 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>
Wed Jan 31 16:17:30 PST 2018-<h1>MEOW MEOW MEOW MEOW</h1>

Live Demo!

Live Demo!

what could possibly go wrong?

In This Talk We Will:

  1. Generate and dockerize a Rails app
  2. Push generated Docker image to ECR
  3. Deploy image to newly-provisioned ECS cluster
  4. Discuss some strategies for applying database migrations

Option 1: Manual Migrations

Option 1: Manual Migrations

Option 1: Manual Migrations

Option 1: Manual Migrations

Option 1: Manual Migrations

Option 1: Manual Migrations

Option 1: Manual Migrations

Option 2: Migrate on Start

# cluster.yml

Parameters:
  DatabaseUrl: { Type: String }
  DockerImage: { Type: String }
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      ContainerDefinitions:
      - Name: !Sub ${AWS::StackName}-container
        Image: !Ref DockerImage
        Command:
        - !Sub "rails db:migrate && rails assets:precompile && rails server -b 0.0.0.0"
        Environment:
        - { Name: RAILS_SERVE_STATIC_FILES, Value: true }
        - { Name: RAILS_LOG_TO_STDOUT, Value: true }
        - { Name: RAILS_ENV, Value: production }
        - { Name: DATABASE_URL, Value: !Ref DatabaseUrl }
        - { Name: SECRET_KEY_BASE, Value: [...] }
        PortMappings:
        - ContainerPort: 3000
        Essential: true
        LogConfiguration: [...]
      Family: !Sub ${AWS::StackName}-task-family
      NetworkMode: awsvpc
      RequiresCompatibilities:
      - FARGATE
      [...]

Option 2: Migrate on Start

# cluster.yml

Parameters:
  DatabaseUrl: { Type: String }
  DockerImage: { Type: String }
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: 256
      Memory: 512
      ContainerDefinitions:
      - Name: !Sub ${AWS::StackName}-container
        Image: !Ref DockerImage
        Command:
        - !Sub "rails db:migrate && rails assets:precompile && rails server -b 0.0.0.0"
        Environment:
        - { Name: RAILS_SERVE_STATIC_FILES, Value: true }
        - { Name: RAILS_LOG_TO_STDOUT, Value: true }
        - { Name: RAILS_ENV, Value: production }
        - { Name: DATABASE_URL, Value: !Ref DatabaseUrl }
        - { Name: SECRET_KEY_BASE, Value: [...] }
        PortMappings:
        - ContainerPort: 3000
        Essential: true
        LogConfiguration: [...]
      Family: !Sub ${AWS::StackName}-task-family
      NetworkMode: awsvpc
      RequiresCompatibilities:
      - FARGATE
      [...]

Option 2: Migrate on Start

Option 2: Migrate on Start

Option 2: Migrate on Start

Option 2: Migrate on Start

Option 2: Migrate on Start

Option 2: Migrate on Start

Option 2: Migrate on Start

Option 2: Migrate on Start

Option 3: RunTask API

MIGRATION_TASK_ARN=$(aws ecs run-task \
    --task-definition migrations \
    --cluster ${CLUSTER_NAME} \
    --count 1 \
    --started-by deploy | jq -r '.["tasks"][0]["taskArn"]')

# waits 600 seconds
aws ecs wait tasks-stopped \
    --cluster ${CLUSTER_NAME} \
    --tasks ${MIGRATION_TASK_ARN}

MIGRATION_EXIT_CODE=$(aws ecs describe-tasks \
    --cluster cluster \
    --tasks ${MIGRATION_TASK_ARN} | jq -r '.["tasks"][0]["containers"][0]["exitCode"]')

if [ "${MIGRATION_EXIT_CODE}" != "0" ] ; then
    # fail build
    exit 1
fi

Option 3: RunTask API

Option 3: RunTask API

Option 3: RunTask API

Option 3: RunTask API

Option 3: RunTask API

Option 3: RunTask API

Option 3: RunTask API

Miscellaneous: Logging

$ awslogs get democluster ALL -s1d # all logs from group in last 24 hours

Miscellaneous: Logging

$ awslogs get democluster ALL -s1d # all logs from group in last 24 hours

democluster ecs/democluster-container/733eb314-0aaa-4ad0-a849-9c357b045aac 172.31.87.245 - - [01/Feb/2018:00:18:45 UTC] "GET /favicon.ico HTTP/1.1" 200 0
democluster ecs/democluster-container/733eb314-0aaa-4ad0-a849-9c357b045aac http://54.86.202.26/game/go/ttlz/index.html -> /favicon.ico
democluster ecs/democluster-container/32ed9165-8a8f-4e6b-b8b0-1f9b0c4532f2 172.31.8.236 - - [01/Feb/2018:00:18:46 UTC] "GET /posts HTTP/1.1" 200 847
democluster ecs/democluster-container/32ed9165-8a8f-4e6b-b8b0-1f9b0c4532f2 - -> /posts
democluster ecs/democluster-container/0a21e832-5640-4e16-af32-cd600c524f62 172.31.8.236 - - [01/Feb/2018:00:18:48 UTC] "GET /posts HTTP/1.1" 200 847
democluster ecs/democluster-container/0a21e832-5640-4e16-af32-cd600c524f62 - -> /posts

Miscellaneous: Logging

$ awslogs get democluster --watch # tail all log streams across cluster

Miscellaneous: Logging

$ awslogs get democluster --watch # tail all log streams across cluster

democluster ecs/democluster-container/79f6fe46-31e0-4e37-8f09-9b00ca19bf6d I, [2018-02-01T00:18:09.697899 #42]  INFO -- : [c16ea66d-a7a1-429b-b706-c771f5ca8eee]   Rendered posts/index.html.erb within layouts/application (2.7ms)

Miscellaneous: Logging

$ awslogs get democluster --watch # tail all log streams across cluster

democluster ecs/democluster-container/79f6fe46-31e0-4e37-8f09-9b00ca19bf6d I, [2018-02-01T00:18:09.697899 #42]  INFO -- : [c16ea66d-a7a1-429b-b706-c771f5ca8eee]   Rendered posts/index.html.erb within layouts/application (2.7ms)
democluster ecs/democluster-container/79f6fe46-31e0-4e37-8f09-9b00ca19bf6d I, [2018-02-01T00:18:09.698349 #42]  INFO -- : [c16ea66d-a7a1-429b-b706-c771f5ca8eee] Completed 200 OK in 4ms (Views: 2.4ms | ActiveRecord: 1.1ms)

Miscellaneous: Logging

$ awslogs get democluster --watch # tail all log streams across cluster

democluster ecs/democluster-container/79f6fe46-31e0-4e37-8f09-9b00ca19bf6d I, [2018-02-01T00:18:09.697899 #42]  INFO -- : [c16ea66d-a7a1-429b-b706-c771f5ca8eee]   Rendered posts/index.html.erb within layouts/application (2.7ms)
democluster ecs/democluster-container/79f6fe46-31e0-4e37-8f09-9b00ca19bf6d I, [2018-02-01T00:18:09.698349 #42]  INFO -- : [c16ea66d-a7a1-429b-b706-c771f5ca8eee] Completed 200 OK in 4ms (Views: 2.4ms | ActiveRecord: 1.1ms)
democluster ecs/democluster-container/79f6fe46-31e0-4e37-8f09-9b00ca19bf6d I, [2018-02-01T00:18:50.419423 #42]  INFO -- : [a2213956-a4ce-4a50-90da-33dc39029475] Started GET "/game/go/jsmt/index.html" for 172.31.87.245 at 2018-02-01 00:18:50 +0000

A Final Note

  • CloudFormation for more than RDS and ECS
  • Reference architecture provides single-command provisioning of:
    • Virtual private cloud
    • Database
    • App cluster
    • CI/CD pipeline + GitHub integration
    • Logging
    • moar!

tinyurl.com/fargate-magic

Blue/Green Rails

Turn-key ECS with CloudFormation

GitHub: laser

Email: erin@carbonfive.com
Fax: 206-867-5309

Blue/Green Rails - LA Ruby on Rails - Feb. 8, 2018

By laser

Made with Slides.com

Blue/Green Rails - LA Ruby on Rails - Feb. 8, 2018

  • 36

More from laser

  • CIS 194: Intro. to Haskell (May 5, 2022)
    laser
    29
  • 2019 C5 Summit: TCP/IP Basics
    laser
    56
  • CIS 194: Intro. to Haskell (July 10, 2018)
    laser
    27