Bamboo & Amazon ECS

Continuous Cloud Integration

What

We want to automate the deployment of newly assembled bits into AWS.

Why

Ensure that QA has a suitable testing environment always available.  We can also use this technique to control production deployments.

How

Combine Bamboo Deployment Plans with Amazon ECS.

Bamboo

Atlassian's Continuous Delivery offering that we use to automate our builds.   We are currently using the 5.9.7 build 5920 - 14 Oct 15 version.

ECS

Amazon's Elastic Container Service.  Automatic Docker container scheduling.  Requires a cluster of EC2 instances.

Basic Sequence

1. Newly pushed code is automatically assembled and tested by Bamboo
2. New artifacts are published to the Docker registry and Artifactory
3. Successfully tested bits trigger a Bamboo Deployment plan.
4. Plan pulls descriptors from Artifactory and pushes them to Amazon
5. ECS replaces old container with the newly minted one
6. Load balancer ensures uninterrupted service during the entire process

Assumptions

• Each project uses both master and development branches
• Each project follows a standard version template
• Each project's primary artifact is a Docker container
• QA does not wish to control when the new bits become visible to the test team

Amazon Environment

Zone A

Zone B

Cluster

EC2

EC2

EC2

EC2

ECS Service Template

{
    "cluster": "${clusterName}",
    "serviceName": "${serviceName}",
    "taskDefinition": "${family}",
    "loadBalancers": [
        {
            "loadBalancerName": "${loadBalancerName}",
            "containerName": "${containerName}",
            "containerPort": 8080
        }
     ],
    "desiredCount": ${desiredCount},
    "clientToken": "${clientToken}",
    "role": "${ecsInstanceRole}",
    "deploymentConfiguration": {
        "maximumPercent": 200,
        "minimumHealthyPercent": 25
    }
}

ECS Task Template

{
    "family": "${family}",
    "containerDefinitions": [
        {
            "name": "${containerName}",
            "image": "${registry}/${imageName}:${imageTag}",
            "memory": 256,
            "portMappings": [
                {
                    "containerPort": 8080,
                    "hostPort": 1234,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "hostname": "${containerName}",
            "dockerLabels": {
                "release": "${imageTag}"
            }
        }
    ]
}

ECS Service Update Template

{
    "cluster": "${clusterName}",
    "service": "${serviceName}",
    "desiredCount": ${desiredCount},
    "taskDefinition": "${family}",
    "deploymentConfiguration": {
        "maximumPercent": 200,
        "minimumHealthyPercent": 25
    }
}

Version Template

• Follow Semantic Versioning Specification
• major.minor.patch.tag
• the tag is calculated based on the branch name
• 1.2.3.RELEASE for the master branch
• 1.2.3.MILESTONE for the development branch
• other branches are ignored
• the template is used for all artifacts, including Docker images and ECS descriptors
• container tag - mycontainer-1.2.3.MILESTONE
• descriptor version - mydescriptor-1.2.3.MILESTON.json

Template Expansion

• Docker container run as a Bamboo Stage
• Uses Bamboo provided values to expand templates
• Publishes resulting descriptors to Artifactory
• hid-generator-1.0.15.RELEASE-task-definition.json
• hid-generator-1.0.15.RELEASE-service-definition.json
• hid-generator-1.0.15.RELEASE-service-update.json

Descriptor Installation

• Run as part of an Environment in a Bamboo Release Plan
• Normally automatically triggered but can be fired off manually
• Installation logic and tools are encapsulated in a Docker container
• Uses Bamboo provided context to move descriptors from Artifactory and into ECS
• Managed by Bamboo's Docker task

Issues

• Python projects do not have a development branch
• Python projects do not use the version template
• Bamboo Deployment Plans do not have the notion of branches (need 2 triggers, one for each branch)
• Installing an ECS descriptor does not guarantee that the container successfully spun up (can probably set up some AWS notifications but might be complex)

Open Questions

• is CD a desired goal?
• is ECS the scheduler we want to use?
• is Hashicorp's Nomad worth investigating?
• do we want to investigate publishing to a "holding area" and install the entire batch at one time? Rundeck, perhaps.
• do we want to explore the notion of an "uber descriptor" that defines an entire collection of containers as a group?
• we could focus more on monitoring to detect if new containers got properly deployed or not