Managing CloudFormation at Scale

Deploying CloudFormation stacks at scale

What is IaC?

• IaC: Infrastructure as a Code.
   
• IaC is a technique used to specify desired state of the infrastructure using configuration or code instead of managing infra manually.

Why IaC?

CloudFormation

AWS IaC service that lets us write IaC in plain texts like json or yaml

Journey

of CloudFormation

1. Fully CloudFormation

2. Shared Responsibility Model

3. Rebuild From scratch

Deploying Services

Baby Step

Baby Steps

Makefiles

create-db:
  aws cloudformation create-stack --stack-name "db-stack" ...
create-cache:
  aws cloudformation create-stack --stack-name "cache-stack" ...
create-api: create-db create-cache
  aws cloudformation create-stack --stack-name "third-cfn-stack" ...
create-app: create-api
  @echo "App Successfully Created!!"

• Alias long command
• define dependency
• parallelize
• complex conditions and dependency can be difficult to maintain
• separate targets for update/delete

Bash scripts

#!/bin/bash
echo 'CREATE DATABASE'
make create-db

echo 'CREATE CACHE'
make create-cache
echo 'WAITING.. DATABASE & CACHE'
make wait-create-cache
make wait-create-db

echo 'CREATE API'
make create-api
make wait-create-api

echo 'APP SUCCESSFULLY CREATED'

• Complex conditions and error handling was easier
• Readability was slightly improved
• Still Sequential
• Addition of removable of any intermediate component was error prone
• supported only create
• monolithic script to manage all the stacks

• deploy stacks in parallel
• deploy stacks in specific orders
• declarative over imperative
• deploy specific stacks
• destroy stacks at scale

Ideal Solution

Serverless Compose

#compose

services:
  db:
    path: db
  cache:
    path: cache
  api:
   path: api
   dependsOn:
     - db
     - cache

• serverless compose
• cloudformation-compose

• each stack is a separate entity to manage
• limit the number of stacks
• use nested stacks whenever possible
• adopt infra CI/CD at early stage
• use some form of orchestration if you have many stacks to manage

Take Away

Deploying Platform

Initial Stage

x service dependencies

• Analyze the dependencies
• Move infra component to the owner service
• Move resources to shared infra
• Duplicate resources if needed
• Remove stale dependency
• Use convention and hard coded values

Resolving hard Dependencies

Environment Launch in Prallel

Acutal Snapshot

• Retry Mechanism
• AWS_MAX_ATTEMPTS
• AWS_RETRY_MODE

Rate Exceeded Issue

• Eliminate hard dependencies among services
• Parallelize the launch of services
• Implement Exponential Retry Mechanism
• Choose declarative over imperative methods

Take Away

With Great Automation Comes Great Responsibility

Thank You!

Questions?

linkendin

github

blog