Serverless at ACG

Lunch at RMA

John McKim

VP of Product & Technology

A Cloud Guru

@johncmckim

Agenda

History of ACG Architecture

Components:

  • Framework & Code structure
  • Application Config
  • Deployments
  • Monitoring
  • Security
  • Costs

 

Q/A session at the end.

ACG MVP

The Architecture Sam built

Microservices V1

Decoupling the Architecture

Microservices v2

Adding a BFF & GraphQL

Microservices V2

Service -> Service messages

Microservices

Conways Law

  • Each team owns it's own microservices end-to-end
    • Development & Operations
  • Expose APIs between services, GraphQL APIs, SNS Topics
  • Experiment independently, learn and share knowledge & practices between teams
  • 5x teams

Student

Team

Mobile

Team

Org Team

Team

A&B

Team

1x Dev Lead

3x Devs

Secret

Team

Frameworks

Serverless Framework

Serverless Framework

  • YAML config that defines a set of functions & resources
  • Good framework for structuring a microservice
  • Abstraction over CloudFormation
  • Plugin system to fill gaps
  • Best option in 2017

Serverless Framework

Example Service

service: email-service
provider:
  name: aws
  runtime: nodejs8.10
  region: us-east-1
  versionFunctions: false
  environment:
    STAGE: ${opt:stage}

plugins:
  - serverless-plugin-aws-alerts
  - serverless-domain-manager
  - serverless-logless-plugin

custom:
  logless:
    logRetention: ${ssm:/${opt:stage}/email/config/LOG_RETENTION_IN_DAYS}
  alerts:
    dashboards: true
    stages:
      - production
    topics:
      ok:
        topic: ${self:service}-${opt:stage}-alerts-ok
      alarm:
        topic: ${self:service}-${opt:stage}-alerts-alarm
      insufficientData:
        topic: ${self:service}-${opt:stage}-alerts-insufficientData
  customDomain:
    domainName: ${cf:acloudguru-api-gateway-${opt:stage}.InternalDomainName}
    stage: ${opt:stage}
    basePath: email
    certificateName: '*.acloud.guru'
    createRoute53Record: false
    endpointType: 'regional'

package:
  exclude:
    - configure/**
    - coverage/**
    - src/**/*.test.js
    - .node-version
    - '*.sh'
    - package.json
    - yarn.lock
    - README.md

functions:
  graphql:
    handler: src/graphql/index.handler
    memorySize: 1536
    timeout: 20
    events:
      - http:
          path: graphql
          method: post
          cors: true
          private: true
    alarms:
      - name: functionInvocations
        threshold: 3000
    tags:
      FunctionType: graphql_api

  emailQueueWorker:
    handler: src/workers/email-queue.handler
    reservedConcurrency: 1
    events:
      - sqs:
          arn: ${ssm:/${opt:stage}/email/config/EMAIL_QUEUE_ARN}
          batchSize: 1

resources:
  Resources:
    EmailTable:
      Type: AWS::DynamoDB::Table
      DeletionPolicy: ${ssm:/${opt:stage}/email/config/EMAIL_TABLE_DELETION_POLICY}
      Properties:
        TableName: ${ssm:/${opt:stage}/email/config/EMAIL_TABLE_NAME}
        ....

    EmailGraphqlSchemaBucket:
      Type: AWS::S3::Bucket
      DeletionPolicy: Delete
      Properties:
        BucketName: ${ssm:/${opt:stage}/email/config/S3_EMAIL_GRAPHQL_SCHEMA_BUCKET}

ACG Project

Mono-repo with services in folders

|_ school
  |_ .buildkite
  |_ backend
    |_ infrastructure
    |_ services
      |_ email-service
        |_ .buildkite
        |_ src
        |_ serverless.yml
  |_ frontends
  |_ mobile
  |_ tools

Application Config

History

  • Encrypted dotenv files - 2016 pre Environment Variables
  • Environment Variables - 2017
    • Pros - not a hack, securely stored*
    • Cons - values in CloudFormation, values not easily re-used, update requires deployment
  • AWS Systems Manager
    • Pros - securely stored, separated from code, supported by Serverless framework
    • Cons - rate limit

Application Config

Moving to Oprah

service: oprah-service
provider: ssm # or ddb

config:
  path: /${stage}/oprah/config
  defaults:
    DB_NAME: my-database
    DB_HOST: 3200
  required:
    DB_TABLE: "some database table name for ${stage}"

secret:
  path: /${stage}/oprah/secret
  required:
    DB_PASSWORD: "secret database password"
  • Pros - nice cli for devs, SSM or DynamoDB Store
  • Cons - custom built

oprah.yml

Application Config

Moving to Oprah

const { makeParameterStore } = require('@a-cloud-guru/parameter-store');
const { STAGE } = process.env;

const parameterStore = makeParameterStore({
  configPath: `/${SERVICE_STAGE}/foo-service/config`,
  secretPath: `/${SERVICE_STAGE}/foo-service/secret`,
  provider: {
    tableName: `oprah-foo-service-${SERVICE_STAGE}`
  }
});

return Promise.all([
  parameterStore.getConfigs([
    'THE_CONFIG_1',
    'THE_CONFIG_2'
  ]),
  parameterStore.getSecrets([
    'THE_SECRET_1'
  ])
])
.then(([configs, secrets]) => ({ ...configs, ...secrets }))
.then(console.log);

example.js

Deployments

Early Days

Monolithic Deployment with TravisCI

  • All services deployed in one script sequentially
  • Staging environments on develop branch
  • Prod environment on master branch

 

  • Pros - Simple to follow
  • Cons
    • Ever slower as we added more services
    • Re-deploy un-changed services
    • Not suited for trunk based development

Deployments

Trunk based dev, monorepo & buildkite

Deployment approach

  • Trunk based development - merge all PRs back to master
  • Code stored in mono-repo
  • Services deployed independently
  • Buildkite pipelines orchestrate deployment

Commit

Build triggered

Starts Pipelines

Deploy to Test

Deploy to Prod

WAIT

Monitoring

CloudWatch + Sumo

CloudWatch

  • Stores logs from Lambdas
  • Tracks Lambda metrics
  • Serverless Plugin - https://github.com/ACloudGuru/serverless-plugin-aws-alerts

 

Sumo

  • Ingests logs from CloudWatch
  • More intelligent alerts from logs
  • Send messages into slack

Security

Fairly familiar

  • Secure configuration - SSM
  • Ops processes - Code Review, Infra as Code, Deployments ect
  • Monitoring - turn on CloudTrail, alert on unusual activity
  • Understand OWASP Top 10 i.e. injection attacks
  • Least privilege IAM policies
  • Data Encryption - at rest & transit

 

Costs

Scales with use & good TOC

Costs

  • API Gateway: $483.38
  • Lambda: $1,635.71
  • DynamoDB: $563.31
  • S3: $585.71

 

Usage

  • CloudFront: 11.4 Million requests, 62.1 GB Downloaded
  • API Gateway: 128 Million Requests
  • Lambda: 196 Million Invocations
  • S3: 3.6 TB Data, 1 Trillion+ requests

Thanks for Listening!

Questions?

johncmckim.me

twitter.com/@johncmckim

medium.com/@johncmckim

rate-my-agent

By John McKim

rate-my-agent

  • 128