Serverless at ACG
Lunch at RMA
John McKim
VP of Product & Technology
A Cloud Guru
@johncmckim
Agenda
History of ACG Architecture
Components:
- Framework & Code structure
- Application Config
- Deployments
- Monitoring
- Security
- Costs
Q/A session at the end.
ACG MVP
The Architecture Sam built
Microservices V1
Decoupling the Architecture
Microservices v2
Adding a BFF & GraphQL
Microservices V2
Service -> Service messages
Microservices
Conways Law
- Each team owns it's own microservices end-to-end
- Development & Operations
- Expose APIs between services, GraphQL APIs, SNS Topics
- Experiment independently, learn and share knowledge & practices between teams
- 5x teams
Student
Team
Mobile
Team
Org Team
Team
A&B
Team
1x Dev Lead
3x Devs
Secret
Team
Frameworks
Serverless Framework
Serverless Framework
- YAML config that defines a set of functions & resources
- Good framework for structuring a microservice
- Abstraction over CloudFormation
- Plugin system to fill gaps
- Best option in 2017
Serverless Framework
Example Service
service: email-service
provider:
name: aws
runtime: nodejs8.10
region: us-east-1
versionFunctions: false
environment:
STAGE: ${opt:stage}
plugins:
- serverless-plugin-aws-alerts
- serverless-domain-manager
- serverless-logless-plugin
custom:
logless:
logRetention: ${ssm:/${opt:stage}/email/config/LOG_RETENTION_IN_DAYS}
alerts:
dashboards: true
stages:
- production
topics:
ok:
topic: ${self:service}-${opt:stage}-alerts-ok
alarm:
topic: ${self:service}-${opt:stage}-alerts-alarm
insufficientData:
topic: ${self:service}-${opt:stage}-alerts-insufficientData
customDomain:
domainName: ${cf:acloudguru-api-gateway-${opt:stage}.InternalDomainName}
stage: ${opt:stage}
basePath: email
certificateName: '*.acloud.guru'
createRoute53Record: false
endpointType: 'regional'
package:
exclude:
- configure/**
- coverage/**
- src/**/*.test.js
- .node-version
- '*.sh'
- package.json
- yarn.lock
- README.md
functions:
graphql:
handler: src/graphql/index.handler
memorySize: 1536
timeout: 20
events:
- http:
path: graphql
method: post
cors: true
private: true
alarms:
- name: functionInvocations
threshold: 3000
tags:
FunctionType: graphql_api
emailQueueWorker:
handler: src/workers/email-queue.handler
reservedConcurrency: 1
events:
- sqs:
arn: ${ssm:/${opt:stage}/email/config/EMAIL_QUEUE_ARN}
batchSize: 1
resources:
Resources:
EmailTable:
Type: AWS::DynamoDB::Table
DeletionPolicy: ${ssm:/${opt:stage}/email/config/EMAIL_TABLE_DELETION_POLICY}
Properties:
TableName: ${ssm:/${opt:stage}/email/config/EMAIL_TABLE_NAME}
....
EmailGraphqlSchemaBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Delete
Properties:
BucketName: ${ssm:/${opt:stage}/email/config/S3_EMAIL_GRAPHQL_SCHEMA_BUCKET}
ACG Project
Mono-repo with services in folders
|_ school
|_ .buildkite
|_ backend
|_ infrastructure
|_ services
|_ email-service
|_ .buildkite
|_ src
|_ serverless.yml
|_ frontends
|_ mobile
|_ tools
Application Config
History
- Encrypted dotenv files - 2016 pre Environment Variables
- Environment Variables - 2017
- Pros - not a hack, securely stored*
- Cons - values in CloudFormation, values not easily re-used, update requires deployment
- AWS Systems Manager
- Pros - securely stored, separated from code, supported by Serverless framework
- Cons - rate limit
Application Config
Moving to Oprah
service: oprah-service
provider: ssm # or ddb
config:
path: /${stage}/oprah/config
defaults:
DB_NAME: my-database
DB_HOST: 3200
required:
DB_TABLE: "some database table name for ${stage}"
secret:
path: /${stage}/oprah/secret
required:
DB_PASSWORD: "secret database password"
- Pros - nice cli for devs, SSM or DynamoDB Store
- Cons - custom built
oprah.yml
Application Config
Moving to Oprah
const { makeParameterStore } = require('@a-cloud-guru/parameter-store');
const { STAGE } = process.env;
const parameterStore = makeParameterStore({
configPath: `/${SERVICE_STAGE}/foo-service/config`,
secretPath: `/${SERVICE_STAGE}/foo-service/secret`,
provider: {
tableName: `oprah-foo-service-${SERVICE_STAGE}`
}
});
return Promise.all([
parameterStore.getConfigs([
'THE_CONFIG_1',
'THE_CONFIG_2'
]),
parameterStore.getSecrets([
'THE_SECRET_1'
])
])
.then(([configs, secrets]) => ({ ...configs, ...secrets }))
.then(console.log);
example.js
Deployments
Early Days
Monolithic Deployment with TravisCI
- All services deployed in one script sequentially
- Staging environments on develop branch
- Prod environment on master branch
- Pros - Simple to follow
- Cons
- Ever slower as we added more services
- Re-deploy un-changed services
- Not suited for trunk based development
Deployments
Trunk based dev, monorepo & buildkite
Deployment approach
- Trunk based development - merge all PRs back to master
- Code stored in mono-repo
- Services deployed independently
- Buildkite pipelines orchestrate deployment
Commit
Build triggered
Starts Pipelines
Deploy to Test
Deploy to Prod
WAIT
Monitoring
CloudWatch + Sumo
CloudWatch
- Stores logs from Lambdas
- Tracks Lambda metrics
- Serverless Plugin - https://github.com/ACloudGuru/serverless-plugin-aws-alerts
Sumo
- Ingests logs from CloudWatch
- More intelligent alerts from logs
- Send messages into slack
Security
Fairly familiar
- Secure configuration - SSM
- Ops processes - Code Review, Infra as Code, Deployments ect
- Monitoring - turn on CloudTrail, alert on unusual activity
- Understand OWASP Top 10 i.e. injection attacks
- Least privilege IAM policies
- Data Encryption - at rest & transit
Costs
Scales with use & good TOC
Costs
- API Gateway: $483.38
- Lambda: $1,635.71
- DynamoDB: $563.31
- S3: $585.71
Usage
- CloudFront: 11.4 Million requests, 62.1 GB Downloaded
- API Gateway: 128 Million Requests
- Lambda: 196 Million Invocations
- S3: 3.6 TB Data, 1 Trillion+ requests
Thanks for Listening!
Questions?
johncmckim.me
twitter.com/@johncmckim
medium.com/@johncmckim
rate-my-agent
By John McKim
rate-my-agent
- 128