DjangoCon Europe 2019
Copenhagen
Neal Todd
Run code without thinking about servers
AWS Lambda
> zappa init
███████╗ █████╗ ██████╗ ██████╗ █████╗ ╚══███╔╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗ ███╔╝ ███████║██████╔╝██████╔╝███████║ ███╔╝ ██╔══██║██╔═══╝ ██╔═══╝ ██╔══██║ ███████╗██║ ██║██║ ██║ ██║ ██║ ╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚═╝ ╚═╝ Welcome to Zappa! Zappa is a system for running server-less Python web applications on AWS Lambda and AWS API Gateway.
Rich Jones / www.gun.io
Lambda
Route 53
Certificate Manager
S3
RDS
VPC
Satellite Ground Station
IAM
API Gateway
512MB Lambda function
➝ One million 800ms requests per month = zero cost
But you must factor in other costs…
Lift and Shift
JSON or YML file in your project directory
zappa init
Let's create speedrun.bygge.net
Full walkthrough at:
"Here's One I Made Earlier"
[[ ! -z "$SITE" ]] || exit
mkdir ${SITE} && cd ${SITE}
python3 -m venv .venv && source .venvs/bin/activate
pip install zappa
pip install zappa zappa-django-utils django-storages awscli
git clone https://github.com/wagtail/bakerydemo.git && cd bakerydemo
pip install -r requirements/base.txt
cat << EOF >> bakerydemo/settings/dev.py
DATABASES = {
'default': {
'ENGINE': 'zappa_django_utils.db.backends.s3sqlite',
'NAME': '${SITE}-sqlite.db',
'BUCKET': '${SITE}-db'
}
}
ALLOWED_HOSTS = ['.eu-west-2.amazonaws.com', '.bygge.net']
INSTALLED_APPS += ('storages',)
AWS_STORAGE_BUCKET_NAME = '${SITE}-static'
AWS_QUERYSTRING_AUTH = False
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEBUG = os.getenv('DEBUG', 'off') == 'on'
EOF
cat << EOF > zappa_settings.json
{
"dev": {
"django_settings": "bakerydemo.settings.dev",
"profile_name": "iam-zappa",
"aws_region": "eu-west-2",
"s3_bucket": "${SITE}-zappa",
"project_name": "${SITE}",
"runtime": "python3.6",
"timeout_seconds": 60,
"domain": "${SITE}.bygge.net",
"certificate_arn": "${CERT_ARN}",
"aws_environment_variables": {"DEBUG": "off"}
}
}
EOF
aws s3api create-bucket --bucket ${SITE}-db --profile iam-zappa \
--region eu-west-2 --create-bucket-configuration LocationConstraint=eu-west-2
aws s3api create-bucket --bucket ${SITE}-static --profile iam-zappa \
--region eu-west-2 --create-bucket-configuration LocationConstraint=eu-west-2
aws s3api put-bucket-cors --bucket ${SITE}-static --profile iam-zappa --cors-configuration \
'{"CORSRules": [{"AllowedOrigins": ["*"], "AllowedMethods": ["GET"]}]}'
aws s3api put-bucket-policy --bucket ${SITE}-static --profile iam-zappa --policy \
'{"Statement": [{"Effect": "Allow","Principal": "*","Action": "s3:GetObject",'\
'"Resource": "arn:aws:s3:::${SITE}-static/*"}]}'
aws s3 sync bakerydemo/media/original_images/ s3://${SITE}-static/original_images/ \
--profile iam-zappa
zappa deploy dev
zappa certify dev -y
zappa manage dev "collectstatic --noinput"
zappa manage dev migrate
zappa manage dev load_initial_data
zappa status dev
zappa update dev
Details:
S3
API
Gateway
λ
venv
S3
Route 53 / ACM
Cloudfront
"exclude": [],
"environment_variables": {},
"aws_environment_variables": {}
zappa tail dev --disable-keep-open --since 5m
zappa invoke dev "print(os.environ)" --raw
zappa undeploy dev
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ZappaStackUpdatePolicies",
"Effect": "Allow",
"Action": [
"apigateway:DELETE",
"apigateway:GET",
"apigateway:PATCH",
"apigateway:POST",
"apigateway:PUT",
AdministratorAccess
Provides full access to AWS services and resources
{
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
Restricting permissions with roles and policies isn't trivial
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ZappaLambdaExecutionRolePolicies",
"Effect": "Allow",
"Action": [
"iam:GetRole",
"iam:PassRole"
],
"Resource": [
"arn:aws:iam::346164020800:role/iam-zappa-ZappaLambdaExecutionRole"
]
},
{
"Sid": "ZappaBucketList",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::speedrun-zappa"
]
},
{
"Sid": "ZappaBucketObjectManagement",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:ListMultipartUploadParts"
],
"Resource": "arn:aws:s3:::speedrun-zappa/*"
}
]
}
"cloudformation:CreateStack",
"cloudformation:DeleteStack",
"cloudformation:DescribeStackResource",
"cloudformation:DescribeStacks",
"cloudformation:ListStackResources",
"cloudformation:UpdateStack",
"events:DeleteRule",
"events:DescribeRule",
"events:ListRuleNamesByTarget",
"events:ListRules",
"events:ListTargetsByRule",
"events:PutRule",
"events:PutTargets",
"events:RemoveTargets",
"lambda:AddPermission",
"lambda:CreateFunction",
"lambda:DeleteFunction",
"lambda:GetPolicy",
"lambda:GetFunction",
"lambda:GetFunctionConfiguration",
"lambda:ListVersionsByFunction",
"lambda:InvokeFunction",
"lambda:RemovePermission",
"lambda:UpdateFunctionCode",
"lambda:UpdateFunctionConfiguration",
"logs:DescribeLogStreams",
"logs:FilterLogEvents",
"route53:ChangeResourceRecordSets",
"route53:GetHostedZone",
"route53:ListHostedZones",
"cloudfront:UpdateDistribution"
],
"Resource": "*"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::speedrun-static/*"
},
{
"Sid": "AllowUserManageBucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::346164020800:user/iam-zappa"
},
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions"
],
"Resource": "arn:aws:s3:::speedrun-static"
},
{
"Sid": "AllowUserManageBucketObjects",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::346164020800:user/iam-zappa"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::speedrun-static/*"
},
{
"Sid": "WatchtowerWagtailLoggingManagement",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup"
],
"Resource": [
"*"
]
},
{
"Sid": "WatchtowerWagtailLogging",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:eu-west-2:346164020800:log-group:speedrun:log-stream:*"
]
}
]
}
RDS - Relational Database Service
Micro instances: $14/month (free 1st year)
SQLite on S3 scales well for high-reads but not suited to high-write concurrency
Aurora Serverless: No free usage. Time & Resource based costs
import dj_database_url
DATABASES = {
'default': dj_database_url.config()
}
+ pip install psycopg2, dj-database-url
- aws s3api create-bucket --bucket ${SITE}-db …
DATABASE_URL = postgres://USER:PASSWORD@HOST:PORT/NAME
Create Database Cluster and application Database in RDS
Get Host URL after provisioning & add to environment variables:
Adjust the setup to replace SQLite with PostgreSQL:
Create Database Cluster, configuring Aurora Compute Units
Minimum and Maximum ACU and Pause time very important.
Defaults to 2 ACU Min and 64 ACU Max and no pause
➝ $100/month even if it does nothing
Set a pause after X minutes of inactivity
➝ Cold start (~30s)
Going "full serverless" is tempting but Aurora Serverless is suited for infrequent, intermittent, or unpredictable workloads
Not a good fit for production Django applications
Virtual Private Cloud
S3 Gateway
S3
API
Gateway
RDS
λ
IGW
Private
Public
NAT
GW
NAT Gateway ➝ $36/mth
Aurora serverless isolated from internet -
Lambda must be in VPC
SQLite on S3
Relative median response time
for requests to the frontend
RDS Postgres
(Micro)
Aurora
Serverless
(2-8ACU)
0.69
0.70
Service | Units | Price per unit (eu-west-2) | Free quota |
---|---|---|---|
Lambda | GB-Second | $0.00001667 | ✓ |
Request | $0.0000002 | ✓ | |
GB of data out | $0.09 | ✓ | |
RDS PostgreSQL | Instance hour | $0.021 (μ) / $0.164 (L) | ✓ (1y) |
GB-month of storage | $0.133 | ✓ (1y) | |
Aurora Serverless | ACU hour | $0.07 | |
Million I/O request | $0.232 | ||
GB-month of storage | $0.116 | ||
S3 | 1,000 GET | $0.00042 | ✓ (1y) |
1,000 PUT | $0.0053 | ✓ (1y) | |
GB of data out | $0.09 | ✓ (1y) | |
GB of storage | $0.024 | ||
Route 53 | Hosted Zone month | $0.50 | |
Million DNS request | $0.40 |
Well, here's another serverless you've gotten me into.
Adam Johnson
Workshop 1
10:45
Zappa | github.com/Miserlou/Zappa |
www.zappa.io | |
AWS Free Tier | aws.amazon.com/free |
Speedrun walkthrough | nealtodd.github.io/serverless-django/djconeu-speedrun/ |
Speedrun screencast | asciinema.org/a/237320 |
Slide deck | slides.com/nealtodd/deck |
Icons from the Noun Project: Hamster by Dream Icons; Server by Ralf Schmitzer;Plug by vigorn; Database by Herbert Spencer.
AWS service icons copyright Amazon Web Services.
Stills from "The Music Box" copyright 1932 CCA. (Watch this film!)
SQLite on S3
Relative response time
for requests to the backend
RDS Postgres
(Micro)
Aurora
Serverless
(2-8ACU)
0.73
0.74
Aurora Serverless 'cold' and 'warm' starts for a Locust swarm of concurrent users.
Scalable but expensive when not paused.
High latency but zero cost when paused.
Not an ideal fit for public Django applications