Serverless Django with Zappa
DjangoCon Europe 2019
Copenhagen
Neal Todd
Serverless
Run code without thinking about servers
AWS Lambda
- Function as a Service (FaaS)
- Stateless
- Full utilisation
- Cost proportional to execution time
- Scale without intervention
Why Serverless Django?
- Because it's there
-
Low traffic / experimental / personal project sites
- Relatively easy to deploy sites
- Low running costs
- Production sites:
- Potential for lower running costs
- Automatic scalability for peaks in demand
> zappa init
███████╗ █████╗ ██████╗ ██████╗ █████╗ ╚══███╔╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗ ███╔╝ ███████║██████╔╝██████╔╝███████║ ███╔╝ ██╔══██║██╔═══╝ ██╔═══╝ ██╔══██║ ███████╗██║ ██║██║ ██║ ██║ ██║ ╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚═╝ ╚═╝ Welcome to Zappa! Zappa is a system for running server-less Python web applications on AWS Lambda and AWS API Gateway.
Rich Jones / www.gun.io
Amazon Web Services
Lambda
Route 53
Certificate Manager
S3
RDS
VPC
Satellite Ground Station
IAM
API Gateway
AWS Free Tier
- Lambda
- 1m requests/mth
- 400k GB-seconds/mth
-
S3
- 5 GB storage
- 20k Get / 2k Put requests
-
RDS Micro
- 750 hr/mth
- 20 GB storage
512MB Lambda function
➝ One million 800ms requests per month = zero cost
But you must factor in other costs…
Lift and Shift
Zappa
Settings
JSON or YML file in your project directory
zappa init
Zappa in action
Let's create speedrun.bygge.net
- Wagtail Bakery Demo as a sample Django application
- S3 bucket for static assets
- S3 bucket for a SQLite database
- Custom Domain Name
Full walkthrough at:
"Here's One I Made Earlier"
[[ ! -z "$SITE" ]] || exit
mkdir ${SITE} && cd ${SITE}
python3 -m venv .venv && source .venvs/bin/activate
pip install zappa
pip install zappa zappa-django-utils django-storages awscli
git clone https://github.com/wagtail/bakerydemo.git && cd bakerydemo
pip install -r requirements/base.txt
cat << EOF >> bakerydemo/settings/dev.py
DATABASES = {
'default': {
'ENGINE': 'zappa_django_utils.db.backends.s3sqlite',
'NAME': '${SITE}-sqlite.db',
'BUCKET': '${SITE}-db'
}
}
ALLOWED_HOSTS = ['.eu-west-2.amazonaws.com', '.bygge.net']
INSTALLED_APPS += ('storages',)
AWS_STORAGE_BUCKET_NAME = '${SITE}-static'
AWS_QUERYSTRING_AUTH = False
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEBUG = os.getenv('DEBUG', 'off') == 'on'
EOF
cat << EOF > zappa_settings.json
{
"dev": {
"django_settings": "bakerydemo.settings.dev",
"profile_name": "iam-zappa",
"aws_region": "eu-west-2",
"s3_bucket": "${SITE}-zappa",
"project_name": "${SITE}",
"runtime": "python3.6",
"timeout_seconds": 60,
"domain": "${SITE}.bygge.net",
"certificate_arn": "${CERT_ARN}",
"aws_environment_variables": {"DEBUG": "off"}
}
}
EOF
aws s3api create-bucket --bucket ${SITE}-db --profile iam-zappa \
--region eu-west-2 --create-bucket-configuration LocationConstraint=eu-west-2
aws s3api create-bucket --bucket ${SITE}-static --profile iam-zappa \
--region eu-west-2 --create-bucket-configuration LocationConstraint=eu-west-2
aws s3api put-bucket-cors --bucket ${SITE}-static --profile iam-zappa --cors-configuration \
'{"CORSRules": [{"AllowedOrigins": ["*"], "AllowedMethods": ["GET"]}]}'
aws s3api put-bucket-policy --bucket ${SITE}-static --profile iam-zappa --policy \
'{"Statement": [{"Effect": "Allow","Principal": "*","Action": "s3:GetObject",'\
'"Resource": "arn:aws:s3:::${SITE}-static/*"}]}'
aws s3 sync bakerydemo/media/original_images/ s3://${SITE}-static/original_images/ \
--profile iam-zappa
zappa deploy dev
zappa certify dev -y
zappa manage dev "collectstatic --noinput"
zappa manage dev migrate
zappa manage dev load_initial_data
zappa status dev
zappa update dev
Details:
S3
API
Gateway
λ
venv
S3
Route 53 / ACM
Cloudfront
"exclude": [],
"environment_variables": {},
"aws_environment_variables": {}
zappa tail dev --disable-keep-open --since 5m
zappa invoke dev "print(os.environ)" --raw
zappa undeploy dev
AWS Roles & Policies
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ZappaStackUpdatePolicies",
"Effect": "Allow",
"Action": [
"apigateway:DELETE",
"apigateway:GET",
"apigateway:PATCH",
"apigateway:POST",
"apigateway:PUT",
AdministratorAccess
Provides full access to AWS services and resources
{
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
Restricting permissions with roles and policies isn't trivial
AWS Roles & Policies
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ZappaLambdaExecutionRolePolicies",
"Effect": "Allow",
"Action": [
"iam:GetRole",
"iam:PassRole"
],
"Resource": [
"arn:aws:iam::346164020800:role/iam-zappa-ZappaLambdaExecutionRole"
]
},
{
"Sid": "ZappaBucketList",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::speedrun-zappa"
]
},
{
"Sid": "ZappaBucketObjectManagement",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:ListMultipartUploadParts"
],
"Resource": "arn:aws:s3:::speedrun-zappa/*"
}
]
}
"cloudformation:CreateStack",
"cloudformation:DeleteStack",
"cloudformation:DescribeStackResource",
"cloudformation:DescribeStacks",
"cloudformation:ListStackResources",
"cloudformation:UpdateStack",
"events:DeleteRule",
"events:DescribeRule",
"events:ListRuleNamesByTarget",
"events:ListRules",
"events:ListTargetsByRule",
"events:PutRule",
"events:PutTargets",
"events:RemoveTargets",
"lambda:AddPermission",
"lambda:CreateFunction",
"lambda:DeleteFunction",
"lambda:GetPolicy",
"lambda:GetFunction",
"lambda:GetFunctionConfiguration",
"lambda:ListVersionsByFunction",
"lambda:InvokeFunction",
"lambda:RemovePermission",
"lambda:UpdateFunctionCode",
"lambda:UpdateFunctionConfiguration",
"logs:DescribeLogStreams",
"logs:FilterLogEvents",
"route53:ChangeResourceRecordSets",
"route53:GetHostedZone",
"route53:ListHostedZones",
"cloudfront:UpdateDistribution"
],
"Resource": "*"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::speedrun-static/*"
},
{
"Sid": "AllowUserManageBucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::346164020800:user/iam-zappa"
},
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions"
],
"Resource": "arn:aws:s3:::speedrun-static"
},
{
"Sid": "AllowUserManageBucketObjects",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::346164020800:user/iam-zappa"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::speedrun-static/*"
},
{
"Sid": "WatchtowerWagtailLoggingManagement",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup"
],
"Resource": [
"*"
]
},
{
"Sid": "WatchtowerWagtailLogging",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:eu-west-2:346164020800:log-group:speedrun:log-stream:*"
]
}
]
}
Databases
RDS - Relational Database Service
- PostgreSQL
- MySQL & MariaDB
- Oracle
- Microsoft SQL Server
- Aurora
- Aurora Serverless
Micro instances: $14/month (free 1st year)
SQLite on S3 scales well for high-reads but not suited to high-write concurrency
Aurora Serverless: No free usage. Time & Resource based costs
PostgreSQL
import dj_database_url
DATABASES = {
'default': dj_database_url.config()
}
+ pip install psycopg2, dj-database-url
- aws s3api create-bucket --bucket ${SITE}-db …
DATABASE_URL = postgres://USER:PASSWORD@HOST:PORT/NAME
Create Database Cluster and application Database in RDS
Get Host URL after provisioning & add to environment variables:
Adjust the setup to replace SQLite with PostgreSQL:
Aurora Serverless
Create Database Cluster, configuring Aurora Compute Units
Minimum and Maximum ACU and Pause time very important.
Defaults to 2 ACU Min and 64 ACU Max and no pause
➝ $100/month even if it does nothing
Set a pause after X minutes of inactivity
➝ Cold start (~30s)
Going "full serverless" is tempting but Aurora Serverless is suited for infrequent, intermittent, or unpredictable workloads
Not a good fit for production Django applications
VPC
Virtual Private Cloud
S3 Gateway
S3
API
Gateway
RDS
λ
IGW
Private
Public
NAT
GW
NAT Gateway ➝ $36/mth
Aurora serverless isolated from internet -
Lambda must be in VPC
Performance under load
SQLite on S3
Relative median response time
for requests to the frontend
RDS Postgres
(Micro)
Aurora
Serverless
(2-8ACU)
0.69
0.70
Zappa miscellany
- Building package from a virtual environment
- Package size limitations
- Timeouts: Gateway vs. Lambda
- Global deployment to all available regions
- Scheduled functions
- Rollbacks
Costs beyond $0 Lambda
Service | Units | Price per unit (eu-west-2) | Free quota |
---|---|---|---|
Lambda | GB-Second | $0.00001667 | ✓ |
Request | $0.0000002 | ✓ | |
GB of data out | $0.09 | ✓ | |
RDS PostgreSQL | Instance hour | $0.021 (μ) / $0.164 (L) | ✓ (1y) |
GB-month of storage | $0.133 | ✓ (1y) | |
Aurora Serverless | ACU hour | $0.07 | |
Million I/O request | $0.232 | ||
GB-month of storage | $0.116 | ||
S3 | 1,000 GET | $0.00042 | ✓ (1y) |
1,000 PUT | $0.0053 | ✓ (1y) | |
GB of data out | $0.09 | ✓ (1y) | |
GB of storage | $0.024 | ||
Route 53 | Hosted Zone month | $0.50 | |
Million DNS request | $0.40 |
Well, here's another serverless you've gotten me into.
Building a Serverless Django Application
Adam Johnson
Workshop 1
10:45
References
Zappa | github.com/Miserlou/Zappa |
www.zappa.io | |
AWS Free Tier | aws.amazon.com/free |
Speedrun walkthrough | nealtodd.github.io/serverless-django/djconeu-speedrun/ |
Speedrun screencast | asciinema.org/a/237320 |
Slide deck | slides.com/nealtodd/deck |
-
Icons from the Noun Project: Hamster by Dream Icons; Server by Ralf Schmitzer;Plug by vigorn; Database by Herbert Spencer.
-
AWS service icons copyright Amazon Web Services.
-
Stills from "The Music Box" copyright 1932 CCA. (Watch this film!)
Performance
SQLite on S3
Relative response time
for requests to the backend
RDS Postgres
(Micro)
Aurora
Serverless
(2-8ACU)
0.73
0.74
Cold starts
Aurora Serverless 'cold' and 'warm' starts for a Locust swarm of concurrent users.
Scalable but expensive when not paused.
High latency but zero cost when paused.
Not an ideal fit for public Django applications
deck
By Neal Todd
deck
- 1,004