Serverless Django with Zappa

DjangoCon Europe 2019

Copenhagen

Neal Todd

Serverless

Run code without thinking about servers

AWS Lambda

  • Function as a Service (FaaS)
  • Stateless
  • Full utilisation
  • Cost proportional to execution time 
  • Scale without intervention

Why Serverless Django?

  • Because it's there
  • Low traffic / experimental / personal project sites
    • Relatively easy to deploy sites
    • Low running costs
  • Production sites:
    • Potential for lower running costs
    • Automatic scalability for peaks in demand
> zappa init
███████╗ █████╗ ██████╗ ██████╗  █████╗
╚══███╔╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗
  ███╔╝ ███████║██████╔╝██████╔╝███████║
 ███╔╝  ██╔══██║██╔═══╝ ██╔═══╝ ██╔══██║
███████╗██║  ██║██║     ██║     ██║  ██║
╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝     ╚═╝  ╚═╝

Welcome to Zappa!

Zappa is a system for running server-less Python web applications on AWS Lambda and AWS API Gateway.

Rich Jones / www.gun.io

Amazon Web Services

Lambda

Route 53

Certificate Manager

S3

RDS

VPC

Satellite Ground Station

IAM

API Gateway

AWS Free Tier

  • Lambda
    • 1m requests/mth
    • 400k GB-seconds/mth
  • S3
    • 5 GB storage
    • 20k Get / 2k Put requests
  • ​RDS Micro
    • 750 hr/mth
    • 20 GB storage

512MB Lambda function

➝ One million 800ms requests per month = zero cost

But you must factor in other costs

Lift and Shift

Zappa

Settings

JSON or YML file in your project directory

 

zappa init

Zappa in action

Let's create speedrun.bygge.net

  • Wagtail Bakery Demo as a sample Django application
  • S3 bucket for static assets
  • S3 bucket for a SQLite database
  • Custom Domain Name

Full walkthrough at:

"Here's One I Made Earlier"

[[ ! -z "$SITE" ]] || exit

mkdir ${SITE} && cd ${SITE}

python3 -m venv .venv && source .venvs/bin/activate

pip install zappa

pip install zappa zappa-django-utils django-storages awscli

git clone https://github.com/wagtail/bakerydemo.git && cd bakerydemo
pip install -r requirements/base.txt

cat << EOF >> bakerydemo/settings/dev.py

DATABASES = {
    'default': {
        'ENGINE': 'zappa_django_utils.db.backends.s3sqlite',
        'NAME': '${SITE}-sqlite.db',
        'BUCKET': '${SITE}-db'
    }
}

ALLOWED_HOSTS = ['.eu-west-2.amazonaws.com', '.bygge.net']

INSTALLED_APPS += ('storages',)
AWS_STORAGE_BUCKET_NAME = '${SITE}-static'
AWS_QUERYSTRING_AUTH = False
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

DEBUG = os.getenv('DEBUG', 'off') == 'on'
EOF
cat << EOF > zappa_settings.json
{

    "dev": {
            "django_settings": "bakerydemo.settings.dev",
            "profile_name": "iam-zappa",
            "aws_region": "eu-west-2",

            "s3_bucket": "${SITE}-zappa",
            "project_name": "${SITE}",
            "runtime": "python3.6",
            "timeout_seconds": 60,

            "domain": "${SITE}.bygge.net",
            "certificate_arn": "${CERT_ARN}",
            "aws_environment_variables": {"DEBUG": "off"}
    }

}
EOF

aws s3api create-bucket --bucket ${SITE}-db --profile iam-zappa \
    --region eu-west-2 --create-bucket-configuration LocationConstraint=eu-west-2
aws s3api create-bucket --bucket ${SITE}-static --profile iam-zappa \
    --region eu-west-2 --create-bucket-configuration LocationConstraint=eu-west-2

aws s3api put-bucket-cors --bucket ${SITE}-static --profile iam-zappa --cors-configuration \
    '{"CORSRules": [{"AllowedOrigins": ["*"], "AllowedMethods": ["GET"]}]}'
aws s3api put-bucket-policy --bucket ${SITE}-static --profile iam-zappa --policy \
    '{"Statement": [{"Effect": "Allow","Principal": "*","Action": "s3:GetObject",'\
    '"Resource": "arn:aws:s3:::${SITE}-static/*"}]}'

aws s3 sync bakerydemo/media/original_images/ s3://${SITE}-static/original_images/ \
    --profile iam-zappa

zappa deploy dev

zappa certify dev -y

zappa manage dev "collectstatic --noinput"
zappa manage dev migrate
zappa manage dev load_initial_data

zappa status dev

zappa update dev

Details:

S3

API

Gateway

λ

venv

S3

Route 53 / ACM
Cloudfront

"exclude": [],
"environment_variables": {},
"aws_environment_variables": {}
zappa tail dev --disable-keep-open --since 5m
zappa invoke dev "print(os.environ)" --raw
zappa undeploy dev

AWS Roles & Policies

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ZappaStackUpdatePolicies",
            "Effect": "Allow",
            "Action": [
                "apigateway:DELETE",
                "apigateway:GET",
                "apigateway:PATCH",
                "apigateway:POST",
                "apigateway:PUT",
AdministratorAccess

 

Provides full access to AWS services and resources

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Restricting permissions with roles and policies isn't trivial

AWS Roles & Policies

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ZappaLambdaExecutionRolePolicies",
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "iam:PassRole"
            ],
            "Resource": [
                "arn:aws:iam::346164020800:role/iam-zappa-ZappaLambdaExecutionRole"
            ]
        },
        {
            "Sid": "ZappaBucketList",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::speedrun-zappa"
            ]
        },
        {
            "Sid": "ZappaBucketObjectManagement",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::speedrun-zappa/*"
        }
    ]
}
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeStackResource",
                "cloudformation:DescribeStacks",
                "cloudformation:ListStackResources",
                "cloudformation:UpdateStack",
                "events:DeleteRule",
                "events:DescribeRule",
                "events:ListRuleNamesByTarget",
                "events:ListRules",
                "events:ListTargetsByRule",
                "events:PutRule",
                "events:PutTargets",
                "events:RemoveTargets",
                "lambda:AddPermission",
                "lambda:CreateFunction",
                "lambda:DeleteFunction",
                "lambda:GetPolicy",
                "lambda:GetFunction",
                "lambda:GetFunctionConfiguration",
                "lambda:ListVersionsByFunction",
                "lambda:InvokeFunction",
                "lambda:RemovePermission",
                "lambda:UpdateFunctionCode",
                "lambda:UpdateFunctionConfiguration",
                "logs:DescribeLogStreams",
                "logs:FilterLogEvents",
                "route53:ChangeResourceRecordSets",
                "route53:GetHostedZone",
                "route53:ListHostedZones",
                "cloudfront:UpdateDistribution"
            ],
            "Resource": "*"
        }
    ]
}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::speedrun-static/*"
        },
        {
            "Sid": "AllowUserManageBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::346164020800:user/iam-zappa"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Resource": "arn:aws:s3:::speedrun-static"
        },
        {
            "Sid": "AllowUserManageBucketObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::346164020800:user/iam-zappa"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::speedrun-static/*"
        },
        {
            "Sid": "WatchtowerWagtailLoggingManagement",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "WatchtowerWagtailLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:eu-west-2:346164020800:log-group:speedrun:log-stream:*"
            ]
        }
    ]
}

Databases

RDS - Relational Database Service

  • PostgreSQL
  • MySQL & MariaDB
  • Oracle
  • Microsoft SQL Server
  • Aurora
  • Aurora Serverless

Micro instances: $14/month (free 1st year)

SQLite on S3 scales well for high-reads but not suited to high-write concurrency

Aurora Serverless: No free usage. Time & Resource based costs

PostgreSQL

import dj_database_url

DATABASES = {
    'default': dj_database_url.config()
}
+ pip install psycopg2, dj-database-url
- aws s3api create-bucket --bucket ${SITE}-db …
DATABASE_URL = postgres://USER:PASSWORD@HOST:PORT/NAME

Create Database Cluster and application Database in RDS

Get Host URL after provisioning & add to environment variables:

Adjust the setup to replace SQLite with PostgreSQL:

Aurora Serverless

Create Database Cluster, configuring Aurora Compute Units

Minimum and Maximum ACU and Pause time very important.

Defaults to 2 ACU Min and 64 ACU Max and no pause

➝ $100/month even if it does nothing

Set a pause after X minutes of inactivity

➝ Cold start (~30s)

Going "full serverless" is tempting but Aurora Serverless is suited for infrequent, intermittent, or unpredictable workloads

Not a good fit for production Django applications

VPC

Virtual Private Cloud

S3 Gateway

S3

API

Gateway

RDS

λ

IGW 

Private

Public

NAT

GW

NAT Gateway ➝ $36/mth

Aurora serverless isolated from internet -

Lambda must be in VPC

Performance under load

SQLite on S3

Relative median response time

for requests to the frontend

RDS Postgres

(Micro) 

Aurora

Serverless

(2-8ACU)

0.69

0.70

Zappa miscellany

  • Building package from a virtual environment
  • Package size limitations
  • Timeouts: Gateway vs. Lambda
  • Global deployment to all available regions
  • Scheduled functions
  • Rollbacks

Costs beyond $0 Lambda

Service Units Price per unit (eu-west-2) Free quota
Lambda GB-Second $0.00001667
Request $0.0000002 
GB of data out $0.09
RDS PostgreSQL Instance hour $0.021 (μ) / $0.164 (L) ✓ (1y)
GB-month of storage $0.133 ✓ (1y)
Aurora Serverless ACU hour $0.07
Million I/O request $0.232
GB-month of storage $0.116
S3 1,000 GET $0.00042 ✓ (1y)
1,000 PUT $0.0053 ✓ (1y)
GB of data out $0.09 ✓ (1y)
GB of storage $0.024
Route 53 Hosted Zone month $0.50
Million DNS request $0.40

Well, here's another serverless you've gotten me into.

Building a Serverless Django Application

Adam Johnson

Workshop 1

10:45

References

Zappa github.com/Miserlou/Zappa
www.zappa.io
AWS Free Tier aws.amazon.com/free
Speedrun walkthrough nealtodd.github.io/serverless-django/djconeu-speedrun/
Speedrun screencast asciinema.org/a/237320
Slide deck slides.com/nealtodd/deck
  • Icons from the Noun Project: Hamster by Dream Icons; Server by  Ralf Schmitzer;Plug by vigorn; Database by Herbert Spencer.

  • AWS  service icons copyright Amazon Web Services.

  • Stills from "The Music Box" copyright 1932 CCA.  (Watch this film!)

Performance

SQLite on S3

Relative response time

for requests to the backend

RDS Postgres

(Micro) 

Aurora

Serverless

(2-8ACU)

0.73

0.74

Cold starts

Aurora Serverless 'cold' and 'warm' starts for a Locust swarm of concurrent users.

Scalable but expensive when not paused.

 

High latency but zero cost when paused.

 

Not an ideal fit for public Django applications

deck

By Neal Todd

deck

  • 1,042