Serverless Django with Zappa

DjangoCon Europe 2019

Copenhagen

Neal Todd

Serverless

Run code without thinking about servers

AWS Lambda

  • Function as a Service (FaaS)
  • Stateless
  • Full utilisation
  • Cost proportional to execution time 
  • Scale without intervention

Why Serverless Django?

  • Because it's there
  • Low traffic / experimental / personal project sites
    • Relatively easy to deploy sites
    • Low running costs
  • Production sites:
    • Potential for lower running costs
    • Automatic scalability for peaks in demand
> zappa init
███████╗ █████╗ ██████╗ ██████╗  █████╗
╚══███╔╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗
  ███╔╝ ███████║██████╔╝██████╔╝███████║
 ███╔╝  ██╔══██║██╔═══╝ ██╔═══╝ ██╔══██║
███████╗██║  ██║██║     ██║     ██║  ██║
╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝     ╚═╝  ╚═╝

Welcome to Zappa!

Zappa is a system for running server-less Python web applications on AWS Lambda and AWS API Gateway.

Rich Jones / www.gun.io

Amazon Web Services

Lambda

Route 53

Certificate Manager

S3

RDS

VPC

Satellite Ground Station

IAM

API Gateway

AWS Free Tier

  • Lambda
    • 1m requests/mth
    • 400k GB-seconds/mth
  • S3
    • 5 GB storage
    • 20k Get / 2k Put requests
  • ​RDS Micro
    • 750 hr/mth
    • 20 GB storage

512MB Lambda function

➝ One million 800ms requests per month = zero cost

But you must factor in other costs

Lift and Shift

Zappa

Settings

JSON or YML file in your project directory

 

zappa init

Zappa in action

Let's create speedrun.bygge.net

  • Wagtail Bakery Demo as a sample Django application
  • S3 bucket for static assets
  • S3 bucket for a SQLite database
  • Custom Domain Name

Full walkthrough at:

"Here's One I Made Earlier"

[[ ! -z "$SITE" ]] || exit

mkdir ${SITE} && cd ${SITE}

python3 -m venv .venv && source .venvs/bin/activate

pip install zappa

pip install zappa zappa-django-utils django-storages awscli

git clone https://github.com/wagtail/bakerydemo.git && cd bakerydemo
pip install -r requirements/base.txt

cat << EOF >> bakerydemo/settings/dev.py

DATABASES = {
    'default': {
        'ENGINE': 'zappa_django_utils.db.backends.s3sqlite',
        'NAME': '${SITE}-sqlite.db',
        'BUCKET': '${SITE}-db'
    }
}

ALLOWED_HOSTS = ['.eu-west-2.amazonaws.com', '.bygge.net']

INSTALLED_APPS += ('storages',)
AWS_STORAGE_BUCKET_NAME = '${SITE}-static'
AWS_QUERYSTRING_AUTH = False
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

DEBUG = os.getenv('DEBUG', 'off') == 'on'
EOF
cat << EOF > zappa_settings.json
{

    "dev": {
            "django_settings": "bakerydemo.settings.dev",
            "profile_name": "iam-zappa",
            "aws_region": "eu-west-2",

            "s3_bucket": "${SITE}-zappa",
            "project_name": "${SITE}",
            "runtime": "python3.6",
            "timeout_seconds": 60,

            "domain": "${SITE}.bygge.net",
            "certificate_arn": "${CERT_ARN}",
            "aws_environment_variables": {"DEBUG": "off"}
    }

}
EOF

aws s3api create-bucket --bucket ${SITE}-db --profile iam-zappa \
    --region eu-west-2 --create-bucket-configuration LocationConstraint=eu-west-2
aws s3api create-bucket --bucket ${SITE}-static --profile iam-zappa \
    --region eu-west-2 --create-bucket-configuration LocationConstraint=eu-west-2

aws s3api put-bucket-cors --bucket ${SITE}-static --profile iam-zappa --cors-configuration \
    '{"CORSRules": [{"AllowedOrigins": ["*"], "AllowedMethods": ["GET"]}]}'
aws s3api put-bucket-policy --bucket ${SITE}-static --profile iam-zappa --policy \
    '{"Statement": [{"Effect": "Allow","Principal": "*","Action": "s3:GetObject",'\
    '"Resource": "arn:aws:s3:::${SITE}-static/*"}]}'

aws s3 sync bakerydemo/media/original_images/ s3://${SITE}-static/original_images/ \
    --profile iam-zappa

zappa deploy dev

zappa certify dev -y

zappa manage dev "collectstatic --noinput"
zappa manage dev migrate
zappa manage dev load_initial_data

zappa status dev

zappa update dev

Details:

S3

API

Gateway

λ

venv

S3

Route 53 / ACM
Cloudfront

"exclude": [],
"environment_variables": {},
"aws_environment_variables": {}
zappa tail dev --disable-keep-open --since 5m
zappa invoke dev "print(os.environ)" --raw
zappa undeploy dev

AWS Roles & Policies

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ZappaStackUpdatePolicies",
            "Effect": "Allow",
            "Action": [
                "apigateway:DELETE",
                "apigateway:GET",
                "apigateway:PATCH",
                "apigateway:POST",
                "apigateway:PUT",
AdministratorAccess

 

Provides full access to AWS services and resources

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Restricting permissions with roles and policies isn't trivial

AWS Roles & Policies

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ZappaLambdaExecutionRolePolicies",
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "iam:PassRole"
            ],
            "Resource": [
                "arn:aws:iam::346164020800:role/iam-zappa-ZappaLambdaExecutionRole"
            ]
        },
        {
            "Sid": "ZappaBucketList",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::speedrun-zappa"
            ]
        },
        {
            "Sid": "ZappaBucketObjectManagement",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::speedrun-zappa/*"
        }
    ]
}
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeStackResource",
                "cloudformation:DescribeStacks",
                "cloudformation:ListStackResources",
                "cloudformation:UpdateStack",
                "events:DeleteRule",
                "events:DescribeRule",
                "events:ListRuleNamesByTarget",
                "events:ListRules",
                "events:ListTargetsByRule",
                "events:PutRule",
                "events:PutTargets",
                "events:RemoveTargets",
                "lambda:AddPermission",
                "lambda:CreateFunction",
                "lambda:DeleteFunction",
                "lambda:GetPolicy",
                "lambda:GetFunction",
                "lambda:GetFunctionConfiguration",
                "lambda:ListVersionsByFunction",
                "lambda:InvokeFunction",
                "lambda:RemovePermission",
                "lambda:UpdateFunctionCode",
                "lambda:UpdateFunctionConfiguration",
                "logs:DescribeLogStreams",
                "logs:FilterLogEvents",
                "route53:ChangeResourceRecordSets",
                "route53:GetHostedZone",
                "route53:ListHostedZones",
                "cloudfront:UpdateDistribution"
            ],
            "Resource": "*"
        }
    ]
}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::speedrun-static/*"
        },
        {
            "Sid": "AllowUserManageBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::346164020800:user/iam-zappa"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Resource": "arn:aws:s3:::speedrun-static"
        },
        {
            "Sid": "AllowUserManageBucketObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::346164020800:user/iam-zappa"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::speedrun-static/*"
        },
        {
            "Sid": "WatchtowerWagtailLoggingManagement",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "WatchtowerWagtailLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:eu-west-2:346164020800:log-group:speedrun:log-stream:*"
            ]
        }
    ]
}

Databases

RDS - Relational Database Service

  • PostgreSQL
  • MySQL & MariaDB
  • Oracle
  • Microsoft SQL Server
  • Aurora
  • Aurora Serverless

Micro instances: $14/month (free 1st year)

SQLite on S3 scales well for high-reads but not suited to high-write concurrency

Aurora Serverless: No free usage. Time & Resource based costs

PostgreSQL

import dj_database_url

DATABASES = {
    'default': dj_database_url.config()
}
+ pip install psycopg2, dj-database-url
- aws s3api create-bucket --bucket ${SITE}-db …
DATABASE_URL = postgres://USER:PASSWORD@HOST:PORT/NAME

Create Database Cluster and application Database in RDS

Get Host URL after provisioning & add to environment variables:

Adjust the setup to replace SQLite with PostgreSQL:

Aurora Serverless

Create Database Cluster, configuring Aurora Compute Units

Minimum and Maximum ACU and Pause time very important.

Defaults to 2 ACU Min and 64 ACU Max and no pause

➝ $100/month even if it does nothing

Set a pause after X minutes of inactivity

➝ Cold start (~30s)

Going "full serverless" is tempting but Aurora Serverless is suited for infrequent, intermittent, or unpredictable workloads

Not a good fit for production Django applications

VPC

Virtual Private Cloud

S3 Gateway

S3

API

Gateway

RDS

λ

IGW 

Private

Public

NAT

GW

NAT Gateway ➝ $36/mth

Aurora serverless isolated from internet -

Lambda must be in VPC

Performance under load

SQLite on S3

Relative median response time

for requests to the frontend

RDS Postgres

(Micro) 

Aurora

Serverless

(2-8ACU)

0.69

0.70

Zappa miscellany

  • Building package from a virtual environment
  • Package size limitations
  • Timeouts: Gateway vs. Lambda
  • Global deployment to all available regions
  • Scheduled functions
  • Rollbacks

Costs beyond $0 Lambda

Service Units Price per unit (eu-west-2) Free quota
Lambda GB-Second $0.00001667
Request $0.0000002 
GB of data out $0.09
RDS PostgreSQL Instance hour $0.021 (μ) / $0.164 (L) ✓ (1y)
GB-month of storage $0.133 ✓ (1y)
Aurora Serverless ACU hour $0.07
Million I/O request $0.232
GB-month of storage $0.116
S3 1,000 GET $0.00042 ✓ (1y)
1,000 PUT $0.0053 ✓ (1y)
GB of data out $0.09 ✓ (1y)
GB of storage $0.024
Route 53 Hosted Zone month $0.50
Million DNS request $0.40

Well, here's another serverless you've gotten me into.

Building a Serverless Django Application

Adam Johnson

Workshop 1

10:45

References

Zappa github.com/Miserlou/Zappa
www.zappa.io
AWS Free Tier aws.amazon.com/free
Speedrun walkthrough nealtodd.github.io/serverless-django/djconeu-speedrun/
Speedrun screencast asciinema.org/a/237320
Slide deck slides.com/nealtodd/deck
  • Icons from the Noun Project: Hamster by Dream Icons; Server by  Ralf Schmitzer;Plug by vigorn; Database by Herbert Spencer.

  • AWS  service icons copyright Amazon Web Services.

  • Stills from "The Music Box" copyright 1932 CCA.  (Watch this film!)

Performance

SQLite on S3

Relative response time

for requests to the backend

RDS Postgres

(Micro) 

Aurora

Serverless

(2-8ACU)

0.73

0.74

Cold starts

Aurora Serverless 'cold' and 'warm' starts for a Locust swarm of concurrent users.

Scalable but expensive when not paused.

 

High latency but zero cost when paused.

 

Not an ideal fit for public Django applications

deck

By Neal Todd