Serverless Scheduling

Using Python Step Functions

bene@theodo.co.uk

Ben Ellerby

@EllerbyBen

serverless-transformation

@EllerbyBen

Serverless

What is this Serverless thing?

Architectural movement
- Developers send application code which is run by the cloud provider in isolated containers abstracted from the developer.
- Use 3rd party services used to manage backend logic and state (e.g. Firebase, Cognito)
A framework with the same name

@EllerbyBen

Why Serverless?

💰 Cost reduction

👷‍♂️ #NoOps

💻 Developers focus on delivering business value

📈 More scalable

🌳 Greener

@EllerbyBen

Not just Lambda (FaaS)

Lambda

Dynamo

API Gateway

Compute

Storage

Data

API Proxy

Cognito

Auth

SQS

Queue

Step Functions

Workflows

Power and Flexibility

@EllerbyBen

FaaS: AWS Lambda

@EllerbyBen

Lambda

Runtimes

@EllerbyBen

Runtimes

@EllerbyBen

3.7, 3.6, 2.7

A Simple Function

@EllerbyBen

def handler_name(event, context): 
    ...
    return some_value

Event: The event body

dict, list, str, int, float, or NoneType

Context: Meta Data

dict

A Simple Function

@EllerbyBen

def my_handler(event, context):
    message = 'Hello {} {}!'.format(event['first_name'], 
                                    event['last_name'])  
    return { 
        'message' : message
    }

Handler Function

@EllerbyBen

A handler function is a function of your code triggered by AWS when your lambda is invoked

Serverless... Framework

@EllerbyBen

Serverless... Framework

@EllerbyBen

npm install -g serverless  

serverless create 
    --template aws-python3
    --path myService

@EllerbyBen

serverless.yml

handler.py

Handler

@EllerbyBen

import json


def hello(event, context):
    body = {
        "message": "Go Serverless v1.0! Your function executed successfully!",
        "input": event
    }

    response = {
        "statusCode": 200,
        "body": json.dumps(body)
    }

    return response

serverless.yml

@EllerbyBen


service: myService

provider:
  name: aws
  runtime: python3.7

functions:
  hello:
    handler: handler.hello

local

@EllerbyBen

➜  sls invoke local -f hello

{
    "statusCode": 200,
    "body": "{\"message\": \"Go Serverless v1.0! Your function executed successfully!\", \"input\": {}}"
}

deploy

@EllerbyBen

➜  sls deploy

➜  sls invoke -f hello

{
    "statusCode": 200,
    "body": "{\"message\": \"Go Serverless v1.0! Your function executed successfully!\", \"input\": {}}"
}

Lambda Triggers

@EllerbyBen

Scheduling Emails

@EllerbyBen

Scheduling Anything

@EllerbyBen

?

Celery

@EllerbyBen

Requires Message Broker
- e.g RabbitMQ
Distributed Task Queue
- Defines a workflow
  - e.g. task 1, task 2 + 3, task 4
- Distributed

Celery Beat

@EllerbyBen

Polls database

Celery beat is a scheduler; It kicks off tasks at regular intervals, that are then executed by available worker nodes in the cluster.

@EllerbyBen

SQL DB

App

Rabbit MQ

Celery

Beat

@EllerbyBen

SQL DB

App

Rabbit MQ

Celery

Beat

@EllerbyBen

Step Functions

@EllerbyBen

Step Functions

AWS Step Functions lets you coordinate multiple AWS services into serverless workflows so you can build and update apps quickly.

@EllerbyBen

Step Functions

@EllerbyBen

Cloudformation

AWSTemplateFormatVersion: '2010-09-09'
Description: An example template for a Step Functions state machine.
Resources:
  MyStateMachine:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      StateMachineName: HelloWorld-StateMachine
      DefinitionString: |-
        {
          "StartAt": "HelloWorld",
          "States": {
            "HelloWorld": {
              "Type": "Task",
              "Resource": "ARN of Function.....",
              "End": true
            }
          }
        }
      RoleArn: arn-of-role
      Tags:
        -
          Key: "keyname1"
          Value: "value1"
        -
          Key: "keyname2"
          Value: "value2"

@EllerbyBen

serverless-step-functions

@EllerbyBen

serverless-step-functions

stepFunctions:
  stateMachines:
    MyStateMachine:
      name: MyMachine
      definition:
        Comment: "Does something cool"
        States:
          HelloWorld:
            Type: Task
            Resource: "HelloLambdaARN"
            End: true

@EllerbyBen

FIFO

Date Queue

+

@EllerbyBen

First In

Date Out

+

@EllerbyBen

FIDO

@EllerbyBen

Start

Wait

Push

End

@EllerbyBen

Start

Wait

Push

End

@EllerbyBen

You are charged based on the number of state transitions required to execute your application.

Start

Wait

Push

End

12/10/2019

push_email

send_email

schedule_email

@EllerbyBen

stepFunctions:
  stateMachines:
    EmailFSM:
      name: EmailScheduling
      definition:
        Comment: "Schedules an email"
        StartAt: WaitForDueDate
        States:
          WaitForDueDate:
            Type: Wait
            TimestampPath: $.dueDate
            Next: PushEmail
          PushEmail:
            Type: Task
            Resource: "arn-of-function"
            End: true

FSM

@EllerbyBen

service: myService

provider:
  name: aws
  runtime: python3.7

functions:
  hello:
    handler: handler.send_email
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - EmailQueue
              - Arn
resources:
  Resources:
    EmailQueue:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "EmailQueue${opt:infraStackName}"

SQS

@EllerbyBen

{
    "dueDate": "2019-03-26T00:00:00.000Z",
    "from ": "bene@theodo.co.uk",
    "to": "pycon@theodo.co.uk",
    "subject": "serverless"
}

event

@EllerbyBen

def schedule_email(event, context):

    sfn = boto3.client('stepfunctions')

    sfn_response = sfn.start_execution(
        stateMachineArn=os.environ.get('SFN_ARN'),
        name=event.schedule,
        input=event['body']
    )

    response = {
        "statusCode": 200,
        "body": event['body']
    }

    return response

schedule_email

@EllerbyBen

def push_email(event, context):
    client = boto3.client('sqs')

    sqs_event = client.send_message(
        QueueUrl=os.environ['SQS_URL'],
        MessageBody=event['body']
    )

    response = {
        "statusCode": 200,
        "body": "EVENT PUSHED"
    }

    return response

push_email

@EllerbyBen

def send_email(event, context):
    message = Mail(
        from_email=event['body']['from'],
        to_emails=event['body']['to'],
        subject=event['body']['subject'],
        html_content='Hello PyCon!'
    )

    sg = SendGridAPIClient(os.environ.get('SENDGRID_API_KEY'))
    response = sg.send(message)
    print(response.status_code)
    print(response.body)
    print(response.headers)


    response = {
        "statusCode": 200,
        "body": "Email Sent"
    }

    return response

send_email

Conclusion

Step functions provide a nice way to create workflows

AWS provides good serverless services for python

The serverless framework with plugins can manage event scheduling

@EllerbyBen

serverless-transformation

@EllerbyBen

npm install -g sls-dev-tools

@EllerbyBen

Serverless Scheduling - Using Python Step Functions in AWS

By Ben Ellerby

Serverless Scheduling - Using Python Step Functions in AWS

Talk given at the AWS Community Summit.

1,000

Serverless Scheduling

Using Python Step Functions

Serverless

What is this Serverless thing?

Why Serverless?

Not just Lambda (FaaS)

Power and Flexibility

FaaS: AWS Lambda

Lambda

Runtimes

Runtimes

A Simple Function

A Simple Function

Handler Function

Serverless... Framework

Serverless... Framework

Handler

serverless.yml

local

deploy

Lambda Triggers

Scheduling Emails

Scheduling Anything

?

Celery

Celery Beat

Step Functions

Step Functions

Step Functions

Cloudformation

serverless-step-functions

serverless-step-functions

FIFO

Date Queue

+

First In

Date Out

+

FIDO

Conclusion

Serverless Scheduling - Using Python Step Functions in AWS

More from Ben Ellerby