Serverless Scheduling

Using Python Step Functions

bene@theodo.co.uk

Ben Ellerby

@EllerbyBen

serverless-transformation

@EllerbyBen

Serverless

What is this Serverless thing?

Architectural movement
- Developers send application code which is run by the cloud provider in isolated containers abstracted from the developer.
- Use 3rd party services used to manage backend logic and state (e.g. Firebase, Cognito)
A framework with the same name

@EllerbyBen

Why Serverless?

💰 Cost reduction

👷‍♂️ #NoOps

💻 Developers focus on delivering business value

📈 More scalable

🌳 Greener

@EllerbyBen

Not just Lambda (FaaS)

Lambda

Dynamo

API Gateway

Compute

Storage

Data

API Proxy

Cognito

Auth

SQS

Queue

Step Functions

Workflows

Power and Flexibility

@EllerbyBen

FaaS: AWS Lambda

@EllerbyBen

Lambda

Runtimes

@EllerbyBen

Runtimes

@EllerbyBen

3.7, 3.6, 2.7

A Simple Function

@EllerbyBen

def handler_name(event, context): 
    ...
    return some_value

Event: The event body

dict, list, str, int, float, or NoneType

Context: Meta Data

dict

A Simple Function

@EllerbyBen

def my_handler(event, context):
    message = 'Hello {} {}!'.format(event['first_name'], 
                                    event['last_name'])  
    return { 
        'message' : message
    }

Handler Function

@EllerbyBen

A handler function is a function of your code triggered by AWS when your lambda is invoked

Serverless... Framework

@EllerbyBen

Serverless... Framework

@EllerbyBen

npm install -g serverless  

serverless create 
    --template aws-python3
    --path myService

@EllerbyBen

serverless.yml

handler.py

Handler

@EllerbyBen

import json


def hello(event, context):
    body = {
        "message": "Go Serverless v1.0! Your function executed successfully!",
        "input": event
    }

    response = {
        "statusCode": 200,
        "body": json.dumps(body)
    }

    return response

serverless.yml

@EllerbyBen


service: myService

provider:
  name: aws
  runtime: python3.7

functions:
  hello:
    handler: handler.hello

local

@EllerbyBen

➜  sls invoke local -f hello

{
    "statusCode": 200,
    "body": "{\"message\": \"Go Serverless v1.0! Your function executed successfully!\", \"input\": {}}"
}

deploy

@EllerbyBen

➜  sls deploy

➜  sls invoke -f hello

{
    "statusCode": 200,
    "body": "{\"message\": \"Go Serverless v1.0! Your function executed successfully!\", \"input\": {}}"
}

Lambda Triggers

@EllerbyBen

Scheduling Emails

@EllerbyBen

Scheduling Anything

@EllerbyBen

?

Celery

@EllerbyBen

Requires Message Broker
- e.g RabbitMQ
Distributed Task Queue
- Defines a workflow
  - e.g. task 1, task 2 + 3, task 4
- Distributed

Celery Beat

@EllerbyBen

Polls database

Celery beat is a scheduler; It kicks off tasks at regular intervals, that are then executed by available worker nodes in the cluster.

@EllerbyBen

SQL DB

App

Rabbit MQ

Celery

Beat

@EllerbyBen

SQL DB

App

Rabbit MQ

Celery

Beat

@EllerbyBen

Step Functions

@EllerbyBen

Step Functions

AWS Step Functions lets you coordinate multiple AWS services into serverless workflows so you can build and update apps quickly.

@EllerbyBen

Step Functions

@EllerbyBen

Cloudformation

AWSTemplateFormatVersion: '2010-09-09'
Description: An example template for a Step Functions state machine.
Resources:
  MyStateMachine:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      StateMachineName: HelloWorld-StateMachine
      DefinitionString: |-
        {
          "StartAt": "HelloWorld",
          "States": {
            "HelloWorld": {
              "Type": "Task",
              "Resource": "ARN of Function.....",
              "End": true
            }
          }
        }
      RoleArn: arn-of-role
      Tags:
        -
          Key: "keyname1"
          Value: "value1"
        -
          Key: "keyname2"
          Value: "value2"

@EllerbyBen

serverless-step-functions

@EllerbyBen

serverless-step-functions

stepFunctions:
  stateMachines:
    MyStateMachine:
      name: MyMachine
      definition:
        Comment: "Does something cool"
        States:
          HelloWorld:
            Type: Task
            Resource: "HelloLambdaARN"
            End: true

@EllerbyBen

FIFO

Date Queue

+

@EllerbyBen

First In

Date Out

+

@EllerbyBen

FIDO

@EllerbyBen

Start

Wait

Push

End

@EllerbyBen

Start

Wait

Push

End

@EllerbyBen

You are charged based on the number of state transitions required to execute your application.

Start

Wait

Push

End

12/10/2019

push_email

send_email

schedule_email

@EllerbyBen

stepFunctions:
  stateMachines:
    EmailFSM:
      name: EmailScheduling
      definition:
        Comment: "Schedules an email"
        StartAt: WaitForDueDate
        States:
          WaitForDueDate:
            Type: Wait
            TimestampPath: $.dueDate
            Next: PushEmail
          PushEmail:
            Type: Task
            Resource: "arn-of-function"
            End: true

FSM

@EllerbyBen

service: myService

provider:
  name: aws
  runtime: python3.7

functions:
  hello:
    handler: handler.send_email
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - EmailQueue
              - Arn
resources:
  Resources:
    EmailQueue:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "EmailQueue${opt:infraStackName}"

SQS

@EllerbyBen

{
    "dueDate": "2019-03-26T00:00:00.000Z",
    "from ": "bene@theodo.co.uk",
    "to": "pycon@theodo.co.uk",
    "subject": "serverless"
}

event

@EllerbyBen

def schedule_email(event, context):

    sfn = boto3.client('stepfunctions')

    sfn_response = sfn.start_execution(
        stateMachineArn=os.environ.get('SFN_ARN'),
        name=event.schedule,
        input=event['body']
    )

    response = {
        "statusCode": 200,
        "body": event['body']
    }

    return response

schedule_email

@EllerbyBen

def push_email(event, context):
    client = boto3.client('sqs')

    sqs_event = client.send_message(
        QueueUrl=os.environ['SQS_URL'],
        MessageBody=event['body']
    )

    response = {
        "statusCode": 200,
        "body": "EVENT PUSHED"
    }

    return response

push_email

@EllerbyBen

def send_email(event, context):
    message = Mail(
        from_email=event['body']['from'],
        to_emails=event['body']['to'],
        subject=event['body']['subject'],
        html_content='Hello PyCon!'
    )

    sg = SendGridAPIClient(os.environ.get('SENDGRID_API_KEY'))
    response = sg.send(message)
    print(response.status_code)
    print(response.body)
    print(response.headers)


    response = {
        "statusCode": 200,
        "body": "Email Sent"
    }

    return response

send_email

Conclusion

Step functions provide a nice way to create workflows

AWS provides good serverless services for python

The serverless framework with plugins can manage event scheduling

@EllerbyBen

serverless-transformation

@EllerbyBen

npm install -g sls-dev-tools

@EllerbyBen

Serverless Scheduling

Using Python Step Functions

Serverless

What is this Serverless thing?

Why Serverless?

Not just Lambda (FaaS)

Power and Flexibility

FaaS: AWS Lambda

Lambda

Runtimes

Runtimes

A Simple Function

A Simple Function

Handler Function

Serverless... Framework

Serverless... Framework

Handler

serverless.yml

local

deploy

Lambda Triggers

Scheduling Emails

Scheduling Anything

?

Celery

Celery Beat

Step Functions

Step Functions

Step Functions

Cloudformation

serverless-step-functions

serverless-step-functions

FIFO

Date Queue

+

First In

Date Out

+

FIDO

Conclusion

Serverless Scheduling - Using Python Step Functions in AWS

More from Ben Ellerby