Amazon SWF

Simple Workflow: Ugly Duckling or Beautiful Swan?

Warren Seymour

Lead Developer, Radify


warren@radify.io


radify.io

Topics

  • Introduction to SWF
  • Case Study
  • Open Source Tools

Introduction

  • A generic solution for building distributed program workflows
  • Provides primitives to implement:
    • Scheduling
    • Concurrency
    • Dependencies
  • Takes care of:
    • Message passing
    • Locking
    • State management
  • Implemented via Deciders & Activities

High-level Summary

Input Data

Decisions & Activities

Output Data

Workflow

Workflow Execution

Decider

  • Think flowchart
  • Program that defines:
    • Execution order of processes
    • Flow of input/output between processes
    • Conditional processes (optional)
    • Concurrent processes (optional)
  • One Decider per Workflow
  • Must be 'cheap' to run

Start

firstToUpper

restToLower

concatenate

Finish

Activities

"hello World"

"Hello world"

Activities

  • Meat of the application
  • Where actual processing takes place
  • May be as 'expensive' or 'cheap' as necessary
function firstToUpper(input: String) {
  return input[0].toUpperCase();
}
function restToLower(input: String) {
  var rest = input.substring(1);
  return rest.toLowerCase();
}
function concat(parts: String[]) {
  return parts.join('');
}

Other Activites

  • Timers
    • Wait n seconds
  • Signals
    • Wait for message
  • Child Workflow
    • Execute and wait for completion
    • Recursion!

The Ugly

  • Dated Console UI
  • Limited SDKs
    • 'Flow Framework'
    • Java, Ruby only
  • CloudWatch Metrics
    • Can't count active workflow executions
  • Lack of other service integrations
    • Trigger execution from S3/SQS/SNS
    • Lambda Deciders

Case Study

The Pixwel Platform: Slurpee 2.0

Pixwel Platform Catalog

Projects

Assets

Files

Slurpee

  • Ingest System for the Pixwel Platform
  • Catalog Object Creation:
    • Assets
    • Previews
    • Files
    • Thumbnails
  • CLI-based PHP script
    • Filename Parsing
    • Platform API
    • ffmpeg
    • S3 + CloudFront
  • Strictly Serial execution

Slurpee 1.5

  • Allows select end-users to Ingest
  • Contextual Drag/Drop
  • S3 Bucket Events
  • Lambda
  • Elastic Transcoder
  • SNS
  • Broad rejection strategy
  • Operates alongside Slurpee 1.0

Slurpee 2.0

  • Replaces 1.0 and 1.5
  • Homogeneous workflow for administrators and end-users
  • Contextual Drag/Drop
  • Filename Parsing
  • Content metadata extraction
  • Concurrent, distributed Processing
  • SWF Activities and Decider
  • Autoscaling for massive workloads

Technical Benefits

  • Decision and Activity logic is strongly separated
    • Easier to add (and maintain!) complex branching and concurrency
    • Long-running activities are the norm
  • Many concepts are abstracted away
    • Message Passing
    • Retry Handling
    • Timeouts
    • Auditing/Logging
  • Fits naturally with Immutable Infrastructure

Technical Challenges

  • Limited Tooling
    • Solution: Open source library
  • Limited CloudWatch metrics
    • Solution: Lambda Function
  • No SWF Templates in CloudFormation
    • Still doing that one by hand :(
  • Scale-in safety
    • How does Autoscaling know which worker(s) to terminate?
    • Solution: Lifecycle Hooks
    • Coming Soon

github.com/radify/swfP

Write Workflows using JavaScript & Promises

  • JavaTypeScript is our new favourite thing
  • Promises for great justice
    • Expressive asynchronous code
    • Parallel execution
    • Error handling/propagation
  • Powers Slurpee 2.0
    • More features & services to come...

Decider Implementation

module.exports = (input, {Promise, activity}) =>
  Promise
    .all([
      activity('firstToUpper', input),
      activity('restToLower', input)
    ])
    .then(parts => activity('concatenate', parts));
$ swfp-decider --file=./decider.js \
  --domain=test \
  --taskList=ucFirst

Activity Implementation

module.exports = {
  firstToUpper: str => str[0].toUpperCase(),
  restToLower:  str => str.substring(1).toLowerCase(),
  concatenate:  str => str.join('')
};
$ swfp-worker --file=./activities.js \
  --domain=test \
  --taskList=ucFirst

Questions

Thanks for Listening!

Warren Seymour

 

warren@radify.io

 

radify.io

Amazon SWF: Ugly Duckling or Beautiful Swan?

By Warren Seymour

Made with Slides.com

Amazon SWF: Ugly Duckling or Beautiful Swan?

  • 1,361

More from Warren Seymour