Three of Four Shades of Blues

Success and failure with Workflow Management Tools

Jowanza Joseph

@jowanza

Agenda

  • Introduce myself
  • Discuss the problems space
  • Explore solutions and trade-offs
  • Discuss future directions
  • Software Engineer at Pluralsight
  • Live in Salt Lake City
  • Road Biking/Model Rocketry

Problems

Responsibilities

  • Represent some flow of actions
  • Track and manage failures
  • Handle retries
  • Cron functionality
  • Work with varied services
  • Reporting

Similarities

  • Widely used
  • Large institutional support
  • Proven at very large scale
  • Good monitoring*

Counters

Gauges

Timers

  • Developed at Airbnb
  • Open Source
  • Workflows as Code
  • Python
  • Large Ecosystem
  • User Interface

Code Snippet

from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.operators.s3_key_sensor import S3KeySensor
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago
schedule = timedelta(minutes=5)
args = {
 'owner': 'airflow',
 'start_date': days_ago(1),
 'depends_on_past': False,
}
dag = DAG(
 dag_id='s3_key_sensor_demo_dag',
 schedule_interval=schedule, 
 default_args=args
)
def new_file_detection(**kwargs):
 print("A new file has arrived in s3 bucket")
 
file_sensor = S3KeySensor(
 task_id='s3_key_sensor_task',
 poke_interval=60 * 30, # (seconds); checking file every half an hour
 timeout=60 * 60 * 12, # timeout in 12 hours
 bucket_key="s3://[bucket_name]/[key]",
 bucket_name=None,
 wildcard_match=False,
 dag=dag)
print_message = PythonOperator(task_id='print_message',
 provide_context=True,
 python_callable=new_file_detection,
 dag=dag)
 
file_sensor >> print_message

DAG

UI

Deployment

Summary

Worked

  • Ecosystem
  • ETL
  • UI

Didn't Work

  • Deployment Model
  • Non-ETL Tasks
  • Infrastructure Separate*
  • Developed at the NSA
  • Open Source
  • Written in Java
  • Large Community
  • Used at Scale

A Flow

UI

Summary

Worked

  • Performance
  • Ecosystem

Didn't Work

  • Deployment Model
  • Resource Efficiency
  • JVM
  • UI
  • Amazon Proprietary Technology
  • Works with the AWS EcoSystem
  • Pay-per-use
  • Works at Scale

Example

Finite State Machine

Example

Code

{
  "Comment": "An example of the Amazon States Language for notification on an AWS Fargate task completion",
  "StartAt": "Run Fargate Task",
  "TimeoutSeconds": 3600,
  "States": {
    "Run Fargate Task": {
      "Type": "Task",
      "Resource": "arn:aws:states:::ecs:runTask.sync",
      "Parameters": {
        "LaunchType": "FARGATE",
        "Cluster": "arn:aws:ecs:ap-northeast-1:123456789012:cluster/FargateTaskNotification-ECSCluster-VHLR20IF9IMP",
        "TaskDefinition": "arn:aws:ecs:ap-northeast-1:123456789012:task-definition/FargateTaskNotification-ECSTaskDefinition-13YOJT8Z2LY5Q:1",
        "NetworkConfiguration": {
          "AwsvpcConfiguration": {
            "Subnets": [
              "subnet-07e1ad3abcfce6758",
              "subnet-04782e7f34ae3efdb"
            ],
            "AssignPublicIp": "ENABLED"
          }
        }
      },
      "Next": "Notify Success",
      "Catch": [
          {
            "ErrorEquals": [ "States.ALL" ],
            "Next": "Notify Failure"
          }
      ]
    },
    "Notify Success": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "Message": "AWS Fargate Task started by Step Functions succeeded",
        "TopicArn": "arn:aws:sns:ap-northeast-1:123456789012:FargateTaskNotification-SNSTopic-1XYW5YD5V0M7C"
      },
      "End": true
    },
    "Notify Failure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "Message": "AWS Fargate Task started by Step Functions failed",
        "TopicArn": "arn:aws:sns:ap-northeast-1:123456789012:FargateTaskNotification-SNSTopic-1XYW5YD5V0M7C"
      },
      "End": true
    }
  }
}

Pricing

$0.025 per 1,000 state transitions

Monitoring

Nested Functions

Summary

Worked

  • Easy to manage
  • Easy to compose
  • Non-programmers liked it
  • Affordable (at our scale)
  • Monitoring

Didn't Work

  • Non-AWS Ecosystem

The Future

  • More AWS Services
  • Open Sourcing some Lambdas
  • Framework

Thanks!

@jowanza

Three of Four Shades of Blues

By Jowanza Joseph

Three of Four Shades of Blues

  • 630