Three of Four Shades of Blues
Success and failure with Workflow Management Tools
Jowanza Joseph
@jowanza
Agenda
- Introduce myself
- Discuss the problems space
- Explore solutions and trade-offs
- Discuss future directions
- Software Engineer at Pluralsight
- Live in Salt Lake City
- Road Biking/Model Rocketry
Problems
Responsibilities
- Represent some flow of actions
- Track and manage failures
- Handle retries
- Cron functionality
- Work with varied services
- Reporting
Similarities
- Widely used
- Large institutional support
- Proven at very large scale
- Good monitoring*
Counters
Gauges
Timers
- Developed at Airbnb
- Open Source
- Workflows as Code
- Python
- Large Ecosystem
- User Interface
Code Snippet
from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.operators.s3_key_sensor import S3KeySensor
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago
schedule = timedelta(minutes=5)
args = {
'owner': 'airflow',
'start_date': days_ago(1),
'depends_on_past': False,
}
dag = DAG(
dag_id='s3_key_sensor_demo_dag',
schedule_interval=schedule,
default_args=args
)
def new_file_detection(**kwargs):
print("A new file has arrived in s3 bucket")
file_sensor = S3KeySensor(
task_id='s3_key_sensor_task',
poke_interval=60 * 30, # (seconds); checking file every half an hour
timeout=60 * 60 * 12, # timeout in 12 hours
bucket_key="s3://[bucket_name]/[key]",
bucket_name=None,
wildcard_match=False,
dag=dag)
print_message = PythonOperator(task_id='print_message',
provide_context=True,
python_callable=new_file_detection,
dag=dag)
file_sensor >> print_message
DAG
UI
Deployment
Summary
Worked
- Ecosystem
- ETL
- UI
Didn't Work
- Deployment Model
- Non-ETL Tasks
- Infrastructure Separate*
- Developed at the NSA
- Open Source
- Written in Java
- Large Community
- Used at Scale
A Flow
UI
Summary
Worked
- Performance
- Ecosystem
Didn't Work
- Deployment Model
- Resource Efficiency
- JVM
- UI
- Amazon Proprietary Technology
- Works with the AWS EcoSystem
- Pay-per-use
- Works at Scale
Example
Finite State Machine
Example
Code
{
"Comment": "An example of the Amazon States Language for notification on an AWS Fargate task completion",
"StartAt": "Run Fargate Task",
"TimeoutSeconds": 3600,
"States": {
"Run Fargate Task": {
"Type": "Task",
"Resource": "arn:aws:states:::ecs:runTask.sync",
"Parameters": {
"LaunchType": "FARGATE",
"Cluster": "arn:aws:ecs:ap-northeast-1:123456789012:cluster/FargateTaskNotification-ECSCluster-VHLR20IF9IMP",
"TaskDefinition": "arn:aws:ecs:ap-northeast-1:123456789012:task-definition/FargateTaskNotification-ECSTaskDefinition-13YOJT8Z2LY5Q:1",
"NetworkConfiguration": {
"AwsvpcConfiguration": {
"Subnets": [
"subnet-07e1ad3abcfce6758",
"subnet-04782e7f34ae3efdb"
],
"AssignPublicIp": "ENABLED"
}
}
},
"Next": "Notify Success",
"Catch": [
{
"ErrorEquals": [ "States.ALL" ],
"Next": "Notify Failure"
}
]
},
"Notify Success": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"Message": "AWS Fargate Task started by Step Functions succeeded",
"TopicArn": "arn:aws:sns:ap-northeast-1:123456789012:FargateTaskNotification-SNSTopic-1XYW5YD5V0M7C"
},
"End": true
},
"Notify Failure": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"Message": "AWS Fargate Task started by Step Functions failed",
"TopicArn": "arn:aws:sns:ap-northeast-1:123456789012:FargateTaskNotification-SNSTopic-1XYW5YD5V0M7C"
},
"End": true
}
}
}
Pricing
$0.025 per 1,000 state transitions
Monitoring
Nested Functions
Summary
Worked
- Easy to manage
- Easy to compose
- Non-programmers liked it
- Affordable (at our scale)
- Monitoring
Didn't Work
- Non-AWS Ecosystem
The Future
- More AWS Services
- Open Sourcing some Lambdas
- Framework
Thanks!
@jowanza
Three of Four Shades of Blues
By Jowanza Joseph
Three of Four Shades of Blues
- 630