Introduction 
to Hadoop Job Scheduling

Overview

Oozie

Ingestion

Data Quality

Azkaban

Falcon

Use Case - Analytics

Oozie

Ships with all major Hadoop distributions

Run a workflow once, or on a schedule

Supports shell scripts, MR, Hive, Pig

Workflows are described in XML

Resources are stored in HDFS

Workflows

Series of actions with success and error transitions

Each action runs in a map-only job

Files can be downloaded from HDFS

No control of where the job runs

Documentation

Coordinators

Run a flow daily/hourly/monthly

Support "nominal time" for re-execution

Can wait for a file to exist in HDFS

SLAs for start time, end time

Must specify first and last execution, may be back-dated

Documentation

Oozie - Tips

The Oozie UI is not very good

HUE is a better alternative as a dashboard

Stdout and stderr can be found in mapper logs

SLA emails aren't enabled by default

Parameterize everything!

Azkaban

Has a web server and an executor server

Each stage is a properties file

Stages have a list of dependencies

Variables can be passed from stage to stage

Resources are uploaded as a ZIP file

Documentation

Pros

• Simple to write flows
• Simple to distribute
• Simple to debug
• Powerful web UI

Cons

• Single point of failure
• Poor parameterization
• No SLAs, no input files
• Job types are useless

Apache Falcon

Provides a higher-level abstraction over Oozie

Supports replication across clusters

Automates re-running failed workflows

Handles late arrival of data

Expires data according to a retention policy

Ingestion

Sanitization

Use Case - Analytics

Data is uploaded to HDFS by a cron job

An Oozie coordinator waits for the upload

SLA emails are sent if data is late

Hive scripts are used to join data

Shell scripts are used for validation

Reports are exported as GZIPed CSVs

Tableau queries denormalized Parquet tables

Thank You!

@alanctgardner

gardner@pythian.com