Open Source AWS Athena 

Jowanza Joseph

jowanza.com

@jowanza

Agenda

  • The Problem
  • AWS Athena
  • OpenStack Swift
  • File Formats
  • PrestoDB
  • Architecture
  • Failure
  • New Architecture
  • Demo

Most Nights

About Me

  • Spark/ Scala
  • Too many opinions
  • Writing a book about Spark
  • Dad

Multi-Tenant Distributed Data Platform

How AWS Athena Works

Replicating

  1. Object store
  2. File serialization
  3. Distributed data processing
  4. Bonus: JDBC
  5. Bonus: On Demand

Step 1. Object Storage

Swift Architecture

Step 2. Presto

Presto Architecture

Presto Benefits

  • Can use s3 as a Hive store
  • Distributed
  • Well Supported

Step 3. File Serialization

Summary

All Together

Recap

It Sucks

  • OpenStack is not Cloud Native
  • Swift is a bad Hive store
  • Swift is not at parity with s3
  • Presto doesn't work well with Swift
  • Presto is not Cloud Native

The Struggles

Minio

Benefits

  • Cloud Native
  • Tiny
  • S3 Compliant
  • Solid Ecosystem

Minio Architecture

Updated Architecture

Create New Table

CREATE TABLE hive.web.request_logs (
  sku varchar,
  pageviews double,
  conversion_rate double
)
WITH (
  format = 'ocr',
  external_location = 's3://databucked/data.orc'
)

Demo

Resources

Made with Slides.com