Hi, I'm Bobby

@yburyug

A Project Proposal

Exploring Options

Veridian Dynamics 

Veridian Dynamics has a problem. They are paying far too much to grab customer data for quoting insurance policies in their new property & casualty division through an API. They have reached an agreement with a data provider to get all the raw data they want, but they need to be able to query it via an API.

The Pieces

• Grab all the data from a server the provider puts it on
• Normalize + Preprocess it all in an ETL pipeline
• Back up all the data given from provider
• Keep data updated with new files
• Put everything in a database
• Expose it via an API

The Gotchas and Whathaveyou's

• These files are far from an RFC compliant CSV, a custom set of rules must be built up
• They range in size from a few megabytes to hundreds of gigabytes
• Update times are random
• Somehow there is no way other than FTP to get this stuff

Potential Options

Rails! The Good

• Team has background in it!
• Lots of best practices for deploying large apps
• Easy to find more team members

• Server costs will be expensive with how much data + throughput we will have
• Complicated builds for workers and processing
• Monolithic

Rails! The Bad

Go?

• Service oriented, simple small components
• Much faster
• No one on the team knows it :(
• Heavy turn from typical interpreted languages

Elixir, Phoenix, BEAM, and OTP

OTP & BEAM

• Battle tested for decades
• Amazing support for concurrency
• Wtf I get a free super fast distributed database built in? Raunch
•  Static analysis is built in? Mmmm
• Distributed communication protocols are built in?
• We are not in Kansas anymore

Elixir

• Simple elegant syntax
• Powerful but minimal standard library
• Easy to pick up the basics
• Can easily interface with Erlang, some libraries might already be here and just not be pure Elixir

Phoenix

• Mind bogglingly fast, consistently
• Architecture feels familiar
• Heavily modular
• Is getting very mature

What is the right answer?

What does this application actually have to do?

• Read and copy files from a remote server
• Listen for change on remote server
• Interface with a service to back up files
• Process and normalize files
• Interface with a database
• Make data available via REST

We have many moving parts, but the processes themselves are simple

Takeaways

When & Why to Use These Tools

When It Works

• Greenfield projects without obscure requirements
• Simple APIs
• Websockets needs
• Small services
• Heavy traffic

When It

Wont

• Super tight deadlines
• Things that cant allocate time for functions that may just be a gem drop-in with Rails
• If you cant invest time in ops and research/learning

Developer Wins

Built In Docs Makes Learning Easy

iex(1)> h String.split

                               def split(binary)

Divides a string into substrings at each Unicode whitespace occurrence with
leading and trailing whitespace ignored. Groups of whitespace are treated as a
single occurrence. Divisions do not occur on non-breaking whitespace.

Clean, Modular Code

defmodule App.CronJob do
  use GenServer

  def start_link() do
    GenServer.start_link(__MODULE__, [])
  end

  def init([]) do
    schedule_work()
    {:ok, []}
  end

  def handle_info(:work, state) do
    state = do_work(state)
    schedule_work()
    {:noreply, state}
  end

  defp do_work(state) do
    2 * 2 # or call some other module/function etc
    IO.puts "running"
  end

  defp schedule_work do
    Process.send_after(self(), :work, (60 * 1_000)) # one/minute
  end
end
GenServer.call(App.CronJob, [])

Oh, the concurrency & parallelism

:observer

Community
CommunityCommunity

Business Wins

Servers Are Expensive? Lol Nah.

Portability

From the docs:

A Competitive Edge

Paul Graham, Beating the Averages

Architecting It All

Essential Flow

• FTP Monitor listens for change
• FTP Download Worker spawned if change happens
• S3 worker backs up new files
• Splitter splits files evenly
• Preprocessor processes files
• Database Loader inserts/updates records
• API lets you query data

Umbrellas

• Many isolated apps
• Build tools make it easy
• Keep everything in one place but easily separable

Simple Modules

File Splitter

• Take in a path
• Split some files by the number of lines specified
• Write it out to a given directory

Database Loader

• Use PG_COPY to update in bulk after processing
• Handle regular updates using normal querying when new data is available

OTP Apps

Supervisors & Trees & Workers, oh my!

Periodic S3 + FTP Listeners

• List the files on FTP server or S3 Bucket, keep track of the state
• If the state changes, send a message

FTP & S3 Downloads/Uploads

• When summoned, download or upload the selected file
• Do this reliably, and do nothing else

File Preprocessor

• Our files need scrubbing. These folks did us no favors here
• Use custom rules etc to create something reasonable

Our Application

Core Ideas

• Isolated, simple components
• Single responsibilities
• Communicate via messages & leverage OTP

Deployment

For the Hobbyist:

Digital Ocean/Cloud Provider

For the Team Lacking an Ops Person:

Heroku

The Big Kid Solution:

AWS

The "I Have a Robust Ops Team" Option:

Self-Administration & Configuration

Deployment Tools

git push heroku master

Exrm

Edeliver