Richard Nixon & RFC 768

Musings On Matters Elixir, OTP, BEAM, and Phoenix that I couldn't think of a name for that doesn't involve a disgraced president or UDP. Explanation coming soon.

Hi, I'm Bobby

@yburyug

What This Talk Is

 

- Designing a large Elixir/Phoenix application

- Deploying it

- The business case for Elixir

- The developer case for Elixir

- A look at when its the best tool

What This Talk Is Not

- Extremely technical at an OTP/micro level

- A complete solution to all your woes

- A beckoning for us all to use these new tools with no real reason

A Project Proposal

Exploring Options

Veridian Dynamics 

Veridian Dynamics has a problem. They are paying far too much to grab customer data for quoting insurance policies in their new property & casualty division through an API. They have reached an agreement with a data provider to get all the raw data they want, but they need to be able to query it via an API.

The Pieces

  • Grab all the data from a server the provider puts it on
  • Normalize + Preprocess it all in an ETL pipeline
  • Back up all the data given from provider
  • Keep data updated with new files
  • Put everything in a database
  • Expose it via an API

The Gotchas and Whathaveyou's

  • These files are far from an RFC compliant CSV, a custom set of rules must be built up
  • They range in size from a few megabytes to hundreds of gigabytes
  • Update times are random
  • Somehow there is no way other than FTP to get this stuff

Potential Options

Rails! The Good

  • Team has background in it!
  • Lots of best practices for deploying large apps
  • Easy to find more team members
  • Server costs will be expensive with how much data + throughput we will have
  • Complicated builds for workers and processing
  • Monolithic

Rails! The Bad

Go?

 

  • Service oriented, simple small components
  • Much faster
  • No one on the team knows it :(
  • Heavy turn from typical interpreted languages

Elixir, Phoenix, BEAM, and OTP

OTP & BEAM

  • Battle tested for decades
  • Amazing support for concurrency
  • Wtf I get a free super fast distributed database built in? Raunch
  •  Static analysis is built in? Mmmm
  • Distributed communication protocols are built in?
  • We are not in Kansas anymore

Elixir

  • Simple elegant syntax
  • Powerful but minimal standard library
  • Easy to pick up the basics
  • Can easily interface with Erlang, some libraries might already be here and just not be pure Elixir

Phoenix

  • Mind bogglingly fast, consistently
  • Architecture feels familiar
  • Heavily modular
  • Is getting very mature

What is the right answer?

What does this application actually have to do?

  • Read and copy files from a remote server
  • Listen for change on remote server
  • Interface with a service to back up files
  • Process and normalize files
  • Interface with a database
  • Make data available via REST

We have many moving parts, but the processes themselves are simple

Takeaways

When & Why to Use These Tools

When It Works

  • Greenfield projects without obscure requirements
  • Simple APIs
  • Websockets needs
  • Small services
  • Heavy traffic

When It

Wont

  • Super tight deadlines
  • Things that cant allocate time for functions that may just be a gem drop-in with Rails
  • If you cant invest time in ops and research/learning

Developer Wins

Built In Docs Makes Learning Easy

iex(1)> h String.split

                               def split(binary)

Divides a string into substrings at each Unicode whitespace occurrence with
leading and trailing whitespace ignored. Groups of whitespace are treated as a
single occurrence. Divisions do not occur on non-breaking whitespace.

Clean, Modular Code

defmodule App.CronJob do
  use GenServer

  def start_link() do
    GenServer.start_link(__MODULE__, [])
  end

  def init([]) do
    schedule_work()
    {:ok, []}
  end

  def handle_info(:work, state) do
    state = do_work(state)
    schedule_work()
    {:noreply, state}
  end

  defp do_work(state) do
    2 * 2 # or call some other module/function etc
    IO.puts "running"
  end

  defp schedule_work do
    Process.send_after(self(), :work, (60 * 1_000)) # one/minute
  end
end
GenServer.call(App.CronJob, [])

Oh, the concurrency & parallelism

:observer

Community
CommunityCommunity

Business Wins

Servers Are Expensive? Lol Nah.

Portability

From the docs:

A Competitive Edge

Paul Graham, Beating the Averages

Architecting It All

Essential Flow

  • FTP Monitor listens for change
  • FTP Download Worker spawned if change happens
  • S3 worker backs up new files
  • Splitter splits files evenly
  • Preprocessor processes files
  • Database Loader inserts/updates records
  • API lets you query data

Umbrellas

  • Many isolated apps
  • Build tools make it easy
  • Keep everything in one place but easily separable

Simple Modules

File Splitter

  • Take in a path
  • Split some files by the number of lines specified
  • Write it out to a given directory

Database Loader

  • Use PG_COPY to update in bulk after processing
  • Handle regular updates using normal querying when new data is available

OTP Apps

Supervisors & Trees & Workers, oh my!

Periodic S3 + FTP Listeners

  • List the files on FTP server or S3 Bucket, keep track of the state
  • If the state changes, send a message

FTP & S3 Downloads/Uploads

  • When summoned, download or upload the selected file
  • Do this reliably, and do nothing else

File Preprocessor

  • Our files need scrubbing. These folks did us no favors here
  • Use custom rules etc to create something reasonable

Phoenix API

  • A simple search endpoint for first/last name

Our Application

Core Ideas

  • Isolated, simple components
  • Single responsibilities
  • Communicate via messages & leverage OTP

Deployment

For the Hobbyist:

Digital Ocean/Cloud Provider

For the Team Lacking an Ops Person:

Heroku

The Big Kid Solution:

AWS

The "I Have a Robust Ops Team" Option:

Self-Administration & Configuration

Deployment Tools

git push heroku master

Exrm

Edeliver

Woes Along The Way

Also:
 

An Idea for a Community Initiative

Q & A

Richard Nixon & RFC 768

By Bobby Grayson

Richard Nixon & RFC 768

  • 1,778