Handling 800krps

With A Team Of Three People

Your Host Tonight

What We Will See

Setup

Architecture

Meet The Team

Engineering

Build Or Not

Conclusions

Setup

MediaSmart

We serve mobile advertising

Match bid offers with campaigns

Real time bidding (OpenRTB)

Make bid and compose ad

In near real time (~80 ms)

Guilty!

We help keep the app ecosystem alive

Some Numbers

50+ countries

Peaks of 800 krps

Average of 500 krps

45 billion requests per day

  35 million daily impressions

Traffic Peaks

Architecture

High level View

Clever Proxy

Filter out 90% of traffic
Decides which requests are less attractive

Written in Erlang (!)

Rewritten in Go (!!)

Saved about 80% of costs
Holds the grunt of the 300 krps with about 10 servers
It's an operational miracle

Pricing Server

Decides the price to pay for each bid

Intensive computations

Predictive algorithm:

estimates CTR of bid offer

Written in Go (!)

Handles queries in ~500 µs (2 ms)

Our Stack

Find Leverage

Use Your Leverage!

Discard traffic fast

Watch what you do online

Take work off hot paths

Optimize hot paths

Make Migrations Fluid

Never commit to anything!

Keep your options open

Isolate external accesses

Maintain interfaces

Meet The Team

Remote By Default

Colleagues all around the world

Constant contact on Slack

Frequent team meetings

Screen sharing essential

Priorities on Mondays

Code Reviews Work!

Improve code

Disseminate knowledge

Increase sense of ownership

Build your team

Educate

Culture

Brave to try new things

Reward initiatives

Mistakes happen

Blame-free

Failure is always an option

Postmortems

Warn operations and sales

Warn customers ASAP

Explain everything, blame nobody

Immediately and thoroughly

Issue recommendations (and follow through)

Engineering

What Is DevOps?

Amateurs talk about tactics,

but professionals study logistics.

Gen. Robert H. Barrow, USMC

noted in 1980.

Paraphrasing:

Amateurs worry about programming,

but professionals care about deployment.

Continuous Deployment

Probably the most important practice in DevOps

Continuous Integration: tested on integration env

Continuous Delivery: tested on several envs

Continuous Deployment: go all the way and deploy!

Use dark launches, feature flags, canary deployments

Deployment Infrastructure

A simple bash script to start

Built for (and with) Node.js packages

Held together with some custom Node.js scripts

Recently migrated to StriderCD

Now with a Graphical Interface!

StriderCD In Action

Monitor Everything

... But Do not Become This Guy

Build Or Not

Primitive Technology

Do not Be Afraid to Build

Starting a green field project can be exhilarating

Helps attract and keep elusive talent

Someone had to build first!

Why let them have all the fun?

Companion article:

http://pinchito.es/2016/build-or-not.html

NIH Syndrome

"Not Invented Here":

Distrust products offered by third parties

Joel Spolsky:

If it is a core business function

do it yourself, no matter what.

As easy as identifying core business functions...

which is easy, right?

Core Business Functions

Banks: insourcing IT

Google: self-driving cars

Facebook: providing Internet access

Apple: music distribution, watches

Should I Build or Not?

There is not a universal path for all destinations

Consider:

• Are you building a state-of-the-art project?
• Which options is more cost-effective?
• Which option is more likely to succeed?
• What is the cost of switching?
  
• Which option is more fun?
  

Build Your Own Orchestrator

Nginx Balancer

Previously using Amazon's Elastic Load Balancer

Eating up to 20% of our monthly bill

Replaced by an Nginx balancer

DNS balancer in Node.js

Custom Lua logging

Filter CPU Load

Conclusions

Architecture Is Not Important

You will not get it right the first time

Make sure you can improve it later

Devise strategies to migrate

Avoid dead-ends

DevOps Is Cool

Not just install a Jenkins instance somewhere

Merge development and sysadmining

Write code to do fun stuff

Build upon your work

Build or Not?

If it gives you a competitive advantage

build it yourself, no matter what.

Thanks!