Handling 800krps

With A Team Of Three People
Your Host Tonight

@pinchito
Alex Fernández
Senior Developer at mediasmart
Organizer at MadridJS, Node.js Madrid, JSDayES
What We Will See
Setup
Architecture
Meet The Team
Engineering
Build Or Not
Conclusions
Setup

MediaSmart
We serve mobile advertising
Match bid offers with campaigns
Real time bidding (OpenRTB)
Make bid and compose ad
In near real time (~80 ms)
Guilty!

We help keep the app ecosystem alive
Some Numbers
50+ countries
Peaks of 800 krps
Average of 500 krps
45 billion requests per day
35 million daily impressions
Traffic Peaks

Architecture

High level View

Clever Proxy
Filter out 90% of traffic
Decides which requests are less attractive
Written in Erlang (!)
Rewritten in Go (!!)
Saved about 80% of costs
Holds the grunt of the 300 krps with about 10 servers
It's an operational miracle
Pricing Server
Decides the price to pay for each bid
Intensive computations
Predictive algorithm:
estimates CTR of bid offer
Written in Go (!)
Handles queries in ~500 µs (2 ms)
Our Stack
Find Leverage

Use Your Leverage!
Discard traffic fast
Watch what you do online
Take work off hot paths
Optimize hot paths
Make Migrations Fluid
Never commit to anything!
Keep your options open
Isolate external accesses
Maintain interfaces
Meet The Team

Remote By Default
Colleagues all around the world
Constant contact on Slack
Frequent team meetings
Screen sharing essential
Priorities on Mondays
Code Reviews Work!
Improve code
Disseminate knowledge
Increase sense of ownership
Build your team
Educate
Culture
Brave to try new things
Reward initiatives
Mistakes happen
Blame-free
Failure is always an option

Postmortems
Warn operations and sales
Warn customers ASAP
Explain everything, blame nobody
Immediately and thoroughly
Issue recommendations (and follow through)
Engineering

What Is DevOps?
Amateurs talk about tactics,
but professionals study logistics.
Gen. Robert H. Barrow, USMC
Paraphrasing:
Amateurs worry about programming,
but professionals care about deployment.
Continuous Deployment
Probably the most important practice in DevOps
Continuous Integration: tested on integration env
Continuous Delivery: tested on several envs
Continuous Deployment: go all the way and deploy!
Deployment Infrastructure
A simple bash script to start
Built for (and with) Node.js packages
Held together with some custom Node.js scripts
Recently
migrated to StriderCD
Now with a Graphical Interface!
StriderCD In Action

Monitor Everything

... But Do not Become This Guy

Build Or Not

Primitive Technology
Do not Be Afraid to Build
Starting a green field project can be exhilarating
Helps attract and keep elusive talent
Someone had to build first!
Why let them have all the fun?
Companion article:
NIH Syndrome
"Not Invented Here":
Distrust products offered by third parties
If it is a core business function
do it yourself, no matter what.
As easy as identifying core business functions...
which is easy, right?
Core Business Functions
Banks: insourcing IT
Google: self-driving cars
Facebook: providing Internet access
Apple: music distribution, watches
Should I Build or Not?
There is not a universal path for all destinations
Consider:
- Are you building a state-of-the-art project?
- Which options is more cost-effective?
- Which option is more likely to succeed?
- What is the cost of switching?
- Which option is more fun?
Build Your Own Orchestrator
Nginx Balancer
Previously using Amazon's Elastic Load Balancer
Eating up to 20% of our monthly bill
Replaced by an Nginx balancer
DNS balancer in Node.js
Custom Lua logging
Filter CPU Load

Conclusions

Architecture Is Not Important
You will not get it right the first time
Make sure you can improve it later
Devise strategies to migrate
Avoid dead-ends
DevOps Is Cool
Not just install a Jenkins instance somewhere
Merge development and sysadmining
Write code to do fun stuff
Build upon your work
Build or Not?
If it gives you a competitive advantage
build it yourself, no matter what.
Thanks!

@pinchito
Handling 800k Requests/Second With A Three-Person Team
By Alex Fernández
Handling 800k Requests/Second With A Three-Person Team
Presentation for mad Scalability: https://www.meetup.com/es-ES/mad-scalability/events/238902239/
- 2,881