Handling 800krps

 

With A Team Of Three People

Your Host Tonight


@pinchito
Alex Fernández
Senior Developer at mediasmart
Organizer at MadridJS, Node.js Madrid, JSDayES

What We Will See


Setup

Architecture

Meet The Team

Engineering

Build Or Not

Conclusions

Setup

MediaSmart


We serve mobile advertising


Match bid offers with campaigns


Real time bidding (OpenRTB)


Make bid and compose ad


In near real time (~80 ms)

Guilty!



We help keep the app ecosystem alive

Some Numbers


50+ countries


Peaks of 800 krps


Average of 500 krps


45 billion requests per day


  35 million daily impressions

Traffic Peaks

Architecture


High level View

Clever Proxy


Filter out 90% of traffic
Decides which requests are less attractive


Written in Erlang (!)

Rewritten in Go (!!)


Saved about 80% of costs
Holds the grunt of the 300 krps with about 10 servers
It's an operational miracle

Pricing Server


Decides the price to pay for each bid

Intensive computations


Predictive algorithm:

estimates CTR of bid offer


Written in Go (!)


Handles queries in ~500 µs (2 ms)

Our Stack

Find Leverage

Use Your Leverage!


Discard traffic fast


Watch what you do online


Take work off hot paths


Optimize hot paths

Make Migrations Fluid


Never commit to anything!


Keep your options open


Isolate external accesses


Maintain interfaces

Meet The Team

Remote By Default


Colleagues all around the world


Constant contact on Slack


Frequent team meetings


Screen sharing essential


Priorities on Mondays

Code Reviews Work!


Improve code


Disseminate knowledge


Increase sense of ownership


Build your team


Educate

Culture


Brave to try new things


Reward initiatives


Mistakes happen


Blame-free

Failure is always an option

Postmortems


Warn operations and sales


Warn customers ASAP


Explain everything, blame nobody


Immediately and thoroughly


Issue recommendations (and follow through)

Engineering


What Is DevOps?


Amateurs talk about tactics,

but professionals study logistics.

Gen. Robert H. Barrow, USMC

noted in 1980.


Paraphrasing:

Amateurs worry about programming,

but professionals care about deployment.


Continuous Deployment


Probably the most important practice in DevOps


Continuous Integration: tested on integration env


Continuous Delivery: tested on several envs


Continuous Deployment: go all the way and deploy!


Use dark launches, feature flags, canary deployments

Deployment Infrastructure


A simple bash script to start


Built for (and with) Node.js packages


Held together with some custom Node.js scripts


Recently migrated to StriderCD


Now with a Graphical Interface!

StriderCD In Action


Monitor Everything

... But Do not Become This Guy


Build Or Not

Primitive Technology


Do not Be Afraid to Build


Starting a green field project can be exhilarating


Helps attract and keep elusive talent


Someone had to build first!

Why let them have all the fun?



Companion article:

http://pinchito.es/2016/build-or-not.html

NIH Syndrome


"Not Invented Here":

Distrust products offered by third parties


Joel Spolsky:

If it is a core business function

do it yourself, no matter what.


As easy as identifying core business functions...

which is easy, right?

Core Business Functions


Banks: insourcing IT


Google: self-driving cars


Facebook: providing Internet access


Apple: music distribution, watches

Should I Build or Not?


There is not a universal path for all destinations


Consider:

  • Are you building a state-of-the-art project?
  • Which options is more cost-effective?
  • Which option is more likely to succeed?
  • What is the cost of switching?
  • Which option is more fun?

Build Your Own Orchestrator


Nginx Balancer


Previously using Amazon's Elastic Load Balancer


Eating up to 20% of our monthly bill


Replaced by an Nginx balancer


DNS balancer in Node.js


Custom Lua logging

Filter CPU Load


Conclusions

Architecture Is Not Important


You will not get it right the first time


Make sure you can improve it later


Devise strategies to migrate


Avoid dead-ends

DevOps Is Cool


Not just install a Jenkins instance somewhere


Merge development and sysadmining


Write code to do fun stuff


Build upon your work

Build or Not?



If it gives you a competitive advantage

build it yourself, no matter what.


Thanks!


@pinchito

Handling 800k Requests/Second With A Three-Person Team

By Alex Fernández

Handling 800k Requests/Second With A Three-Person Team

Presentation for mad Scalability: https://www.meetup.com/es-ES/mad-scalability/events/238902239/

  • 3,086