Feeding locusts at

🦗 1K QPS 🦗

per restaurant

(while serving quality food)

Guillaume Gelin 

ramnes.eu 🇪🇺 🇫🇷 


😊​


08:30 AM

Bob wakes up.

💺🖥️
🤗🖥️
💺🖥️
💺🖥️
09:00 AM

Bob arrives at his restaurant.

He is the one taking orders.

💺🖥️
😁🖥️🦗
💺🖥️
💺🖥️
10:00 AM

The first client comes in!

💺🖥️
🙂🖥️🦗🦗
💺🖥️
💺🖥️
11:00 AM

A queue starts.

💺🖥️
😱🖥️🦗🦗🦗🦗🦗🦗🦗🦗
💺🖥️
💺🖥️
11:30 AM

Locusts are hungry early.

🙂🖥️🦗🦗
😅🖥️🦗🦗
🙂🖥️🦗🦗
🙂🖥️🦗🦗
12:00 AM

Coworkers arrive! Fiuh.

12:30 AM

Queues are full: global panic!

😱🖥️🦗🦗🦗🦗🦗🦗🦗🦗
😭🖥️🦗🦗🦗🦗🦗🦗🦗🦗
😱🖥️🦗🦗🦗🦗🦗🦗🦗🦗
😱🖥️🦗🦗🦗🦗🦗🦗🦗🦗
12:35 AM

Crisis meeting

🤔
🤔  🤔
🤔

​

12:40 AM

Bob has an idea...

🤔
😀  🤔
🤔

​

12:41 AM

What if we used robots?


😀🤖

​

😎🤖🦗🦗
😍🤖🦗🦗
😎🤖🦗🦗
😎🤖🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
12:45 AM

Locusts are served much faster,
the queue starts to disappear

13:00 AM

But wait, no!
More and more locusts are coming in...

😱🤖🦗🦗🦗🦗🦗🦗🦗🦗
😭🤖🦗🦗🦗🦗🦗🦗🦗🦗
😱🤖🦗🦗🦗🦗🦗🦗🦗🦗
😱🤖🦗🦗🦗🦗🦗🦗🦗🦗

Scaling Python to

1K QPS

per server

(not doing Hello Worlds)

Quick summary

🦗 = ?
💺🖥️ = ?  
😊​ = ?
🤖 = ?

It's simple until you

make it complicated.

Step 0

Have something to scale

Have something to scale

😊

  • 4 CPU - 16 GB

  • Nginx :80 → :5000

  • Flask

  • SQLite

  • Flask-SQLAlchemy

  • Flask-WTF

  • Flask-Bcrypt

  • Register

  • Log in and log out

  • List tweets

  • Write new tweets

  • List all users

  • Follow and unfollow

Step 1

Identify threats and
define a protocol

Indentify threats

Reads

  • List tweets

  • List all users

Writes

  • Register

  • Log in and log out

  • Write new tweets

  • Follow and unfollow

Reads

  • List tweets

  • List all users

Writes

  • Register

  • Log in and log out

  • Write new tweets

  • Follow and unfollow

Define a protocol

  • Only one server (no horizontal scaling allowed)

  • Response time's 95% percentile should stay under 2 seconds

  • Failure rate should stay at 0%

  • Register 1x

  • Log in and log out 2x

  • List tweets 200x

  • Write new tweets 20x

  • List all users 5x

  • Follow and unfollow 5x

https://locust.io

$ python run.py

50 QPS

Step 2

Parallelize

https://gunicorn.org

https://uwsgi-docs.rtfd.io

$ uwsgi --http :5000 --processes 4 --file run.py --callable app
$ gunicorn --bind :5000 --workers 4 run:app
$ cat tweepy.ini
[uwsgi]
master = true
protocol = http
socket = 0.0.0.0:5000
file = run.py
callable = app
processes = 4

$ uwsgi tweepy.ini

180 QPS

Stateful versus stateless

Step 3

Check the database usage

Leverage RAM

Paging is hard

but necessary

Explain

Transaction logs

250 QPS

Step 4

Cache reads

There are only two hard things in computer science: cache invalidation

and naming things.

tweets_cache = {}


@app.route("/tweets", methods=["GET"])
def get_tweets():
    ...
    tweets = tweets_cache.get(user_id)
    if not tweets:
        tweets = query_tweets(...)
        tweets_cache[user_id] = tweets
    ...


@app.route("/tweets", methods=["POST"])
def post_tweets():
    ...
    tweet.save()
    tweets_cache.pop(user_id)
    ...

@cache()
def query_tweets(user_id, ...):
    ...


@app.route("/tweets", methods=["GET"])
def get_tweets():
    ...
    query_tweets(user_id, ...)
    ...


@app.route("/tweets", methods=["POST"])
def post_tweets():
    ...
    query_tweets.invalidate(user_id)
    ...

But, the application is stateful now?

http://memcached.org

https://redis.io

350 QPS

Step 5

Achieve concurrency

https://gevent.org

In the future:

asyncio + ASGI

https://github.com/encode/uvicorn

https://github.com/django/daphne

http://channels.rtfd.io

https://gitlab.com/pgjones/hypercorn

https://gitlab.com/pgjones/quart

CPU versus I/O

Databases writes

550 QPS

Step 6

???

😱🤖🦗🦗🦗🦗🦗🦗🦗🦗
😭🤖🦗🦗🦗🦗🦗🦗🦗🦗
😱🤖🦗🦗🦗🦗🦗🦗🦗🦗
😱🤖🦗🦗🦗🦗🦗🦗🦗🦗

Step 6

Scale vertically!

😎🤖🦗🦗
😎🤖🦗🦗
😎🤖🦗🦗
😎🤖🦗🦗
😎🤖🦗🦗
😎🤖🦗🦗
😎🤖🦗🦗
😎🤖🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗🦗🦗
🦗🦗
🦗🦗
🦗🦗

 

🦗🦗
🦗🦗
🦗🦗
🦗🦗🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗
🦗🦗🦗🦗
🦗🦗
🦗🦗
🦗🦗

1K QPS

But remember...

Thank you!

https://slides.com/ramnes/scale-1k

https://github.com/ramnes/tweepy

https://github.com/numberly

Made with Slides.com