Performance in Python applications

@woakas

Ubidots

Gustavo Angulo

February 2020

woakas

Coffee / 5-7

13 years

2 sons

CTO / Co-founder

Ubidots

Ubidots

+ 40k Developers

+ 30 M per day

The problem

  • Slowness
  • Downs
  • I don't know what's happening
  • Bad experience for users

The source

Bad code
Database

Physical

Language

Human

Bad code

from flask import Flask, escape, request
import time

app = Flask(__name__)

@app.route("/add")
def add():
    ## Adding some time
    time.sleep(1) 
    a = float(request.args.get("a", 0))
    b = float(request.args.get("b", 0))
    return "{}".format(a + b)

WRK

# lua
request = function()
    wrk.method = "GET"
    return wrk.format(
        "GET", '/add?a=' .. math.random(0, 100) .. '&b=' .. math.random(0, 100)
    )
end


# bash
$ wrk -s add_random.lua -c 50 -t 1 -d 10 http://localhost:5000

# Flask
FLASK_APP=slow_code.py flask run --reload --without-threads

Big O notation

https://www.geeksforgeeks.org/analysis-algorithms-big-o-analysis/

Bad code

from flask import Flask, escape, request

app = Flask(__name__)

@app.route("/add/cube")
def add():
    t = 0
    mn, mx = (0, 100)
    a = int(request.args.get("size", 0))
    for i in range(a):
        for j in range(a):
            for k in range(a):
                t += i + j
    return "{}".format(t)

Database

Database

  • Limits
  • # queries per second
  • Joins
  • Request / Queries

Django

devices = Device.objects.filter(owner__username='woakas') 
for d in devices:
    print(d.owner)

# 66 queries

devices = Device.objects.filter(owner__username='woakas').prefetch_related('owner') 
for d in devices:
    print(d.owner)

# 2 queries

Redis

➜  ~ redis-benchmark -t get
====== GET ======
  100000 requests completed in 1.36 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

95.94% <= 1 milliseconds
99.97% <= 2 milliseconds
100.00% <= 2 milliseconds
73583.52 requests per second

Physical

Limits

Limits

fs.file-max=65535

net.core.somaxconn = 65536

net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_window_scaling = 1 
net.ipv4.tcp_max_syn_backlog = 65536

Limits

Network

CPU

RAM

Disc

IO

Language

Sanic / async

from sanic import Sanic
from sanic.response import json
import time

app = Sanic()


@app.route("/")
async def test(request):
    # time.sleep(1)
    return json({"hello": "world"})


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

Languages

Human

Developer

Thank you

@woakas

woakas@ubidots.com

pycon2020

By Gustavo Angulo

pycon2020

  • 701