Python - GIL

The good the bad and the ugly

woakas

Coffee / 5-7

14 years

2 sons

CTO / Co-founder

Global Interpreter Lock

Byte-code One by One

CPython

Pros

  • Prevents race conditions.
  • Increased speed of single-threaded programs.
  • No deadlocks
  • Easy to test

Cons

  • Threaded programs run slower
  • Threading is not concurrent

The good

How does it work?

David Beazley, http://dabeaz.com/

How does it work?

David Beazley, http://dabeaz.com/

Single thread

import time


def sum():
    total = 0
    for i in range(50_000_000):
        total += 1
    return total


def main():
    start = time.time()
    sum()
    end = time.time()
    total_time = int((end - start) * 1000)
    print(f"Time in milliseconds - {total_time}")


main()

Two threads

import time
from threading import Thread


def fn(n):
    total = 0
    for i in range(n):
        total += 1
    return total


def main():
    count = 50_000_000
    t1 = Thread(target=fn, args=(count // 2,))
    t2 = Thread(target=fn, args=(count // 2,))

    start = time.time()
    t1.start()
    t2.start()
    t1.join()
    t2.join()
    end = time.time()
    total_time = int((end - start) * 1000)

    print(f"Time in milliseconds - {total_time}")


main()

Multiprocessing

import time
import multiprocessing


def fn(n):
    total = 0
    for i in range(n):
        total += 1
    return total


def main():
    count = 50_000_000
    p1 = multiprocessing.Process(target=fn, args=(count // 2,))
    p2 = multiprocessing.Process(target=fn, args=(count // 2,))
    start = time.time()
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    end = time.time()
    total_time = int((end - start) * 1000)

    print(f"Time in milliseconds - {total_time}")


main()

Results

?????

Single thread
Avg: 3.123 ms

 

Two threads

Avg: 2.945 ms

The bad?????

Single thread
Avg: 3.123 ms

 

Two threads

Avg: 2.945 ms

Tracing the Python GIL

pip install tabulate per4m viztracer

Tracing the Python GIL

# Single Thread
perf record -e sched:sched_switch  -e sched:sched_process_fork -e 'sched:sched_wak*' \
   -k CLOCK_MONOTONIC  --call-graph dwarf -- viztracer -o viztracer-single_thread_gil.json --ignore_frozen -m  single_thread
perf script --no-inline | per4m perf2trace sched -o perf-single_thread.json
viztracer --log_multiprocess --combine perf-single_thread.json viztracer-single_thread_gil.json -o data-single_thread.html


# Two Threads
perf record -e sched:sched_switch  -e sched:sched_process_fork -e 'sched:sched_wak*' \
    -k CLOCK_MONOTONIC  --call-graph dwarf -- viztracer -o viztracer-two_threads-gil.json --ignore_frozen -m two_threads
perf script --no-inline | per4m perf2trace sched -o perf-two_threads.json
viztracer --combine perf-two_threads.json viztracer-two_threads-gil.json  -o data-two_threads.html


# Multiprocessing
viztracer --log_multiprocess multi_processing.py -o data-multi_processing.html

Tracing the Python GIL

Tracing the Python GIL

Tracing the Python GIL

Tracing the Python GIL

Tracing the Python GIL

The ugly

The ugly

GIL

Know it, understand it and use it

It's not magic, it's just code

References

 

@woakas

GIL The good the bad and the ugly

By Gustavo Angulo

GIL The good the bad and the ugly

  • 690