Multiprocessing in Python
David Taylor
"prooffreader"
data scientist, blogger, pythonista, nerd
thatsthejoke.jpg
Disclaimer
Serial computing
Do a thing
Do another thing
Do a third thing
Three things are done!
Parallel computing
Do a thing
Do another thing
Do a third thing
Three things are done!
START
START
SPLIT
COMBINE
Code
Interpreter
Program/Bytecode
Kernel
Process
Thread
.py
/usr/bin/python
.pyc
OS
RAM
Code
Interpreter
Program/Bytecode
Kernel
Process
Thread
.py
/usr/bin/python
.pyc
OS
RAM
Thread
Thread
Multithreading
Code
Interpreter
Program/Bytecode
Kernel
Process
Thread
.py
/usr/bin/python
.pyc
OS
RAM
Thread
Thread
Process
Process
Multiprocessing
CPUs
Python:
GIL
(Global Interpreter Lock)
Only one thread for most tasks
Therefore, only one process
multiprocessing module
(alternative: concurrent.futures in Py 3)
cores
time
(sec)
30,000 items
1 operation per number
30,000 items
20 operations per number
1,000,000 items
1 operation per number
CPython
vs.
Cython
(also PyPy/JIT/RPython)
Normal CPython code
Compiled CPython Code
56% faster
Compiled Cython Code
47X faster!
Libraries with
C binaries
e.g. Numpy
Multicomputing example
Desktop
pymongo, boto, pysftp
Digital Ocean
MongoDB server
AWS
20 instances +
Multicomputing example
Desktop
pymongo, boto, pysftp
Digital Ocean
MongoDB server
AWS
20 instances +
Epilogue: