Python in the browser
Enhancing the Scientific Python ecosystem's interoperability with Pyodide
Agriya Khetarpal
Slides to follow along
About me ๐
- Computer science + applied math background
- Twenty-two years old
- Software engineer at Quansight
- Privileged to contribute to the Scientific Python ecosystem and help maintain the Pyodide ecosystem
- Other work:
autograd, and the PyBaMM ecosystem (Python Battery Mathematical Modelling) - Interested in
- Python packaging ๐ฆ๐
- Scientific computing โ๐งช
- Compilers and toolchains ๐ ๏ธโ๏ธ
- Documentation and technical writing ๐๐
- ...and more ๐พ
- The Pyodide ecosystem
- Pyodide and the heart of Scientific Python
- The Pyodide build system
- Utilities for interactive documentation
Outline ๐๏ธ
The Pyodide ecosystem
Python implementations ๐
- CPython โ the most widely used (and the reference) implementation of the Python programming language
- However; other implementations, exist, too!
- CPython, cross-compiled via the Emscripten toolchain to WebAssembly (WASM)
- Entire distribution in itself, rather than just an implementation
- Runs on all browsers
- Chromium derivatives, Firefox, and Safari
- Even on Android and iOS!
- and in JavaScript runtimes, such as Node.js, Bun, Deno
- Write once, run anywhere
WebAssembly and Emscripten
- Compiles C/C++ code to WASM (
wasm32-emscripten) via LLVM - Provides an entire toolchain for this, including the compiler and the linker, and build utilities
-
emcc, as a drop-in replacement forgcc
- Binary format for a stack-based VM, intended as a compilation target
- Inherently designed to be secure and to run in constrained environments
- Supported by most modern browsers
Nope, it isn't Java (yet?)
Society if Python were to run on 3 billion devices
Python + WASM?
- RustPython and Pyodide both offer an implementation of the Python programming language that can run within a WASM runtime
- However, Pyodide can run CPython extension modules โ while this use case is limited for RustPython
More than just โPython in WebAssemblyโ
+
+
Python/JavaScript
Foreign function interface
WebAssembly + JavaScript stdlib
{
{}
Glue code
micropip
(CDN)
225+ packages!
(PyPI)
pure Python wheels
and essential low-level libraries
A demo: let's play a game ๐ฎ
Yes, pygame-ce and SDL2 work, too, since version 0.26! Yay!
Play along!
https://agriyakhetarpal.github.io/sudoku/ โ (works better on laptops)
Motivation
- Scientific Python projects and their environments are tricky to install and manage
- For maintainers and core developers of packages: difficult to target various platforms and architectures
- Slightly steep learning curve sometimes a euphemism for "learn it yourself like we all did"
- Lower barrier to entry to learning Python, especially for new programmers
XKCD 1987
- Documentation for scientific software needs to do more (than what it's currently doing)
- Some guidelines
- Explanatory (descriptive)
- Illustrative (graphs, diagrams, visual aids, and more)
- Interactive (a boost to user experience)
https://diataxis.fr/
Motivation
Diรกtaxis, from the Ancient Greek ฮดแฟแพฐฬฯแพฐฮพแฟฯ: dia (across) and taxis (arrangement).
Literate programming environments for Python
New entrant!
Jupyter running natively in the browser โ powered by WASM and Pyodide!
Google Colab, Binder, and JupyterLab (and Jupyter Notebook), and more
Client-side execution โ all code runs via your CPU and hardware
Server-side execution โ all code runs on a server (unless running locally, of course)
Pyodide's limitations
- Some modules from Python standard library are unavailable
- No
threadingormultiprocessing - Limited sockets support (no
socketfrom the stdlib)
- No
- Browser security restrictions โ here, the browser is the runtime, and a browser can't replace an operating system
- Operating system features such as access to peripherals are restricted, and means of accessing the file system within the runtime are limited
- Using multiple threads is limited to just web workers and service workers
Words by the Google Chrome team, comics adaptation by Scott McCloud
This work is licensed under the Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 License.
The browser is inherently one of the most secure pieces of software on your computer!
The browser is inherently one of the most secure pieces of software on your computer!
Words by the Google Chrome team, comics adaptation by Scott McCloud
This work is licensed under the Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 License.
Pyodide at the heart of Scientific Python
Scientific Python packages
Quick show of hands: how many of you have used five?
three?
one?
Pyodide's limitations: part II
- Bitness: 32-bit memory addresses, and 4 GiB of usable memory
- No support for floating point exceptions (WASM limitation)
- Pyodide is tied to the Emscripten ABI, which changes with every bump to the Emscripten version. CPython + packages need to be re-compiled.
- and more
Extending support for a new platform
- Fix compilation errors and/or failing tests
- Fix a few crashes and fatal errors at runtime
- Organise a CI job, so that nothing breaks in the future
- Quite straightforward (but also complex!)
The Pyodide build system
The structure of a Pyodide CI job
- Builds your package out-of-tree (outside the Pyodide repository); as opposed to in-tree
- Test suites can be run using
pytest, or other test runners
Start
Check out sources
Set up
Python and Emscripten
pyodide build
Invoke
Run tests
Install Node.js
End
- Standards-compliant distribution builder, mimics
pypa/build -
pyodide buildbuilds a wheel - For pure Python packages, this is the same as
pip install build && python -m build - Special utilities to set up cross-build environments, and to manipulate Python's
sysconfigdata to trick cross-compilation - Works nicely with build backends for compiled packages: Meson, scikit-build-core + pybind11, and so on
-
pypa/cibuildwheelgained support for Pyodide builds this year!
The pyodide-build tool
Emscripten
Drop-in replacements for GNU GCC and ld, and wrappers for CMake, Make, and other build tools
Pyodide wheels?
{distribution}-{version}-{python tag}-{abi tag}-{platform tag}.whlPure Python wheels are named mypackage-0.1.0-py3-none-any.whl
Compiled wheels get tagged as
mypackage-0.1.0-cp312-cp312-emscripten_3_1_58-wasm32-any.whli.e., reflecting the Emscripten version in the ABI tag. This has now changed to pyodide_20XY_Z to reflect the current year and build number.
Pyodide wheels
They will contain Emscripten-compiled shared objects. Let's see an example from the scipy.odr module (orthogonal distance regression)
manylinux-amd64 wheel
WebAssembly wheel
โโโ __init__.py
โโโ __odrpack.cpython-312-x86_64-linux-gnu.so
โโโ _odrpack.py
โโโ models.py
โโโ odrpack.py
โโโ __init__.py
โโโ __odrpack.cpython-312-wasm32-emscripten.so
โโโ _odrpack.py
โโโ models.py
โโโ odrpack.py
ELF binaries
WASM binaries
0x7F magic number
\0asm magic number
Linker laments
- Fortran compilers that compile to a
wasm32-emscriptentarget are nascent (flangand LFortran) - No support with conventional compilers (
gfortran) and commercial ones (NAG,ifx,ifort, etc.) - To build SciPy, which contains several Fortran-based Python extension modules, one must use
f2cto convert all the FORTRAN 77 code to C code - The C code generated is then compiled to WebAssembly
- Due to language specifications, this means Fortran libraries like LAPACK have to be patched extensively.
- Problems with BLAS and LAPACK libraries are quite common in SciPy; they are used heavily to achieve great speedups for linear algebra operations
Linker laments
Function signature mismatches
- Sometimes easy, sometimes difficult; always tedious to fix!
- Caused by either incorrect code,
f2cbugs, or wrapper differences, or sometimes, even bad FORTRAN 77 code - Reason: WASM has strict runtime guarantees, and something that would raise a warning at compile time with other compilation targets is an error here at runtime
- Different return types, or different number of arguments
- These raise fatal errors in Pyodide, and there are several in SciPy :(
call_indirect (param i32 i32) (result i32)
func $funcname (param $var0 i32 i32 i32) (result i32)
function pointer defined with two inputs
when the symbol asks for three
Culmination
Interactive documentation
Interactive programming
- Real-time code execution: run code examples directly in docs
- Immediate feedback loops: test API functionality in real-time
- Great for learning and experimentation
- Improves learning experience for users
The NumPy and SciPy docs are now interactive!
How? jupyterlite-sphinx
A Sphinx extension providing a set of utilities to convert doctest-based examples to notebooks
provides a Pyodide kernel for those noteboks
+
+
Simply add the .. try_examples::
reST stub to your example's docstring:
>>> x = 2 >>> y = 2 >>> x + y 4
and a button in the HTML source is generated for you!
pip install jupyterlite-sphinx jupyterlite-pyodide-kernel
Another kernel, jupyterlite-xeus is available as well, which works with emscripten-forge recipes.
Let's try it out!
Future things to do
- Build smaller wheels; strip test suites and unneeded files if packages include them
- Especially useful for bandwidth preservation and for those with throttled connections
- Interactive documentation adapted for more Scientific Python projects, likely through a SPEC document: Scientific Python Enhancement Proposal
- Build farms for Pyodide (similar to
conda-forge) - and lastly, enhanced support across Scientific Python projects
- We've had a new 0.26.3 release with additions for the JavaScript Promise Integration (JSPI) APIs โ we'd like to further improve support!
Pyodide and friends
emscripten-forge โ like conda-forge, but for wasm32-emscripten as a target
Projects and contemporaries I'm inspired by
PySheets โ a powerful Pythonic spreadsheet UI, running in the browser
PyCafe โ a platform for sharing and running Python browser applications
reactive Jupyter notebooks that can be deployed as WASM data apps
PyScript โ a framework for Python web applications (uses MicroPython and Pyodide)
Takeaways
- Pyodide has undoubtedly opened new possibilities for Python running in the browser
- We discussed some nifty things about the Pyodide build system
- Interactive documentation benefits everyone, from beginners to experienced users, and we've discussed in brief on how to set it all up โ please carry the legacy forward!
- There is a lot of ongoing work to improve both the Pyodide build system and the Scientific Python ecosystem in Pyodide
- Interested in adding your package or contributing? Please reach out to us! https://github.com/pyodide/pyodide
Thank you for your time!
Please feel free to say hello!
in/agriyakhetarpal
agriyakhetarpal
agriyakhetarpal
agriyakhetarpal [at] outlook [dot] comContent licensed under the CC-by-SA Attribution-ShareAlike Version 4.0 International License
Special thanks to
Ralf Gommers
Albert Steppi
Melissa Mendonรงa
Matthias Bussonnier
Robert Hood Chatham
Gyeongjae Choi
Loรฏc Estรจve
Emoji by openmoji.org
These slides
Pyodide
@agriyakhetarpal@fosstodon.org
Python in the browser Enhancing the Scientific Python ecosystem's interoperability with Pyodide Agriya Khetarpal Slides to follow along
PyCon APAC, 2024
By Agriya Khetarpal
PyCon APAC, 2024
- 113
