Python in the browser
Enhancing the Scientific Python ecosystem's interoperability with Pyodide
Agriya Khetarpal
About me 😁
- Computer science + applied math background
- Software engineer at Quansight
- Privileged to contribute to the Scientific Python and help maintain the Pyodide ecosystems
- Interested in
- Python packaging 📦🐍
- Scientific computing ➗🧪
- Compilers and toolchains 🛠️⛓️
- Documentation and technical writing 📝🌉
- ...and more 👾
- The Pyodide ecosystem
- Pyodide and the heart of Scientific Python
- The Pyodide build system
- Utilities for interactive documentation
Outline 🎙️
The Pyodide ecosystem
Python implementations 🐍
- CPython – the most widely used (and the reference) implementation of the Python programming language
- However; other implementations, exist, too!
- CPython, cross-compiled via the Emscripten toolchain to WebAssembly (WASM)
- Entire distribution in itself, rather than just an implementation
- Runs on all browsers
- Chromium derivatives, Firefox, and Safari
- Even on Android and iOS!
- and in JavaScript runtimes, such as Node.js, Bun, Deno
- Write once, run anywhere
Slide on WebAssembly and Emscripten here in two halves
- Bullet One
- Bullet Two
- Bullet Three
Nope, it isn't Java (yet?)
Society if Python were to run on 3 billion devices
Python + WASM?
- RustPython and Pyodide both offer implementations of Python that can run within a WASM runtime
- Bullet Three
More than just “Python in WASM”
Python/JavaScript
Foreign function interface
WebAssembly + JavaScript stdlib
{
Glue code
micropip
(CDN)
225+ packages!
(PyPI)
pure Python wheels
and essential low-level libraries
A demo: let's play a game 🎮
Motivation
- Python projects and their environments are tricky to install and manage
- For maintainers and core developers of packages: difficult to target various platforms and architectures
- Slightly steep learning curve a euphemism for "learn it yourself like we all did"
- Lower barrier to entry to learning Python, especially for new programmers
XKCD 1987
- Documentation for scientific software needs to do more (than what it's currently doing)
- Some guidelines
- Explanatory (descriptive)
- Illustrative (graphs, diagrams, visual aids, and more)
- Interactive (a boost to user experience)
https://diataxis.fr/
Motivation
Diátaxis, from the Ancient Greek δῐᾰ́τᾰξῐς: dia (across) and taxis (arrangement).
Where can one write Python? gotta REwork this slide...
New entrant!
Jupyter running natively in the browser – powered by WASM, powered by Pyodide!
in code editors & IDEs
through literate programming environments
via containers and virtualisation software
Pyodide's limitations
- Some modules from Python standard library are unavailable
- No
threading
ormultiprocessing
- Limited sockets support (no
socket
from the stdlib)
- No
- Browser security restrictions – here, the browser is the runtime, and a browser can't replace an operating system
- Operating system features such as access to peripherals are restricted, and means of accessing the file system within the runtime are limited
- Using multiple threads is limited to just web workers and service workers
Words by the Google Chrome team, comics adaptation by Scott McCloud
This work is licensed under the Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 License.
The browser is inherently one of the most secure pieces of software on your computer!
The browser is inherently one of the most secure pieces of software on your computer!
Words by the Google Chrome team, comics adaptation by Scott McCloud
This work is licensed under the Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 License.
Pyodide at the heart of Scientific Python
Scientific Python packages
Quick show of hands: how many of you have used five?
three?
one?
Pyodide's limitations: part II
- Bitness: 32-bit memory addresses, and 4 GiB of usable memory
- No support for floating point exceptions (WASM limitation)
- Pyodide is tied to the Emscripten ABI, which changes with every bump to the Emscripten version. CPython + packages need to be re-compiled.
Extending support for a new platform
- Fix failing tests or compilation errors
- Fix crashes and fatal errors
- Organise a CI job, so that nothing breaks in the future
- Quite straightforward (but also complex!)
The Pyodide build system
The structure of a Pyodide CI job
- Builds your package out-of-tree (outside the Pyodide repository); as opposed to in-tree
- Test suites can be run using
pytest
, or other test runners
Start
Check out sources
Set up
Python and Emscripten
pyodide build
Invoke
Run tests
Install Node.js
End
- Standards-compliant distribution builder, mimics
pypa/build
-
pyodide build
builds a wheel - For pure Python packages, this is the same as
pip install build && python -m build
- Special utilities to set up cross-build environments, and to manipulate Python's
sysconfig
data to trick cross-compilation
The pyodide-build
tool
Emscripten
Drop-in replacements for GNU
GCC
and ld
, and wrappers for CMake
, Make
, and other build tools
Pyodide wheels?
{distribution}-{version}-{python tag}-{abi tag}-{platform tag}.whl
Pure Python wheels are named mypackage-0.1.0-py3-none-any.whl
Compiled wheels get tagged as
mypackage-0.1.0-cp312-cp312-emscripten_3_1_58-wasm32-any.whl
i.e., reflecting the Emscripten version in the ABI tag. This has now changed to pyodide_20XY_Z
to reflect the current year and build number.
Pyodide wheels
They will contain Emscripten-compiled shared objects. Let's see an example from the scipy.odr
module (orthogonal distance regression)
manylinux-amd64
wheel
WebAssembly wheel
├── __init__.py
├── __odrpack.cpython-312-x86_64-linux-gnu.so
├── _odrpack.py
├── models.py
└── odrpack.py
├── __init__.py
├── __odrpack.cpython-312-wasm32-emscripten.so
├── _odrpack.py
├── models.py
└── odrpack.py
ELF binaries
WASM binaries
0x7F
magic number
\0asm
magic number
Linker laments
- Fortran compilers that compile to a
wasm32-emscripten
target are nascent (flang
and LFortran) - No support with conventional compilers (
gfortran
) and commercial ones (NAG,ifx
,ifort
, etc.) - To build SciPy, which contains several Fortran-based Python extension modules, one must use
f2c
to convert all the Fortran 77 code to C code - The C code generated is then compiled to WebAssembly
- Due to language specifications, this means Fortran libraries like LAPACK have to be patched extensively.
- Problems with BLAS and LAPACK libraries are quite common in SciPy; they are used heavily to achieve great speedups for linear algebra operations
Linker laments
Function signature mismatches
- Sometimes easy, sometimes difficult; always tedious to fix!
- Caused by either incorrect code,
f2c
bugs, or wrapper differences - Reason: WASM has strict runtime guarantees, and something that would raise a warning at compile time is an error here at runtime
- Different return types, or different number of arguments
- These raise fatal errors in Pyodide, and there are several in SciPy :(
call_indirect (param i32 i32) (result i32)
func $funcname (param $var0 i32 i32 i32) (result i32)
function pointer defined with two inputs
when the symbol asks for three
Culmination
Interactive documentation
Interactive programming
- Real-time code execution: run code examples directly in docs
- Immediate feedback loops: test API functionality in real-time
- Great for learning and experimentation
- Improves learning experience for users
The NumPy and SciPy docs are now interactive!
How? jupyterlite-sphinx
A Sphinx
extension providing a set of utilities to convert doctest-based examples to notebooks
provides a Pyodide kernel for those noteboks
Simply add the .. try_examples
::
reST stub to your example's docstring:
>>> x = 2 >>> y = 2 >>> x + y 4
and a button in the HTML source is generated for you!
pip install jupyterlite-sphinx
Let's try it out!
Future things to do
- Build smaller wheels; strip test suites and unneeded files if packages include them
- Especially useful for bandwidth preservation and for those with throttled connections
- Interactive documentation adapted for more Scientific Python projects
- Build farms for Pyodide (similar to conda-forge)
- and lastly, enhanced support across Scientific Python projects
Pyodide and friends
emscripten-forge – like conda-forge, but for wasm32-emscripten
as a target
Projects and contemporaries I'm inspired by
PySheets – a powerful Pythonic spreadsheet UI, running in the browser
PyCafe – a platform for sharing and running Python browser applications
reactive Jupyter notebooks that can be deployed as WASM data apps
PyScript – a framework for Python web applications (uses MicroPython and Pyodide)
Takeaways
- Pyodide has undoubtedly opened new possibilities for Python running in the browser
- We discussed some nifty things about the Pyodide build system
- Interactive documentation benefits everyone, from beginners to experienced users, and we've discussed in brief on how to set it all up – please carry the legacy forward!
- There is a lot of ongoing work to improve both the Pyodide build system and the Scientific Python ecosystem in Pyodide
- Interested in adding your package or contributing? Please reach out to us! https://github.com/pyodide/pyodide
Thank you for your time!
Please feel free to say hello!
in/agriyakhetarpal
agriyakhetarpal
agriyakhetarpal
agriyakhetarpal [at] outlook [dot] com
Content licensed under the CC-by-SA Attribution-ShareAlike Version 4.0 International License
Special thanks to (note: pending permission from them)
Ralf Gommers
Albert Steppi
Melissa Mendonça
Robert Hood Chatham
Gyeongjae Choi
Loïc Estève
Emoji by openmoji.org
pycon-india-2024
By Agriya Khetarpal
pycon-india-2024
- 26