Python in the browser

Enhancing the Scientific Python ecosystem's interoperability with Pyodide

Agriya Khetarpal

About me 😁

  • Computer science + applied math background
  • Software engineer at Quansight
  • Privileged to contribute to the Scientific Python and help maintain the Pyodide ecosystems
  • Interested in
    • Python packaging 📦🐍
    • Scientific computing ➗🧪
    • Compilers and toolchains 🛠️⛓️
    • Documentation and technical writing 📝🌉
    • ...and more 👾
  • The Pyodide ecosystem
  • Pyodide and the heart of Scientific Python
  • The Pyodide build system
  • Utilities for interactive documentation

Outline 🎙️

The Pyodide ecosystem

Python implementations 🐍

  • CPython – the most widely used (and the reference) implementation of the Python programming language
  • However; other implementations, exist, too!
  • CPython, cross-compiled via the Emscripten toolchain to WebAssembly (WASM)
  • Entire distribution in itself, rather than just an implementation
  • Runs on all browsers
    • Chromium derivatives, Firefox, and Safari
    • Even on Android and iOS!
    • and in JavaScript runtimes, such as Node.js, Bun, Deno
    • Write once, run anywhere

Slide on WebAssembly and Emscripten here in two halves

  • Bullet One
  • Bullet Two
  • Bullet Three

Nope, it isn't Java (yet?)

Society if Python were to run on 3 billion devices

Python + WASM?

  • RustPython and Pyodide both offer implementations of Python that can run within a WASM runtime
  •  
  • Bullet Three

More than just “Python in WASM”

+

Python/JavaScript

Foreign function interface

WebAssembly + JavaScript stdlib

{

{}

Glue code

micropip

(CDN)

225+ packages!

(PyPI)

pure Python wheels

and essential low-level libraries

A demo: let's play a game 🎮

Motivation

  • Python projects and their environments are tricky to install and manage
  • For maintainers and core developers of packages: difficult to target various platforms and architectures
  • Slightly steep learning curve a euphemism for "learn it yourself like we all did"
  • Lower barrier to entry to learning Python, especially for new programmers

XKCD 1987

  • Documentation for scientific software needs to do more (than what it's currently doing)
  • Some guidelines
    • Explanatory (descriptive)
    • Illustrative (graphs, diagrams, visual aids, and more)
    • Interactive (a boost to user experience)
https://diataxis.fr/

Motivation

Diátaxis, from the Ancient Greek δῐᾰ́τᾰξῐς: dia (across) and taxis (arrangement).

Where can one write Python? gotta REwork this slide...

New entrant!

Jupyter running natively in the browser – powered by WASM, powered by Pyodide!

in code editors & IDEs

through literate programming environments

via containers and virtualisation software

Pyodide's limitations

  • Some modules from Python standard library are unavailable
    • No threading or multiprocessing
    • Limited sockets support (no socket from the stdlib)
  • Browser security restrictions – here, the browser is the runtime, and a browser can't replace an operating system
  • Operating system features such as access to peripherals are restricted, and means of accessing the file system within the runtime are limited
  • Using multiple threads is limited to just web workers and service workers

Words by the Google Chrome team, comics adaptation by Scott McCloud
 

This work is licensed under the Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 License.

The browser is inherently one of the most secure pieces of software on your computer!

The browser is inherently one of the most secure pieces of software on your computer!

Words by the Google Chrome team, comics adaptation by Scott McCloud
 

This work is licensed under the Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 License.

Pyodide at the heart of Scientific Python

Scientific Python packages

Quick show of hands: how many of you have used five?

three?

one?

Pyodide's limitations: part II

  • Bitness: 32-bit memory addresses, and 4 GiB of usable memory
  • No support for floating point exceptions (WASM limitation)
  • Pyodide is tied to the Emscripten ABI, which changes with every bump to the Emscripten version. CPython + packages need to be re-compiled.

Extending support for a new platform

  • Fix failing tests or compilation errors
  • Fix crashes and fatal errors
  • Organise a CI job, so that nothing breaks in the future
  • Quite straightforward (but also complex!)

The Pyodide build system

The structure of a Pyodide CI job

  • Builds your package out-of-tree (outside the Pyodide repository); as opposed to in-tree
  • Test suites can be run using pytest, or other test runners

Start

Check out sources

Set up
Python and Emscripten

pyodide build

Invoke

Run tests

Install Node.js

End

  • Standards-compliant distribution builder, mimics pypa/build
  • pyodide build builds a wheel
  • For pure Python packages, this is the same as pip install build && python -m build
  • Special utilities to set up cross-build environments, and to manipulate Python's sysconfig data to trick cross-compilation

The pyodide-build tool

Emscripten

Drop-in replacements for GNU GCC and ld, and wrappers for CMake, Make, and other build tools

Pyodide wheels?

{distribution}-{version}-{python tag}-{abi tag}-{platform tag}.whl

Pure Python wheels are named mypackage-0.1.0-py3-none-any.whl

Compiled wheels get tagged as

mypackage-0.1.0-cp312-cp312-emscripten_3_1_58-wasm32-any.whl

i.e., reflecting the Emscripten version in the ABI tag. This has now changed to pyodide_20XY_Z to reflect the current year and build number.

Pyodide wheels

They will contain Emscripten-compiled shared objects. Let's see an example from the scipy.odr module (orthogonal distance regression)

manylinux-amd64 wheel

WebAssembly wheel

├── __init__.py

├── __odrpack.cpython-312-x86_64-linux-gnu.so

├── _odrpack.py

├── models.py

└── odrpack.py

├── __init__.py

├── __odrpack.cpython-312-wasm32-emscripten.so

├── _odrpack.py

├── models.py

└── odrpack.py

ELF binaries

WASM binaries

0x7F magic number

\0asm magic number

Linker laments

  • Fortran compilers that compile to a wasm32-emscripten target are nascent (flang and LFortran)
  • No support with conventional compilers (gfortran) and commercial ones (NAG, ifx, ifort, etc.)
  • To build SciPy, which contains several Fortran-based Python extension modules, one must use f2c to convert all the Fortran 77 code to C code
  • The C code generated is then compiled to WebAssembly
  • Due to language specifications, this means Fortran libraries like LAPACK have to be patched extensively.
  • Problems with BLAS and LAPACK libraries are quite common in SciPy; they are used heavily to achieve great speedups for linear algebra operations

Linker laments

Function signature mismatches

  • Sometimes easy, sometimes difficult; always tedious to fix!
  • Caused by either incorrect code, f2c bugs, or wrapper differences
  • Reason: WASM has strict runtime guarantees, and something that would raise a warning at compile time is an error here at runtime
  • Different return types, or different number of arguments
  • These raise fatal errors in Pyodide, and there are several in SciPy :(
call_indirect (param i32 i32) (result i32)
func $funcname (param $var0 i32 i32 i32) (result i32)

function pointer defined with two inputs

when the symbol asks for three

Culmination

Interactive documentation

Interactive programming

  • Real-time code execution: run code examples directly in docs
  • Immediate feedback loops: test API functionality in real-time
  • Great for learning and experimentation
  • Improves learning experience for users

The NumPy and SciPy docs are now interactive!

How? jupyterlite-sphinx

A Sphinx extension providing a set of utilities to convert doctest-based examples to notebooks

provides a Pyodide kernel for those noteboks

+

Simply add the .. try_examples:: 

reST stub to your example's docstring:

>>> x = 2
>>> y = 2
>>> x + y
4

and a button in the HTML source is generated for you!

pip install jupyterlite-sphinx

Let's try it out!

Future things to do

  • Build smaller wheels; strip test suites and unneeded files if packages include them
    • Especially useful for bandwidth preservation and for those with throttled connections
  • Interactive documentation adapted for more Scientific Python projects
  • Build farms for Pyodide (similar to conda-forge)
  • and lastly, enhanced support across Scientific Python projects

Pyodide and friends

emscripten-forge – like conda-forge, but for wasm32-emscripten as a target

Projects and contemporaries I'm inspired by

More? https://pyodide.org/en/stable/project/related-projects.html

PySheets – a powerful Pythonic spreadsheet UI,  running in the browser

PyCafe – a platform for sharing and running Python browser applications

reactive Jupyter notebooks that can be deployed as WASM data apps

PyScript – a framework for Python web applications (uses MicroPython and Pyodide)

Takeaways

  • Pyodide has undoubtedly opened new possibilities for Python running in the browser
  • We discussed some nifty things about the Pyodide build system
  • Interactive documentation benefits everyone, from beginners to experienced users, and we've discussed in brief on how to set it all up – please carry the legacy forward!
  • There is a lot of ongoing work to improve both the Pyodide build system and the Scientific Python ecosystem in Pyodide
  • Interested in adding your package or contributing? Please reach out to us! https://github.com/pyodide/pyodide

Thank you for your time!

Please feel free to say hello!

in/agriyakhetarpal

agriyakhetarpal

agriyakhetarpal

agriyakhetarpal [at] outlook [dot] com

Content licensed under the CC-by-SA Attribution-ShareAlike Version 4.0 International License

Special thanks to (note: pending permission from them)

Ralf Gommers

Albert Steppi

Melissa Mendonça

Robert Hood Chatham

Gyeongjae Choi

Loïc Estève

Emoji by openmoji.org