IPySpaghetti
Changing Notebook paradigm to solve their pitfalls?
Dr. Corentin Cadiou — @cphyc — github.com/cphyc
IPySpaghetti
Changing Notebook paradigm to solve their pitfalls?
| Example | Issue |
|---|---|
| Let's execute out of order! | ❌ Reproducibility |
| Let's overwrite this global variable! | ❌ Global scope pollution |
| Let's couple this cell to its surroundings! | ❌ Testability |
| Let's copy and paste this piece of code! | ❌ Reusability |
Jupyter notebooks permit bad habits
See J. Grus' talk @ JupyterCon
IPySpaghetti
Changing Notebook paradigm to solve their pitfalls?
Some solutions exist:
| Issue | Elyra | DF notebook | execution_dependencies |
|---|---|---|---|
| Reproducibility | ✅ * | ✅ (automatic) | ✅ (manually) |
| Scope pollution | partially | ||
| Testability | ✅ * | ||
| Reusability | ✅ * |
* using a graph of notebooks, not enforced at notebook level
Links: Elyra, Dataflow notebook, execution_dependencies
IPySpaghetti
Changing Notebook paradigm to solve their pitfalls?
Jupyter Notebook
Based on "read-eval-print loop":
- write cell
- evaluate cell
- print output
IPySpaghetti
Changing Notebook paradigm to solve their pitfalls?
Jupyter Notebook
IPySpaghetti
Based on "read-eval-print loop":
- write cell
- evaluate cell
- print output
Based on dataflow paradigm:
- write node
- insert node in graph
- execute graph
1
2
3
IPySpaghetti
Changing Notebook paradigm to solve their pitfalls?
Jupyter Notebook
IPySpaghetti
IPySpaghetti
Changing Notebook paradigm to solve their pitfalls?
Replacing REPL → Dataflow
| Reproducibility | ✅ Enforced by graph |
| Scope pollution | ✅ Each node has its scope |
| Testability | ✅ Relies on regular .py functions |
| Reusability | ✅ Functions can be reused |
Advantages
Drawbacks
Less interactivity
Steeper learning curve
More clicks
Code not very mature, bad doc
No community (yet!)
Reuses JLab interactivity (widgets, ...)
Graph executable w/ parallel scheduler
Backed by regular .py file
Could use "node bookshelf"
Technical details
Inspirations
IPySpaghetti
By Corentin Cadiou
IPySpaghetti
Poster for the 2021 Scipy Conference
