Using Nix for Repeatable Python Environments

Daniel Wheeler

github.com/wd15

Overview

Packaging concerns
What is Nix?
How can you use Nix?
- Nix language tutorial
- Python examples, tips and tricks
How to fix Nix for Python

Packaging Concerns

$ conda activate my-clean-env
$ python --version
Python 3.7.3
$ conda install sfepy
...
$ python --version
Python 2.7.15
$ conda deactivate

$ conda activate my-new-clean-env
$ python --version
Python 3.6.7
$ conda install sfepy
...
$ python --version
Python 3.6.7

My PYTHON VERSION CHANGED!

$ conda install fipy python=3.6
...
$ python -c "import numpy; numpy.__config__.show()"
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
...
$ conda install sfepy python=3.6
...
$ python -c "import numpy; numpy.__config__.show()"
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
...

$ source activate clean-env
$ conda install sfepy python=3.6
...
$ python -c "import numpy; numpy.__config__.show()"
blas_mkl_info:
    NOT AVAILABLE
...
$ conda install fipy python=3.6
...
$ python -c "import numpy; numpy.__config__.show()"
blas_mkl_info:
    NOT AVAILABLE
...

Inconsistent outcomes

outcomes are dependent on state!

rollbaCKS

$ conda activate my-clean-env
$ python --version
Python 3.7.3
$ conda install fipy
...
$ python --version
Python 3.7.3
$ conda install sfepy
...
$ python --version
Python 2.7.15
$ conda remove sfepy
...
$ python --version
Python 2.7.15

No rollback capability in most package managers

dIAMOND dEPENDENCY

FiPy

Sfepy

Openblas

Numpy

Blas (MKL)

myenv

Questions

Why isn't it safe to update my environment?
Why can't I rollback to a previous environment?
Why can't I build with multiple versions of the same library?
Why is testing with different configurations so difficult?

Why isn't package management more like version control?

What is Nix?

Nix*

Nix (package manager)
Nix (language)
Nixpkgs (collection of Nix package recipes)
NixOS (linux distro)
Other Nix jargon:
- NixOps (tools for deploying Nix and NixOS)
- DisNix (tools for deploying services (like Ansible))
- Hydra CI system
- Guix (GNU form of Nix, based on Scheme)
- Cachix (Cache your Nix builds)

How does nIX Work?

Nixpkgs

NIx Package Manager

Nix Expression

Derivation

/nix/store

Instantiate

Realize

Impure, but hashed

default.nix

SHA

Derivation is independent of /nix/store

/nix/store/n1abx6l20rq73h8k84lpcysnm2d33jz2-python3.6-numpy-1.14.0

/nix/store/9w1m9b1zqa2k64pdaj20c39ay0vj2gc4-gfortran-7.3.0-lib

"NixOS: A Purely Functional Linux Distribution"

In this article we show that we can overcome these problems by moving to a purely functional system configuration model. This means that all static parts of a system (such as software packages,configuration files and system startup scripts) are built by pure functions and are immutable, stored in a way analogous to a heap in a purely functional language.

purely functional data structure (immutable)

Dolstra et al.

repology.org/statistics

NixOS

Nix Tutorial

nix-repl> 1 + 2  # Integers
3

nix-repl> 1 / 3.  # Floats
0.333333

nix-repl> builtins.typeOf { a = 1; }  # Sets
"set"

nix-repl> { a = 1; b = 2; }.b
2

nix-repl> rec { a = 2; b = a; }
{ a = 2; b = 2; }

nix-repl> ./hello_world.txt  # Paths
/home/wd15/hello_world.txt

nix-repl> f = x: x * x

nix-repl> f 2  # purely functional (expressions and lazy)
4

The BASics

nix-repl> data = builtins.readFile ./hello_world.txt  # read from a file
    
nix-repl> data
"Hello World!\n"

nix-repl> builtins.toFile "foo.txt" data  # write to a file
"/nix/store/mm4fq57cw5gkjgn1c2yysgrq6cd1w6z2-foo.txt"

nix-repl> builtins.toFile "foo.txt" (data + "foo")
"/nix/store/xshshqgxxp9hsgrf8a7k229vw5f2rj9q-foo.txt"

ImPURE CheatING

Can Only write to the /nix/store!

Read and write files

Two versions of the same file

nix-repl> :l <nixpkgs>
Added 10482 variables.

nix-repl> d = builtins.derivation {
          name = "my-hello";
          builder = "${bash}/bin/bash";
          args = [ ./build.sh ];
          src = ./my-hello;
          system = builtins.currentSystem; }

nix-repl> d
«derivation /nix/store/9d80ryw1lanalq4avvjh2hjd7sckid27-my-hello.drv»

Instantiate

A Derivation

{
  "/nix/store/9d80ryw1lanalq4avvjh2hjd7sckid27-my-hello.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/ka2vfi17abpi2np3i5xikjjdymd5wldf-my-hello"
      }
    },
    "inputSrcs": [
      "/nix/store/r3wkhhmj6rn3hpm04cw6yd9vc9yy9bhp-build.sh",
      "/nix/store/y8iplxzxh00i32iiz4pl1zw8g95pfvh0-my-hello"
    ],
    "inputDrvs": {
      "/nix/store/wlmr9cbm9zsn6bxx204zpxpbpwg7vxiq-bash-4.4-p23.drv": [
        "out"
      ]
    },
    "platform": "x86_64-linux",
    "builder": "/nix/store/r47p5pzx52m3n34vdgqpk5rvqgm0m24m-bash-4.4-p23/bin/bash",
    "args": [
      "/nix/store/r3wkhhmj6rn3hpm04cw6yd9vc9yy9bhp-build.sh"
    ],
    "env": {
      "builder": "/nix/store/r47p5pzx52m3n34vdgqpk5rvqgm0m24m-bash-4.4-p23/bin/bash",
      "name": "my-hello",
      "out": "/nix/store/ka2vfi17abpi2np3i5xikjjdymd5wldf-my-hello",
      "src": "/nix/store/y8iplxzxh00i32iiz4pl1zw8g95pfvh0-my-hello",
      "system": "x86_64-linux"
    }
  }
}

Python Examples, Tips and Tricks

Imperative Installation

Nix supports imperative installation

$ nix-env -i hello
installing ‘hello-2.10’
...

$ hello
Hello, world!

$ nix-env -q
hello-2.10

$ nix-env --list-generations
   1   2019-06-20 13:56:08

$ nix-env -i python

$ nix-env -q
hello-2.10
python-2.7.16

$ nix-env --list-generations
   1   2018-08-15 10:46:38
   2   2018-08-15 10:47:22   (current)

$ nix-env --rollback
switching from generation 2 to 1

$ nix-env --list-generations
   1   2019-06-20 13:56:08   (current)
   2   2019-06-20 13:57:39

$ nix-env -q
hello-2.10

BUT NOT FOR PYTHON

Need to use nix-shell

$ nix-shell --pure -p python27 -p python27Packages.numpy -p python27Packages.scipy
these paths will be fetched (52.90 MiB download, 245.07 MiB unpacked):
...

[nix-shell] $ echo $PATH
/nix/store/qz40q38lask9zx6h0rgwf1qbcllclx82-bash-interactive-4.4-p23/bin:/nix/store/qpzwm6z4igakmqr4n4k6k3q0a4bqy3ws-patchelf-0.9/bin:/nix/store/1ap5d85630s3ksal6xgkjnbglmbng3kg-gcc-wrapper-7.4.0/bin:/nix/store/d8k7bv2w9g669dv7r9z4wrr9cnzdncdv-gcc-7.4.0/bin:/nix/store/2fxzw4ilrgc4klppk1nc50vgcwfphh1s-glibc-2.27-bin/bin:/nix/store/wpjdad5wpylnpqbjw4dbnih8f6q32l43-coreutils-8.31/bin:/nix/store/yzijh65sak6z06cdvrg9wi2d1964g6h3-binutils-wrapper-2.31.1/bin:/nix/store/c222w06ysx899n7r1jqaw96l6x6g8q9i-binutils-2.31.1/bin:/nix/store/2fxzw4ilrgc4klppk1nc50vgcwfphh1s-glibc-2.27-bin/bin:/nix/store/wpjdad5wpylnpqbjw4dbnih8f6q32l43-coreutils-8.31/bin:/nix/store/9yi3j41iwz045cn8h6brghi3lzsd886k-python-2.7.16/bin:/nix/store/7fr8a5qqyhlws3cnklnr37qywxdrihx3-python2.7-numpy-1.16.3/bin:/nix/store/9yi3j41iwz045cn8h6brghi3lzsd886k-python-2.7.16/bin:/nix/store/ssarsbjq5w5r4rqb37n24kg8h6q0nc0c-python2.7-setuptools-41.0.1/bin:/nix/store/wpjdad5wpylnpqbjw4dbnih8f6q32l43-coreutils-8.31/bin:/nix/store/d9y878m6hk96mc05pz4vdzqrlqcfl7rs-findutils-4.6.0/bin:/nix/store/xnvj5phwyv5fj4idkrwass1243nn5n19-diffutils-3.7/bin:/nix/store/g92ybkzhiqcw7xsz5yq1dayjjlhll914-gnused-4.7/bin:/nix/store/da2jifip9xsab81yh34h887a1fwnz9gd-gnugrep-3.3/bin:/nix/store/vk9rvr21phkc2jfb765rnrlhb171ywh2-gawk-4.2.1/bin:/nix/store/d34nmar6fd15yf24rw8h5aiiwcywlzbq-gnutar-1.32/bin:/nix/store/f1689ai31ca0m7j6q8lpyf10z16rmrmw-gzip-1.10/bin:/nix/store/cr2fv6r0bic97wmakjdxcwbvyp4hqn2z-bzip2-1.0.6.0.1-bin/bin:/nix/store/cpn4i0g3s7m5l0i6v1i7k855ylfxfn0g-gnumake-4.2.1/bin:/nix/store/fyxcppddjg7abrand8n10gwzm5gknc48-bash-4.4-p23/bin:/nix/store/1nnnsl2l0hcnzcy4410pbpicsvkfkzj9-patch-2.7.6/bin:/nix/store/rshc59lkp5j44fjzg2a5hi2ril4dd7ka-xz-5.2.4-bin/bin

[nix-shell] $ echo $PYTHONPATH
/nix/store/9yi3j41iwz045cn8h6brghi3lzsd886k-python-2.7.16/lib/python2.7/site-packages:/nix/store/7fr8a5qqyhlws3cnklnr37qywxdrihx3-python2.7-numpy-1.16.3/lib/python2.7/site-packages:/nix/store/ssarsbjq5w5r4rqb37n24kg8h6q0nc0c-python2.7-setuptools-41.0.1/lib/python2.7/site-packages:/nix/store/4kkq89k2gag7fq9kr1gfhp3vvcfbhzwc-python2.7-scipy-1.2.1/lib/python2.7/site-packages

Each package installed in a separate site-packages directory
Requires new hash if installed into base site-packages

Completely pure environment

Repeatable environment

# shell.nix
let
  nixpkgs = import <nixpkgs> {};
  pypkgs = nixpkgs.python36Packages;
in
  (pypkgs.python.withPackages (x: [ x.numpy x.scipy ])).env

$ nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/6a3f5bcb061e1822f50e299f5616a0731636e4e7.tar.gz

Script the previous nix-shell command

and maybe even specify the version of nixpkgs

Specify NIXPKGs Version

# Usage:
#   nixpkgs = import nixpkgs_version.nix;
#
let
  inherit (import <nixpkgs> {}) fetchFromGitHub;
  nixpkgs_download = fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs-channels";
    rev = "61deecdc34fc609d0f805b434101f3c8ae3b807a";  # release version: 18.09-beta
    sha256 = "147xyn8brvkfgz1z3jbk13w00h6camnf6z0bz0r21g9pn1vv7sb0";
  };
in
  import nixpkgs_download {}

Fully repeatable by enforcing the Nixpkgs version

Prevent Version Drift!!!

buildPythonPackage

  pypkgs.buildPythonPackage rec {
    pname = "scipy";
    version = "0.19.1";
    src = pypkgs.fetchPypi {
      inherit pname version;
      sha256 = "1rl411bvla6q7qfdb47fpdnyjhfgzl6smpha33n9ar1klykjr6m1";
    };
    buildInputs = [
      pypkgs.numpy
      nixpkgs.pkgs.gcc
      nixpkgs.pkgs.gfortran
    ];
    doCheck = false;
  };

Build derivations for Python packages
Fetch from PyPI
Specify build time, run time, test time and native dependencies
Either runs "setup.py install" or "setup.py develop"

Pypi2NIX

$ pypi2nix -V "3.6" -r requirements.txt
$ nix-shell --pure requirements.nix

Fast Nix expression generation
Complicated to debug
Hardwires Python version
New Nixpkgs (19.03) now installs missing Python dependencies into

${tmp_path}/lib/pythonX.Y/site-packages

Mixed Language EnvironmentS

$ nix-shell --pure

[nix-shell]$ python --version
Python 3.6.4

[nix-shell]$ node --version
v6.13.0

[nix-shell]$ jekyll serve
Configuration file: /home/wd15/git/pfhub/_config.yml
...

Maintain web app that uses Python, Ruby (Jekyll) and Node packages
Share with other developers
Deploy on CI with exact same build
Extremely difficult to maintain without Nix

Temporary approach (more pure)

Pip Cheats

$ nix-shell --pure my_python_expression.nix
...

[nix-shell]$ echo $tmp_path
/run/user/33396/tmp.XY6m7ClaxE

[nix-shell]$ pip install --install-option="--prefix=$tmp_path" toolz
/nix/store/sh9x58f3gbs...
...
Successfully installed toolz

Permanent approach (less pure)

postShellHook = ''
  SOURCE_DATE_EPOCH=$(date +%s)
  export PYTHONUSERBASE=$PWD/.local
  export USER_SITE=`python -c "import site; print(site.USER_SITE)"`
  export PYTHONPATH=$PYTHONPATH:$USER_SITE
  export PATH=$PATH:$PYTHONUSERBASE/bin
'';

$ nix-shell --pure my_python_expression.nix
...

[nix-shell]$ pip install --user toolz
/nix/store/sh9x58f3gbs...
...
Successfully installed toolz

store in .local

buildInputs = [pypkgs.pip, ...

Nix on Travis CI

language: nix
env:
  global:
    - STORE=$HOME/nix-store
cache:
  directories:
    - $STORE
before_install:
  - sudo mkdir -p /etc/nix
  - echo "binary-caches = https://cache.nixos.org/ file://$STORE" | sudo tee -a /etc/nix/nix.conf > /dev/null
  - echo 'require-sigs = false' | sudo tee -a /etc/nix/nix.conf > /dev/null
before_cache:
  - mkdir -p $STORE
  - nix copy --to file://$STORE -f default.nix buildInputs
install:
  - nix-shell --pure --command "echo 'run nix-shell once for install'"
script:
  - nix-shell --pure --command "py.test"

Create a local store for cached builds
Copy the build to the local store for caching
Use the same build for development and testing

How to fix Nix

Can't move /nix/store
Disk space can be large
Steep learning curve
Poor documentation
Can't install Python imperatively
Extreme pressure on Nixpkgs from all of PyPi, Node, Caball etc.

Nix Frustrations

Improve nix for Python users

Make nix-env work for Python
- allow imperative installation of Python packages
Maintain multiple versions of major packages in Nixpkgs
Something like callHackage for Python to pull and build directly from PyPI

pandoc = nixpkgs.haskellPackages.callHackage "pandoc" "2.4" {};

Need a declarative standard for Python instead of setup.py, setup.cfg, requirements.txt, MANIFEST.in etc.
Community of users

Happy to help with Nix

Thanks!

overridePythonAttrs

  pypkgs.scipy.overridePythonAttrs rec {
    version = "0.19.1";
    src = pypkgs.fetchPypi {
      pname = "scipy";
      inherit version;
      sha256 = "1rl411bvla6q7qfdb47fpdnyjhfgzl6smpha33n9ar1klykjr6m1";
    };
  };

Override existing nixpkgs recipes
Use overridePythonAttrs to override package attributes
Use override to override package arguments

nix-repl> f = x: x * x  # functions
    
nix-repl> f 2
4

nix-repl> g = x: y: x / y

nix-repl> h = g 9  # currying

nix-repl> h 3
3

nix-repl> { a = 1 / 0; b = 2; }.b
2

nix-repl> { a = 1 / 0; b = 2; }.a  # lazy
error: division by zero, at (string):1:7

Using Nix for Repeatable Python Environments

Daniel Wheeler

Overview

Packaging Concerns

My PYTHON VERSION CHANGED!

Inconsistent outcomes

outcomes are dependent on state!

rollbaCKS

No rollback capability in most package managers

dIAMOND dEPENDENCY

Questions

Why isn't package management more like version control?

What is Nix?

Nix*

How does nIX Work?

Nixpkgs

NIx Package Manager

Nix Expression

Derivation

Derivation is independent of /nix/store

"NixOS: A Purely Functional Linux Distribution"

purely functional data structure (immutable)

repology.org/statistics

Nix Tutorial

The BASics

ImPURE CheatING

Can Only write to the /nix/store!

Read and write files

Two versions of the same file

Instantiate

A Derivation

Python Examples, Tips and Tricks

Imperative Installation

Nix supports imperative installation

BUT NOT FOR PYTHON

Need to use nix-shell

Completely pure environment

Repeatable environment

Script the previous nix-shell command

and maybe even specify the version of nixpkgs

Specify NIXPKGs Version

Fully repeatable by enforcing the Nixpkgs version

Prevent Version Drift!!!

Pypi2NIX

Mixed Language EnvironmentS

Pip Cheats

Nix on Travis CI

How to fix Nix

Nix Frustrations

Improve nix for Python users

Happy to help with Nix

Thanks!

PURE

Using Nix for Repeatable Python Environments

More from Daniel Wheeler