Domen Kožar / @iElectric @ nixcon 2015

Outline

  • History

  • setup.py

  • Caveats in packaging
  • buildPythonPackage
  • Caveats in Nix
  • Upcoming improvements

Distutils

from distutils.core import setup

setup(name="sadness", version="0.1", py_modules=["pure_sadness.py"])
$ cat setup.py
$ sudo python setup.py install

Build and install packages

2000: INCLUDED in Python standard library

Build C extensions

$ python setup.py build_ext

No dependency management

No way to reproduce an installation 

Create a tarball

$ python setup.py sdist

No download capabilities (PyPi)

Setuptools (2004)

from setuptools import setup, find_packages

setup(name="sadness", version="0.1", packages=find_packages())
$ cat setup.py
$ easy_install Django 

Installer for PyPI

Eggs (zipfile binary format)

$ python setup.py build_egg

No sanity in code whatsoever

No release for years since 2009

Monkeypatches distutils :-(

$ python setup.py sdist

No uninstall

Whole packaging process from a single setup.py

TRIVIA

easy_install parses all links on https://pypi.python.org/pypi/Django/1.8

and for each link parsed HTML for a possible tarball release

pip 1.5 (2014-01-01)

stops parsing links

IAN BICKING: VIRTUALENV (2007)

$ virtualenv --no-site-packages test/
New python executable in test/bin/python2.7
Also creating executable in test/bin/python
Installing setuptools...done.

$ ls test
bin  include  lib

$ source bin/activate

IAN BICKING: PIP (2008)

$ pip install Django
$ pip uninstall Django

Ever been frustrated with easy_install? Interested in an alternative?  - Ian Bicking

UX friendly

separates build/install phases

"freeze" dependency versions

TAREK Ziadé: DISTRIBUTE (2008)

TAREK Ziadé: DISTUTILS2 (2008)

Authors will have to write a setup.cfg file and run a few commands to package and distribute their code - Tarek Ziadé

Burnout (2012). Distutils2 abandoned, distribute merges back into setuptools

Providing a backward compatible version to replace Setuptools and make all distributions that depend on Setuptools work as before, but with less bugs and behaviorial issues - Tarek Ziadé

SPECIFICATION DRIVEN DEVELOPMENT

  • PEP 241 (2001): Metadata 1.0
  • PEP 314 (2003): Metadata 1.1 adds license, platform, download URL, ..
  • PEP 345 (2005): Metadata 1.2, better dependencies
  • PEP 376 (2009): Metadata per environment
  • PEP 425 (2012): Compatibility Tags for Built Distributions Version
  • PEP 426 (2012): Metadata 2.0 (draft)
  • PEP 427 (2012): Wheel binary format
  • PEP 440 (2013): Versioning
  • PEP 458 (2013): TUF online keys
  • PEP 459 (2013): Metadata extensions
  • PEP 470 (2014): Removal of external hosting
  • PEP 496 (2015): Environment markers
  • PEP 503 (2015): PyPI simple protocol

setup.py

setup(name='mr.bob',
      version='0.1.3',
      description='Bob renders directory structure templates',
      long_description=read('README.rst') + '\n' + read('HISTORY.rst'),
      classifiers=[
          "Programming Language :: Python :: Implementation :: CPython",
          "Programming Language :: Python :: Implementation :: PyPy",
          "Programming Language :: Python :: 2.7",
          "Programming Language :: Python :: 3",
          "Programming Language :: Python :: 3.3",
          "Programming Language :: Python :: 3.4",
      ],
      author='Domen Kozar, Tom Lazar',
      url='https://github.com/iElectric/mr.bob.git',
      license='BSD',
      packages=find_packages(),
      install_requires=['six>=1.2.0'],
      extras_require={
          'test': [
              'nose',
              'coverage<3.6dev',
              'flake8>2.0',
              'mock',
          ],
          'development': [
              'zest.releaser',
              'Sphinx',
          ],
      },
      entry_points="""
      [console_scripts]
      mrbob = mrbob.cli:main
      """,
      include_package_data=True)
$ nix-shell -p pythonPackages.pbr --command 'python -c "import sys;print(sys.path)"'
['',
 '/nix/store/9cb0j2y4rsh3mvdizhv22maibp8ifawf-python2.7-pbr-0.9.0/lib/python2.7/site-packages',
 '/nix/store/qhnd3bmlvqqvqw2rdy7fk3rc12mw4z2h-python-recursive-pth-loader-1.0/lib/python2.7/site-packages',
 '/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/site-packages',
 '/nix/store/v930s7l861my7haprl8r2w0d9aar29cs-python2.7-setuptools-18.2/lib/python2.7/site-packages',
 '/nix/store/v930s7l861my7haprl8r2w0d9aar29cs-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg',
 '/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python27.zip',
 '/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7',
 '/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/plat-linux2',
 '/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/lib-tk',
 '/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/lib-old',
 '/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/lib-dynload']
$ nix-shell -p pythonPackages.pbr --command "echo \$PYTHONPATH"
/nix/store/9cb0j2y4rsh3mvdizhv22maibp8ifawf-python2.7-pbr-0.9.0/lib/python2.7/site-packages:
/nix/store/qhnd3bmlvqqvqw2rdy7fk3rc12mw4z2h-python-recursive-pth-loader-1.0/lib/python2.7/site-packages:
/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/site-packages:
/nix/store/v930s7l861my7haprl8r2w0d9aar29cs-python2.7-setuptools-18.2/lib/python2.7/site-packages

$PYTHONPATH

sys.path

Caveats in Python packaging

  • setup_requires (build time dependencies)

  • install_requires (run time dependencies)
  • extras_require (optional dependencies)
  • No consistency

  • No convention how to define test dependencies

  • Three separate DAGs

CIRCULAR DEPENDENCIES

  • PIP PHASEs are ALL OR NOTHING

  • mostly ugly workarounds

  1. "bootstrapping"

  2. B.buildInputs = A.override( buildInputs excluding B )

tests: single source of truth

  • No tests in source

  • Mocking prevents testing a dependency upgrade

  • FRAGILE: UID 0, $HOME, networking, transient failures, ...

but (sometimes) ...

PYTHON VERSIONING SUPPORT

EXTREMELY UNDERUSED

  • "Programming Language :: Python :: 2.7"
  • "Programming Language :: Python :: 3.3"
  • "Programming Language :: Python :: 3.4"
  • "Programming Language :: Python :: 3.5"

 

CLASSIFIERS

NoT USED FOR automation

supported versions manually maintained

MANIFEST.in

rules what files to include into tarball

$ cat MANIFEST.in
graft mypackage

but .. 

setuptools-git: invoke git and get a list of files beloning to repository

DEPENDS ON .git AT BUILD/INSTALL

ANYTHING YOU CAN THINK OF SHOULD BE IN SETUP.py

import sys

if sys.version_info < (2, 7):
    install_requires.append("argparse")
try:
    import argparse
except ImportError:
    install_requires.append('argparse')
argparse; python_version < '2.7'(PEP496)
# if Homebrew is installed, use its lib and include directories

            import subprocess
            try:
                prefix = subprocess.check_output(
                    ['brew', '--prefix']
                ).strip().decode('latin1')
            except:
                # Homebrew not installed
                prefix = None

            ft_prefix = None

            if prefix:
                # add Homebrew's include and lib directories
                _add_directory(library_dirs, os.path.join(prefix, 'lib'))
                _add_directory(include_dirs, os.path.join(prefix, 'include'))
                ft_prefix = os.path.join(prefix, 'opt', 'freetype')

            if ft_prefix and os.path.isdir(ft_prefix):
                # freetype might not be linked into Homebrew's prefix
                _add_directory(library_dirs, os.path.join(ft_prefix, 'lib'))
                _add_directory(
                    include_dirs, os.path.join(ft_prefix, 'include'))
            else:
                # fall back to freetype from XQuartz if
                # Homebrew's freetype is missing
                _add_directory(library_dirs, "/usr/X11/lib")
                _add_directory(include_dirs, "/usr/X11/include")

https://github.com/python-pillow/Pillow/blob/master/setup.py

ANYTHING ...

Nix: buildPythonPackage

if disabled
then throw "${name} not supported for interpreter ${python.executable}"
else
python.stdenv.mkDerivation (builtins.removeAttrs attrs ["disabled"] // {
  inherit doCheck;

  name = namePrefix + name;

  buildInputs = [
    wrapPython
    (distutils-cfg.override { extraCfg = distutilsExtraCfg; })
  ] ++ buildInputs ++ pythonPath
    ++ (lib.optional (lib.hasSuffix "zip" attrs.src.name or "") unzip);

  propagatedBuildInputs = propagatedBuildInputs ++ [ recursivePthLoader python setuptools ];
  configurePhase = attrs.configurePhase or ''
    runHook preConfigure

    export DETERMINISTIC_BUILD=1

    sed -i '0,/import distutils/s//import setuptools;import distutils/' setup.py
    sed -i '0,/from distutils/s//import setuptools;from distutils/' setup.py

    runHook postConfigure
  '';
  checkPhase = attrs.checkPhase or ''
      runHook preCheck

      ${python}/bin/${python.executable} setup.py test

      runHook postCheck
  '';
  buildPhase = attrs.buildPhase or ''
    runHook preBuild

    ${python}/bin/${python.executable} setup.py build \
      ${lib.concatStringsSep " " setupPyBuildFlags}

    runHook postBuild
  '';
   installPhase = attrs.installPhase or ''
    runHook preInstall

    mkdir -p "$out/${python.sitePackages}"

    export PYTHONPATH="$out/${python.sitePackages}:$PYTHONPATH"

    ${python}/bin/${python.executable} setup.py install \
      --install-lib=$out/${python.sitePackages} \
      --old-and-unmanageable \
      --prefix="$out" ${lib.concatStringsSep " " setupPyInstallFlags}

    eapth="$out/${python.sitePackages}/easy-install.pth"
    if [ -e "$eapth" ]; then
        mv "$eapth" $(dirname "$eapth")/${name}.pth
    fi

    rm -f $out/${python.sitePackages}/site.py*

    runHook postInstall
  '';
  postFixup = attrs.postFixup or ''
      wrapPythonPrograms

      createBuildInputsPth build-inputs "$buildInputStrings"
      for inputsfile in propagated-build-inputs propagated-native-build-inputs; do
        if test -e $out/nix-support/$inputsfile; then
            createBuildInputsPth $inputsfile "$(cat $out/nix-support/$inputsfile)"
        fi
      done
    '';
  shellHook = attrs.shellHook or ''
    ${preShellHook}
    if test -e setup.py; then
       tmp_path=$(mktemp -d)
       export PATH="$tmp_path/bin:$PATH"
       export PYTHONPATH="$tmp_path/${python.sitePackages}:$PYTHONPATH"
       ${python.interpreter} setup.py develop --prefix $tmp_path
    fi
    ${postShellHook}
  '';

CAVEATS in nix

SETUPTOOLS NAMESPACES #2412

collision between 
`/nix/store/...-logilab-astng-0.24.3/lib/python2.7/site-packages/logilab/__init__.py' and 
`/nix/store/...-logilab-common-0.61.0/lib/python2.7/site-packages/logilab/__init__.py'

are not supported:

will be supported with wheels #329

$ cat setup.py
setup(
    name="logilab.astng",
    # ...
    namespace_packages=['logilab'],
)

$ cat logilab/__init__.py
__import__('pkg_resources').declare_namespace(__name__)

What's going on?

two installed versions won't conflict

If Django 1.6 and Django 1.7 are in closure,

the first one in $PYTHONPATH will be used

will be supported with wheels #329 and the installation will abort

Imperative package management

  1. $ nix-env -i python2.7 python2.7-numpy
  2. $ python -C "import numpy"
    
  3. I got

(#10597, #792, #10693)

YES, Up for discussion

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numpy

is this expected?

Problem

python3 doesn't have $out/bin/python, but $out/bin/python3

sed -i 's@python@${python.interpreter}@' .testr.conf

Solution

Problem

tests depend on binaries from package being installed

Solution

doCheck = false;
doInstallCheck = true;

installCheckPhase = ''
   export PATH=$PATH:$out/bin
   ${python.interpreter} setup.py test
'';

Problem

Solution

propagatedBuildInputs = optionals (!isPyPy) [ cffi ];
propagatedBuildInputs = optionals (pythonOlder "3.4") self.enum34;

some interpreters already ship with a library

Problem

Solution

python decodes files from filesystem based on locale 

buildInputs = [ pkgs.glibcLocales ];

preCheck = ''
  export LC_ALL="en_US.UTF-8"
'';

Problem

Solution

installation is looking for a specific version of the package

substituteInPlace setup.py --replace "httplib2==0.8" "httplib2"

Problem

Solution

ImportError for readline, sqlite3, curses, crypt

propagatedBuildInputs = [ python.modules.curses ];

UPCOMING IMPROVEMENTS

WHEELS (PEP427)

python setup.py bdist_wheel
python setup.py build
pip install *.whl
python setup.py install

In other words

A wheel is a ZIP-format archive with a specially formatted filename and the .whl extension

Generate nix packages

setup.py -> static metadata (PEP426)

HASKELL-ng alike api

backwards incompatibility

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.

 

Questions?

  1. http://packaging.python.org
  2. http://nixos.org/nixpkgs/manual/#sec-python

twitter.com/iElectric

The sorry state of Python packaging and how it reflects in Nix

By Domen Kožar

The sorry state of Python packaging and how it reflects in Nix

  • 3,292