Outline

• History

• setup.py

• Caveats in packaging
• buildPythonPackage
• Caveats in Nix
• Upcoming improvements

Distutils

from distutils.core import setup

setup(name="sadness", version="0.1", py_modules=["pure_sadness.py"])

$ cat setup.py

Setuptools (2004)

from setuptools import setup, find_packages

setup(name="sadness", version="0.1", packages=find_packages())

$ cat setup.py

TRIVIA

IAN BICKING: VIRTUALENV (2007)

$ virtualenv --no-site-packages test/
New python executable in test/bin/python2.7
Also creating executable in test/bin/python
Installing setuptools...done.

$ ls test
bin  include  lib

$ source bin/activate

TAREK Ziadé: DISTRIBUTE (2008)

Providing a backward compatible version to replace Setuptools and make all distributions that depend on Setuptools work as before, but with less bugs and behaviorial issues - Tarek Ziadé

SPECIFICATION DRIVEN DEVELOPMENT

• PEP 241 (2001): Metadata 1.0
• PEP 314 (2003): Metadata 1.1 adds license, platform, download URL, ..
• PEP 345 (2005): Metadata 1.2, better dependencies
• PEP 376 (2009): Metadata per environment
• PEP 425 (2012): Compatibility Tags for Built Distributions Version
• PEP 426 (2012): Metadata 2.0 (draft)
• PEP 427 (2012): Wheel binary format
• PEP 440 (2013): Versioning
• PEP 458 (2013): TUF online keys
• PEP 459 (2013): Metadata extensions
• PEP 470 (2014): Removal of external hosting
• PEP 496 (2015): Environment markers
• PEP 503 (2015): PyPI simple protocol

setup.py

setup(name='mr.bob',
      version='0.1.3',
      description='Bob renders directory structure templates',
      long_description=read('README.rst') + '\n' + read('HISTORY.rst'),
      classifiers=[
          "Programming Language :: Python :: Implementation :: CPython",
          "Programming Language :: Python :: Implementation :: PyPy",
          "Programming Language :: Python :: 2.7",
          "Programming Language :: Python :: 3",
          "Programming Language :: Python :: 3.3",
          "Programming Language :: Python :: 3.4",
      ],
      author='Domen Kozar, Tom Lazar',
      url='https://github.com/iElectric/mr.bob.git',
      license='BSD',
      packages=find_packages(),
      install_requires=['six>=1.2.0'],
      extras_require={
          'test': [
              'nose',
              'coverage<3.6dev',
              'flake8>2.0',
              'mock',
          ],
          'development': [
              'zest.releaser',
              'Sphinx',
          ],
      },
      entry_points="""
      [console_scripts]
      mrbob = mrbob.cli:main
      """,
      include_package_data=True)

$ nix-shell -p pythonPackages.pbr --command "echo \$PYTHONPATH"
/nix/store/9cb0j2y4rsh3mvdizhv22maibp8ifawf-python2.7-pbr-0.9.0/lib/python2.7/site-packages:
/nix/store/qhnd3bmlvqqvqw2rdy7fk3rc12mw4z2h-python-recursive-pth-loader-1.0/lib/python2.7/site-packages:
/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/site-packages:
/nix/store/v930s7l861my7haprl8r2w0d9aar29cs-python2.7-setuptools-18.2/lib/python2.7/site-packages

$PYTHONPATH

Caveats in Python packaging

• setup_requires (build time dependencies)

• install_requires (run time dependencies)
• extras_require (optional dependencies)

• No consistency

• No convention how to define test dependencies

• Three separate DAGs

CIRCULAR DEPENDENCIES

• PIP PHASEs are ALL OR NOTHING

• mostly ugly workarounds

1. "bootstrapping"

2. B.buildInputs = A.override( buildInputs excluding B )

tests: single source of truth

• No tests in source

• Mocking prevents testing a dependency upgrade

• FRAGILE: UID 0, $HOME, networking, transient failures, ...

PYTHON VERSIONING SUPPORT

• "Programming Language :: Python :: 2.7"
• "Programming Language :: Python :: 3.3"
• "Programming Language :: Python :: 3.4"
• "Programming Language :: Python :: 3.5"

MANIFEST.in

rules what files to include into tarball

ANYTHING YOU CAN THINK OF SHOULD BE IN SETUP.py

# if Homebrew is installed, use its lib and include directories

            import subprocess
            try:
                prefix = subprocess.check_output(
                    ['brew', '--prefix']
                ).strip().decode('latin1')
            except:
                # Homebrew not installed
                prefix = None

            ft_prefix = None

            if prefix:
                # add Homebrew's include and lib directories
                _add_directory(library_dirs, os.path.join(prefix, 'lib'))
                _add_directory(include_dirs, os.path.join(prefix, 'include'))
                ft_prefix = os.path.join(prefix, 'opt', 'freetype')

            if ft_prefix and os.path.isdir(ft_prefix):
                # freetype might not be linked into Homebrew's prefix
                _add_directory(library_dirs, os.path.join(ft_prefix, 'lib'))
                _add_directory(
                    include_dirs, os.path.join(ft_prefix, 'include'))
            else:
                # fall back to freetype from XQuartz if
                # Homebrew's freetype is missing
                _add_directory(library_dirs, "/usr/X11/lib")
                _add_directory(include_dirs, "/usr/X11/include")

https://github.com/python-pillow/Pillow/blob/master/setup.py

ANYTHING ...

Nix: buildPythonPackage

if disabled
then throw "${name} not supported for interpreter ${python.executable}"
else

python.stdenv.mkDerivation (builtins.removeAttrs attrs ["disabled"] // {
  inherit doCheck;

  name = namePrefix + name;

  buildInputs = [
    wrapPython
    (distutils-cfg.override { extraCfg = distutilsExtraCfg; })
  ] ++ buildInputs ++ pythonPath
    ++ (lib.optional (lib.hasSuffix "zip" attrs.src.name or "") unzip);

  propagatedBuildInputs = propagatedBuildInputs ++ [ recursivePthLoader python setuptools ];

  configurePhase = attrs.configurePhase or ''
    runHook preConfigure

    export DETERMINISTIC_BUILD=1

    sed -i '0,/import distutils/s//import setuptools;import distutils/' setup.py
    sed -i '0,/from distutils/s//import setuptools;from distutils/' setup.py

    runHook postConfigure
  '';

  checkPhase = attrs.checkPhase or ''
      runHook preCheck

      ${python}/bin/${python.executable} setup.py test

      runHook postCheck
  '';

  buildPhase = attrs.buildPhase or ''
    runHook preBuild

    ${python}/bin/${python.executable} setup.py build \
      ${lib.concatStringsSep " " setupPyBuildFlags}

    runHook postBuild
  '';

   installPhase = attrs.installPhase or ''
    runHook preInstall

    mkdir -p "$out/${python.sitePackages}"

    export PYTHONPATH="$out/${python.sitePackages}:$PYTHONPATH"

    ${python}/bin/${python.executable} setup.py install \
      --install-lib=$out/${python.sitePackages} \
      --old-and-unmanageable \
      --prefix="$out" ${lib.concatStringsSep " " setupPyInstallFlags}

    eapth="$out/${python.sitePackages}/easy-install.pth"
    if [ -e "$eapth" ]; then
        mv "$eapth" $(dirname "$eapth")/${name}.pth
    fi

    rm -f $out/${python.sitePackages}/site.py*

    runHook postInstall
  '';

  postFixup = attrs.postFixup or ''
      wrapPythonPrograms

      createBuildInputsPth build-inputs "$buildInputStrings"
      for inputsfile in propagated-build-inputs propagated-native-build-inputs; do
        if test -e $out/nix-support/$inputsfile; then
            createBuildInputsPth $inputsfile "$(cat $out/nix-support/$inputsfile)"
        fi
      done
    '';

  shellHook = attrs.shellHook or ''
    ${preShellHook}
    if test -e setup.py; then
       tmp_path=$(mktemp -d)
       export PATH="$tmp_path/bin:$PATH"
       export PYTHONPATH="$tmp_path/${python.sitePackages}:$PYTHONPATH"
       ${python.interpreter} setup.py develop --prefix $tmp_path
    fi
    ${postShellHook}
  '';

CAVEATS in nix

SETUPTOOLS NAMESPACES #2412

two installed versions won't conflict

If Django 1.6 and Django 1.7 are in closure,

the first one in $PYTHONPATH will be used

Imperative package management

1. $ nix-env -i python2.7 python2.7-numpy

2. $ python -C "import numpy"
   

3. I got

(#10597, #792, #10693)

Problem

python3 doesn't have $out/bin/python, but $out/bin/python3

Problem

tests depend on binaries from package being installed

Problem

some interpreters already ship with a library

Problem

python decodes files from filesystem based on locale 

Problem

installation is looking for a specific version of the package

Problem

ImportError for readline, sqlite3, curses, crypt

UPCOMING IMPROVEMENTS

WHEELS (PEP427)

A wheel is a ZIP-format archive with a specially formatted filename and the .whl extension

Generate nix packages

setup.py -> static metadata (PEP426)

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.

Questions?

1. http://packaging.python.org
2. http://nixos.org/nixpkgs/manual/#sec-python

twitter.com/iElectric