Domen Kožar / @iElectric @ nixcon 2015
Outline
-
History
-
setup.py
- Caveats in packaging
- buildPythonPackage
- Caveats in Nix
- Upcoming improvements
Distutils
from distutils.core import setup
setup(name="sadness", version="0.1", py_modules=["pure_sadness.py"])
$ cat setup.py
$ sudo python setup.py install
Build and install packages
2000: INCLUDED in Python standard library
Build C extensions
$ python setup.py build_ext
No dependency management
No way to reproduce an installation
Create a tarball
$ python setup.py sdist
No download capabilities (PyPi)
Setuptools (2004)
from setuptools import setup, find_packages
setup(name="sadness", version="0.1", packages=find_packages())
$ cat setup.py
$ easy_install Django
Installer for PyPI
Eggs (zipfile binary format)
$ python setup.py build_egg
No sanity in code whatsoever
No release for years since 2009
Monkeypatches distutils :-(
$ python setup.py sdist
No uninstall
Whole packaging process from a single setup.py
TRIVIA
easy_install parses all links on https://pypi.python.org/pypi/Django/1.8
and for each link parsed HTML for a possible tarball release
pip 1.5 (2014-01-01)
stops parsing links
IAN BICKING: VIRTUALENV (2007)
$ virtualenv --no-site-packages test/
New python executable in test/bin/python2.7
Also creating executable in test/bin/python
Installing setuptools...done.
$ ls test
bin include lib
$ source bin/activate
IAN BICKING: PIP (2008)
$ pip install Django
$ pip uninstall Django
Ever been frustrated with easy_install? Interested in an alternative? - Ian Bicking
UX friendly
separates build/install phases
"freeze" dependency versions
TAREK Ziadé: DISTRIBUTE (2008)
TAREK Ziadé: DISTUTILS2 (2008)
Authors will have to write a setup.cfg file and run a few commands to package and distribute their code - Tarek Ziadé
Burnout (2012). Distutils2 abandoned, distribute merges back into setuptools
Providing a backward compatible version to replace Setuptools and make all distributions that depend on Setuptools work as before, but with less bugs and behaviorial issues - Tarek Ziadé
SPECIFICATION DRIVEN DEVELOPMENT
- PEP 241 (2001): Metadata 1.0
- PEP 314 (2003): Metadata 1.1 adds license, platform, download URL, ..
- PEP 345 (2005): Metadata 1.2, better dependencies
- PEP 376 (2009): Metadata per environment
- PEP 425 (2012): Compatibility Tags for Built Distributions Version
- PEP 426 (2012): Metadata 2.0 (draft)
- PEP 427 (2012): Wheel binary format
- PEP 440 (2013): Versioning
- PEP 458 (2013): TUF online keys
- PEP 459 (2013): Metadata extensions
- PEP 470 (2014): Removal of external hosting
- PEP 496 (2015): Environment markers
- PEP 503 (2015): PyPI simple protocol
setup.py
setup(name='mr.bob',
version='0.1.3',
description='Bob renders directory structure templates',
long_description=read('README.rst') + '\n' + read('HISTORY.rst'),
classifiers=[
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.3",
"Programming Language :: Python :: 3.4",
],
author='Domen Kozar, Tom Lazar',
url='https://github.com/iElectric/mr.bob.git',
license='BSD',
packages=find_packages(),
install_requires=['six>=1.2.0'],
extras_require={
'test': [
'nose',
'coverage<3.6dev',
'flake8>2.0',
'mock',
],
'development': [
'zest.releaser',
'Sphinx',
],
},
entry_points="""
[console_scripts]
mrbob = mrbob.cli:main
""",
include_package_data=True)
$ nix-shell -p pythonPackages.pbr --command 'python -c "import sys;print(sys.path)"'
['',
'/nix/store/9cb0j2y4rsh3mvdizhv22maibp8ifawf-python2.7-pbr-0.9.0/lib/python2.7/site-packages',
'/nix/store/qhnd3bmlvqqvqw2rdy7fk3rc12mw4z2h-python-recursive-pth-loader-1.0/lib/python2.7/site-packages',
'/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/site-packages',
'/nix/store/v930s7l861my7haprl8r2w0d9aar29cs-python2.7-setuptools-18.2/lib/python2.7/site-packages',
'/nix/store/v930s7l861my7haprl8r2w0d9aar29cs-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg',
'/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python27.zip',
'/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7',
'/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/plat-linux2',
'/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/lib-tk',
'/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/lib-old',
'/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/lib-dynload']
$ nix-shell -p pythonPackages.pbr --command "echo \$PYTHONPATH"
/nix/store/9cb0j2y4rsh3mvdizhv22maibp8ifawf-python2.7-pbr-0.9.0/lib/python2.7/site-packages:
/nix/store/qhnd3bmlvqqvqw2rdy7fk3rc12mw4z2h-python-recursive-pth-loader-1.0/lib/python2.7/site-packages:
/nix/store/49zg30nx308mad45yd9vd4pk3jlddkks-python-2.7.10/lib/python2.7/site-packages:
/nix/store/v930s7l861my7haprl8r2w0d9aar29cs-python2.7-setuptools-18.2/lib/python2.7/site-packages
$PYTHONPATH
sys.path
Caveats in Python packaging
-
setup_requires (build time dependencies)
- install_requires (run time dependencies)
- extras_require (optional dependencies)
-
No consistency
-
No convention how to define test dependencies
-
Three separate DAGs
CIRCULAR DEPENDENCIES
-
PIP PHASEs are ALL OR NOTHING
-
mostly ugly workarounds
-
"bootstrapping"
-
B.buildInputs = A.override( buildInputs excluding B )
tests: single source of truth
-
No tests in source
-
Mocking prevents testing a dependency upgrade
-
FRAGILE: UID 0, $HOME, networking, transient failures, ...
but (sometimes) ...
PYTHON VERSIONING SUPPORT
EXTREMELY UNDERUSED
- "Programming Language :: Python :: 2.7"
- "Programming Language :: Python :: 3.3"
- "Programming Language :: Python :: 3.4"
- "Programming Language :: Python :: 3.5"
CLASSIFIERS
NoT USED FOR automation
supported versions manually maintained
MANIFEST.in
rules what files to include into tarball
$ cat MANIFEST.in
graft mypackage
but ..
setuptools-git: invoke git and get a list of files beloning to repository
DEPENDS ON .git AT BUILD/INSTALL
ANYTHING YOU CAN THINK OF SHOULD BE IN SETUP.py
import sys
if sys.version_info < (2, 7):
install_requires.append("argparse")
try:
import argparse
except ImportError:
install_requires.append('argparse')
argparse; python_version < '2.7'(PEP496)
# if Homebrew is installed, use its lib and include directories
import subprocess
try:
prefix = subprocess.check_output(
['brew', '--prefix']
).strip().decode('latin1')
except:
# Homebrew not installed
prefix = None
ft_prefix = None
if prefix:
# add Homebrew's include and lib directories
_add_directory(library_dirs, os.path.join(prefix, 'lib'))
_add_directory(include_dirs, os.path.join(prefix, 'include'))
ft_prefix = os.path.join(prefix, 'opt', 'freetype')
if ft_prefix and os.path.isdir(ft_prefix):
# freetype might not be linked into Homebrew's prefix
_add_directory(library_dirs, os.path.join(ft_prefix, 'lib'))
_add_directory(
include_dirs, os.path.join(ft_prefix, 'include'))
else:
# fall back to freetype from XQuartz if
# Homebrew's freetype is missing
_add_directory(library_dirs, "/usr/X11/lib")
_add_directory(include_dirs, "/usr/X11/include")
https://github.com/python-pillow/Pillow/blob/master/setup.py
ANYTHING ...
Nix: buildPythonPackage
if disabled
then throw "${name} not supported for interpreter ${python.executable}"
else
python.stdenv.mkDerivation (builtins.removeAttrs attrs ["disabled"] // {
inherit doCheck;
name = namePrefix + name;
buildInputs = [
wrapPython
(distutils-cfg.override { extraCfg = distutilsExtraCfg; })
] ++ buildInputs ++ pythonPath
++ (lib.optional (lib.hasSuffix "zip" attrs.src.name or "") unzip);
propagatedBuildInputs = propagatedBuildInputs ++ [ recursivePthLoader python setuptools ];
configurePhase = attrs.configurePhase or ''
runHook preConfigure
export DETERMINISTIC_BUILD=1
sed -i '0,/import distutils/s//import setuptools;import distutils/' setup.py
sed -i '0,/from distutils/s//import setuptools;from distutils/' setup.py
runHook postConfigure
'';
checkPhase = attrs.checkPhase or ''
runHook preCheck
${python}/bin/${python.executable} setup.py test
runHook postCheck
'';
buildPhase = attrs.buildPhase or ''
runHook preBuild
${python}/bin/${python.executable} setup.py build \
${lib.concatStringsSep " " setupPyBuildFlags}
runHook postBuild
'';
installPhase = attrs.installPhase or ''
runHook preInstall
mkdir -p "$out/${python.sitePackages}"
export PYTHONPATH="$out/${python.sitePackages}:$PYTHONPATH"
${python}/bin/${python.executable} setup.py install \
--install-lib=$out/${python.sitePackages} \
--old-and-unmanageable \
--prefix="$out" ${lib.concatStringsSep " " setupPyInstallFlags}
eapth="$out/${python.sitePackages}/easy-install.pth"
if [ -e "$eapth" ]; then
mv "$eapth" $(dirname "$eapth")/${name}.pth
fi
rm -f $out/${python.sitePackages}/site.py*
runHook postInstall
'';
postFixup = attrs.postFixup or ''
wrapPythonPrograms
createBuildInputsPth build-inputs "$buildInputStrings"
for inputsfile in propagated-build-inputs propagated-native-build-inputs; do
if test -e $out/nix-support/$inputsfile; then
createBuildInputsPth $inputsfile "$(cat $out/nix-support/$inputsfile)"
fi
done
'';
shellHook = attrs.shellHook or ''
${preShellHook}
if test -e setup.py; then
tmp_path=$(mktemp -d)
export PATH="$tmp_path/bin:$PATH"
export PYTHONPATH="$tmp_path/${python.sitePackages}:$PYTHONPATH"
${python.interpreter} setup.py develop --prefix $tmp_path
fi
${postShellHook}
'';
CAVEATS in nix
SETUPTOOLS NAMESPACES #2412
collision between
`/nix/store/...-logilab-astng-0.24.3/lib/python2.7/site-packages/logilab/__init__.py' and
`/nix/store/...-logilab-common-0.61.0/lib/python2.7/site-packages/logilab/__init__.py'
are not supported:
will be supported with wheels #329
$ cat setup.py
setup(
name="logilab.astng",
# ...
namespace_packages=['logilab'],
)
$ cat logilab/__init__.py
__import__('pkg_resources').declare_namespace(__name__)
What's going on?
two installed versions won't conflict
If Django 1.6 and Django 1.7 are in closure,
the first one in $PYTHONPATH will be used
will be supported with wheels #329 and the installation will abort
Imperative package management
-
$ nix-env -i python2.7 python2.7-numpy
-
$ python -C "import numpy"
- I got
(#10597, #792, #10693)
YES, Up for discussion
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named numpy
is this expected?
Problem
python3 doesn't have $out/bin/python, but $out/bin/python3
sed -i 's@python@${python.interpreter}@' .testr.conf
Solution
Problem
tests depend on binaries from package being installed
Solution
doCheck = false;
doInstallCheck = true;
installCheckPhase = ''
export PATH=$PATH:$out/bin
${python.interpreter} setup.py test
'';
Problem
Solution
propagatedBuildInputs = optionals (!isPyPy) [ cffi ];
propagatedBuildInputs = optionals (pythonOlder "3.4") self.enum34;
some interpreters already ship with a library
Problem
Solution
python decodes files from filesystem based on locale
buildInputs = [ pkgs.glibcLocales ];
preCheck = ''
export LC_ALL="en_US.UTF-8"
'';
Problem
Solution
installation is looking for a specific version of the package
substituteInPlace setup.py --replace "httplib2==0.8" "httplib2"
Problem
Solution
ImportError for readline, sqlite3, curses, crypt
propagatedBuildInputs = [ python.modules.curses ];
UPCOMING IMPROVEMENTS
WHEELS (PEP427)
python setup.py bdist_wheel
python setup.py build
pip install *.whl
python setup.py install
In other words
A wheel is a ZIP-format archive with a specially formatted filename and the .whl extension
Generate nix packages
setup.py -> static metadata (PEP426)
HASKELL-ng alike api
backwards incompatibility
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Questions?
- http://packaging.python.org
- http://nixos.org/nixpkgs/manual/#sec-python
twitter.com/iElectric
The sorry state of Python packaging and how it reflects in Nix
By Domen Kožar
The sorry state of Python packaging and how it reflects in Nix
- 3,292