Python Packaging

Jakub Wasielak

http://blog.pykonik.org/

http://koderek.edu.pl/

facebook.com/startechkrk

https://pl.pycon.org/2017/

What? Why?

Architecture

https://packaging.python.org/current/

Installation Tool Recommendations

pip
virtualenv

Packaging Tool Recommendations

setuptools
bdist_wheel
twine

$ python setup.py --help-commands
Standard commands:
  build             build everything needed to install
  build_py          "build" pure Python modules (copy to build directory)
  build_ext         build C/C++ extensions (compile/link to build directory)
  build_clib        build C/C++ libraries used by Python extensions
  build_scripts     "build" scripts (copy and fixup #! line)
  clean             clean up temporary files from 'build' command
  install           install everything from build directory
  install_lib       install all Python modules (extensions and pure Python)
  install_headers   install C/C++ header files
  install_scripts   install scripts (Python or otherwise)
  install_data      install data files
  sdist             create a source distribution (tarball, zip file, etc.)
  register          register the distribution with the Python package index
  bdist             create a built (binary) distribution
  bdist_dumb        create a "dumb" built distribution
  bdist_rpm         create an RPM distribution
  bdist_wininst     create an executable installer for MS Windows
  upload            upload binary package to PyPI
  check             perform some checks on the package

Extra commands:
  ... this one goes on

Asking for help?

setup.py

import os
from setuptools import setup

setup(
    name = "an_example_pypi_project",
    version = "0.0.4",
    author = "Jakub Wasielak",
    author_email = "kuba.wasielak@gmail.com",
    description = ("An demonstration of how to create, document, and publish "
                                   "to the cheese shop a5 pypi.org."),
    license = "BSD",
    keywords = "example documentation tutorial",
    url = "http://packages.python.org/an_example_pypi_project",
    packages=['an_example_pypi_project', 'tests'],
    long_description=read('README'),
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Topic :: Utilities",
        "License :: OSI Approved :: BSD License",
    ],
)

(source: https://pythonhosted.org/an_example_pypi_project/setuptools.html)

setup.py

import os
from setuptools import setup

setup(
    name = "an_example_pypi_project",
    version = "0.0.4",
    author = "Jakub Wasielak",
    author_email = "kuba.wasielak@gmail.com",
    description = ("An demonstration of how to create, document, and publish "
                                   "to the cheese shop a5 pypi.org."),
    license = "BSD",
    keywords = "example documentation tutorial",
    url = "http://packages.python.org/an_example_pypi_project",
    packages=['an_example_pypi_project', 'tests'],
    long_description=read('README'),
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Topic :: Utilities",
        "License :: OSI Approved :: BSD License",
    ],
)

(source: https://pythonhosted.org/an_example_pypi_project/setuptools.html)

setup.py

import os
from setuptools import setup

setup(
    name = "an_example_pypi_project",
    version = "0.0.4",
    author = "Jakub Wasielak",
    author_email = "kuba.wasielak@gmail.com",
    description = ("An demonstration of how to create, document, and publish "
                                   "to the cheese shop a5 pypi.org."),
    license = "BSD",
    keywords = "example documentation tutorial",
    url = "http://packages.python.org/an_example_pypi_project",
    packages=['an_example_pypi_project', 'tests'],
    long_description=read('README'),
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Topic :: Utilities",
        "License :: OSI Approved :: BSD License",
    ],
)

setuptools_scm

setup(
    name = "an_example_pypi_project",
    use_scm_version=True,
    setup_requires=['setuptools_scm'],
    # ...
)

PEP 440

https://www.python.org/dev/peps/pep-0440/

setup.py

import os
from setuptools import setup

setup(
    name = "an_example_pypi_project",
    version = "0.0.4",
    author = "Jakub Wasielak",
    author_email = "kuba.wasielak@gmail.com",
    description = ("An demonstration of how to create, document, and publish "
                                   "to the cheese shop a5 pypi.org."),
    license = "BSD",
    keywords = "example documentation tutorial",
    url = "http://packages.python.org/an_example_pypi_project",
    packages=['an_example_pypi_project', 'tests'],
    long_description=read('README'),
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Topic :: Utilities",
        "License :: OSI Approved :: BSD License",
    ],
)

setup.py

import os
from setuptools import setup

setup(
    name = "an_example_pypi_project",
    version = "0.0.4",
    author = "Jakub Wasielak",
    author_email = "kuba.wasielak@gmail.com",
    description = ("An demonstration of how to create, document, and publish "
                                   "to the cheese shop a5 pypi.org."),
    license = "BSD",
    keywords = "example documentation tutorial",
    url = "http://packages.python.org/an_example_pypi_project",
    packages=['an_example_pypi_project', 'tests'],
    long_description=read('README'),
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Topic :: Utilities",
        "License :: OSI Approved :: BSD License",
    ],
)

(source: https://pythonhosted.org/an_example_pypi_project/setuptools.html)

setup.py

import os
from setuptools import setup

setup(
    name = "an_example_pypi_project",
    version = "0.0.4",
    author = "Jakub Wasielak",
    author_email = "kuba.wasielak@gmail.com",
    description = ("An demonstration of how to create, document, and publish "
                                   "to the cheese shop a5 pypi.org."),
    license = "BSD",
    keywords = "example documentation tutorial",
    url = "http://packages.python.org/an_example_pypi_project",
    packages=['an_example_pypi_project', 'tests'],
    long_description=read('README'),
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Topic :: Utilities",
        "License :: OSI Approved :: BSD License",
    ],
)

(source: https://pythonhosted.org/an_example_pypi_project/setuptools.html)

setup.py

import os
from setuptools import setup

setup(
    name = "an_example_pypi_project",
    version = "0.0.4",
    author = "Jakub Wasielak",
    author_email = "kuba.wasielak@gmail.com",
    description = ("An demonstration of how to create, document, and publish "
                                   "to the cheese shop a5 pypi.org."),
    license = "BSD",
    keywords = "example documentation tutorial",
    url = "http://packages.python.org/an_example_pypi_project",
    packages=['an_example_pypi_project', 'tests'],
    long_description=read('README'),
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Topic :: Utilities",
        "License :: OSI Approved :: BSD License",
    ],
)

(source: https://pythonhosted.org/an_example_pypi_project/setuptools.html)

setup.py

Best classifier?

"Private :: Do Not Upload"

setup.py

import os
from setuptools import setup

setup(
    name = "an_example_pypi_project",
    version = "0.0.4",
    author = "Jakub Wasielak",
    author_email = "kuba.wasielak@gmail.com",
    description = ("An demonstration of how to create, document, and publish "
                                   "to the cheese shop a5 pypi.org."),
    license = "BSD",
    keywords = "example documentation tutorial",
    url = "http://packages.python.org/an_example_pypi_project",
    packages=['an_example_pypi_project', 'tests'],
    long_description=read('README'),
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Topic :: Utilities",
        "License :: OSI Approved :: BSD License",
    ],
)

(source: https://pythonhosted.org/an_example_pypi_project/setuptools.html)

setup.py

import os
from setuptools import setup, find_packages

PACKAGES = find_packages(where="src")

setup(
    # ...
    packages=PACKAGES,
    # ...
)

(source: https://pythonhosted.org/an_example_pypi_project/setuptools.html)

There's more!

setup(
    name = "an_example_pypi_project",
    # ...
    install_requires=[
        "cherrypy==3.5",
        "lxml",
        "Pillow>=2.1,<3dev"
    ],
)

Extras

setup(
    # ...
    install_requires=[
        "cherrypy>=3.5,<3.6dev",
        "lxml",
        "Pillow>=2.1,<3dev"
    ],
    extras_require=dict(
        doc = ['Sphinx>=1.3'],
        notebook = ['notebook', 'ipywidgets'],
        # ... (^ comes from IPython)
    )
)

python setup.py install 'ipython[notebook]'

Tests? Why not!

setup(
    # ...
    install_requires=[
        "cherrypy>=3.5,<3.6dev",
        "lxml",
        "Pillow>=2.1,<3dev"
    ],
    tests_require=[
        'Pyro>=3.16,<4dev',
        'pytest>=2.3',
        'selenium'
    ]
)

python setup.py test

or...

setup(
    # ...
    install_requires=[
        "cherrypy>=3.5,<3.6dev",
        "lxml",
        "Pillow>=2.1,<3dev"
    ],
    extras_require={
        'testing': [
            'Pyro>=3.16,<4dev',
            'pytest>=2.3',
            'selenium'
        ]
    }
)

And tox to install

Entry Points

setup(
    # ...
    'entry_points': {
        'console_scripts': ['virtualenv=virtualenv:main'],
    },
)

$ python virtualenv.py my_venv

vs.

$ virtualenv my_venv

setup.cfg

[global]
verbose = 1

[bdist_wheel]
universal = 1

[metadata]
license_file = LICENSE

[easy_install]
index_url = https://devpi.company.net/root/sth/+simple/

[tool:pytest]
norecursedirs = build env services *.egg project/lib/test

MANIFEST.in

include CHANGES.txt
include project/handlers/*.html
recursive-include project/static

setuptools_scm

everything not in .gitignore will get used!

install vs. develop

$ python setup.py install
...
$ pip freeze | grep project
project==22.2

$ python setup.py develop
...
$ pip freeze | grep project
-e git+ssh://you@your.repo.url/Repositories/Team/project@id_123#egg=project

vs.

eggs

wheels

Wheels advantages

official PEP (pep-0427)
no .pyc files inside (one wheel for both Pythons, .pyc files will be generated upon installation)
richer file naming conversion
versioning
installation of C components does not require compiler

distribution-1.0-1-py27-none-any.whl

No pycrypto, SQLAlchemy, MySQL-python, tornado

Collecting requirements-dev.txt
Installing collected packages: requirements-dev.txt

requirements.txt

$ pip install requirements-dev.txt

devpi

http://doc.devpi.net/latest/

Your projects, your packages

.pypirc

[distutils]
index-servers =
    my_devpi

[my_devpi]
repository: https://devpi-master.company.net/root/my_devpi/
username: {your username}
password: {your password}

python setup.py sdist upload -r my_devpi

python setup.py sdist register -r my_devpi upload -r my_devpi

Upload

Or twine

twine upload dist/*

python setup.py sdist bdist_wheel

Create

Upload

export TWINE_USERNAME=foo
export TWINE_PASSWORD=bar
twine upload dist/*

Better Upload

Test your package

.pypirc

[distutils]
index-servers =
    test

[test]
repository: https://testpypi.python.org/pypi
# repository: https://test.pypi.org/legacy/
username: {your username}
password: {your password}

pip install -i https://testpypi.python.org/pypi <package_name>

pip install dist/package-1.0.0.tar.gz
pip install dist/package-1.0.0-py2.py3-none-any.whl

PEP 20, last line

Namespaces are one honking great idea -- let's do more of those!

Namespaces

sound/                          Top-level package
      __init__.py               Initialize the sound package
      formats/                  Subpackage for file format conversions
              __init__.py
              wavread.py
              wavwrite.py
              aiffread.py
              aiffwrite.py
              auread.py
              auwrite.py
              ...
      effects/                  Subpackage for sound effects
              __init__.py
              echo.py
              surround.py
              reverse.py
              ...
      filters/                  Subpackage for filters
              __init__.py
              equalizer.py
              vocoder.py
              karaoke.py
              ...

Namespaces

import sound.effects.echo
import sound.effects.surround
from sound.effects import *

pip cool features

(that easy_install doesn't have)

easy_install can finish up with a partially completed installation
better console output
reasons for actions are kept
native support for git, mercurial, bazaar
uninstallation of packages
pip freeze
pip install -r requirements.txt

What's next?

https://www.pypa.io/en/latest/roadmap/

Pipfile

[[source]]
url = 'https://pypi.python.org/simple'
verify_ssl = true

[requires]
python_version = '2.7'

[packages]
requests = { extras = ['socks'] }
Django = '>1.10'
pinax = { git = 'git://github.com/pinax/pinax.git', ref = '1.4', editable = true }

[dev-packages]
nose = '*'

Using TOML (Tom's Obvious, Minimal Language)

PyPI?

https://pypi.python.org/pypi

https://pypi.org/

Warehouse

https://github.com/pypa/warehouse

THANKS!

QUESTIONS?

https://about.me/jakub.wasielak

python packaging

By Kuba Wasielak

python packaging

1,753

Python Packaging

What? Why?

Architecture

https://packaging.python.org/current/

Installation Tool Recommendations

Packaging Tool Recommendations

Asking for help?

setup.py

setup.py

setup.py

setuptools_scm

setup.py

setup.py

setup.py

setup.py

setup.py

Best classifier?

setup.py

setup.py

There's more!

Extras

Tests? Why not!

or...

Entry Points

setup.cfg

MANIFEST.in

setuptools_scm

install vs. develop

vs.

eggs

wheels

Wheels advantages

requirements.txt

devpi

Your projects, your packages

Or twine

Test your package

PEP 20, last line

Namespaces

Namespaces

pip cool features

What's next?

Pipfile

PyPI?

Warehouse

Recommended reading

THANKS!

QUESTIONS?

python packaging

More from Kuba Wasielak