What I do ?

What I will be talking about ?

• How to manage multiple python versions ?
• How to manage virtual environments ?
• How to manage project dependencies ?

Pyenv - Python version management

(https://github.com/pyenv/pyenv)

• Tool for managing multiple versions of Python interpreter
• Supports CPython, PyPy, Stackless, JPython, IronPython
• Similar to nvm (node.js) and rvm, rbenv (Ruby)
• Does not work on Windows

How pyenv works ?

Pyenv inserts ~/.pyenv/shims at the beginning of PATH, which contains shims, small executables, which are responsible for passing commands to correct pyenv.

When we're invoking a command e.g. pip, following steps are performed:

1. Search PATH for executable called pip
2. Find pyenv shim named pip
3. Execute shim named pip, which passes command to correct pyenv

How pyenv chooses Python version ?

1. Environment variable PYENV_VERSION (pyenv shell command)
2. .python-version file in current directory (pyenv local command)
3. .python-version file in parent directories (search up to /)
4. ~/.pyenv/version file (pyenv global command)

Pyenv update

(https://github.com/pyenv/pyenv-update)

• Pyenv plugin that provides option to update pyenv with pyenv update command
• Updates not only pyenv, but also other installed plugins, e.g. pyenv-virtualenv
• Updates available Python versions

How to use pyenv ?

Managing available Python versions

Managing currently used Python version

Virtual environments in Python

How virtualenv works ?

• Activate - inserts /path/to/env/bin at the beginning of PATH
• sys.path - list of search paths for modules during imports
• 3rd party virtualenv - modified site.py, orig-prefix.txt allows to include standard library (sys.prefix)
• venv - pyvenv.cfg file (sys.prefix i sys.base_prefix)
• Packages are installed to /path/to/env/lib/pythonX.Y/site-packages

home = /usr/bin
include-system-site-packages = false
version = 3.5.2

Example pyvenv.cfg file:

How to use virtualenvs ?

pip install virtualenv
virtualenv /path/to/env

Python < 3.3

pyvenv /path/to/env

Python 3.3 and higher (deprecated since version 3.6)

Recommended since version 3.5

python3 -m venv /path/to/env

Activate newly created virtualenv

source /path/to/env/bin/activate

Deactivate virtualenv

deactivate

Pyenv + virtualenv

(https://github.com/pyenv/pyenv-virtualenv)

• Pyenv plugin for managing virtual environments with pyenv
• Has support for managing conda environments
• Supports third-party virtualenv as well as venv from standard library

How to use pyenv-virtualenv ?

Creating virtualenvs with pyenv-virtualenv

Working with virtualenvs

Listing existing virtualenvs

Removing virtualenvs

Conda

(https://github.com/conda/conda)

• Handles python versions, environments and packages management
• Multi-platform (Win/macOS/Linux)
• Supports pip for installing packages in conda environments
• Installs binary packages (no need to compile during installation)
• Possible to install packages written in other languages like C/C++, Java, Scala, JavaScript, FORTRAN, R
• Comes with Anaconda and Miniconda

How to use conda ?

Creating environments

Working with environments

Listing existing environments

Removing environments

How to use conda ?

Installing packages

List installed packages

Uninstalling packages

Why should we care about dependency management ?

• List of external libraries that our application/script needs to run (very often in form of requirements.txt file)
• Logical separation of dependencies (e.g. separate list for tests, separate list for people to read, separate list for machines)
• Stable and repeatable builds on different environments

pipreqs

(https://github.com/bndr/pipreqs)

Generates requirements.txt file with dependencies, based on imported packages in your project

Example usage:

Example result in requirements.txt:

pipreqs /path/to/project

wheel==0.23.0
Yarg==0.1.9
docopt==0.6.2

pip-tools

(https://github.com/jazzband/pip-tools)

• Two separate lists of dependencies - for humans and for machines
• Tool that let's you "pin" your dependencies
• pip-compile + pip-sync

How to use pip-tools ?

Flask
requests

requirements.in file:

pip-compile
pip-compile --output-file requirements.txt requirements.in

Using pip-compile to produce requirements.txt:

requirements.txt file:

certifi==2017.7.27.1      # via requests
chardet==3.0.4            # via requests
click==6.7                # via flask
flask==0.12.2
idna==2.6                 # via requests
itsdangerous==0.24        # via flask
jinja2==2.9.6             # via flask
markupsafe==1.0           # via jinja2
requests==2.18.4
urllib3==1.22             # via requests
werkzeug==0.12.2          # via flask

How to use pip-tools ?

pip-compile --upgrade-package flask

Update specific package:

pip-compile --generate-hashes

Using pip-compile with generating hashes:

requirements.txt file with hashes:

certifi==2017.7.27.1 \
    --hash=sha256:40523d2efb60523e113b44602298f0960e900388cf3bb6043f645cf57ea9e3f5 \
    --hash=sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704 \
    # via requests
(...)

Syncing current environment with requirements.txt:

pip-sync
pip-sync requirements.txt

Pipfile

(https://github.com/pypa/pipfile)

• "Requirements.txt" 2.0 - proposition of new standard
• Supported by PyPA (Python Packaging Authority)
• Replace "requirements.txt" with Pipfile and Pipfile.lock
• Pipfile - uses TOML syntax, contains top-level dependencies
• Pipfile.lock - contains deterministic set of dependencies, used for creating repeatable environments for your application

Pipfile example

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"

[packages]
requests = "*"


[requires]
python_version = "3.6"

Pipfile.lock example

{
    "_meta": {
        "hash": {
            "sha256": "a97c6ee0bb0a72c606f78b9f7a8088b61d838cb6312a26574d2126ad09acfead"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.6.2",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "4.4.0-53-generic",
            "platform_system": "Linux",
            "platform_version": "#74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016",
            "python_full_version": "3.6.2",
            "python_version": "3.6",
            "sys_platform": "linux"
        },
        "pipfile-spec": 3,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704",
                "sha256:40523d2efb60523e113b44602298f0960e900388cf3bb6043f645cf57ea9e3f5"
            ],
            "version": "==2017.7.27.1"
        },

pipenv

(https://github.com/kennethreitz/pipenv)

• Tool recommended by PyPA
• Mixture of pip, Pipfile, virtualenv
• Automatically creates virtualenvs for you
• Automatically adds/removes dependencies from Pipfile during installation/uninstallation
• Creates Pipfile.lock file
• Can install dependencies from Pipfile.lock
• By default uses hashes for packages verification
• Integrates with pyenv for python version management

How to use pipenv ?

pipenv --python 3.6.2

Creating new project:

pipenv install requests
pipenv install pytest --dev
pipenv uninstall requests

Installing and uninstalling packages:

Installing all dependencies:

pipenv install
pipenv install --dev

Spawning a shell with virtualenv:

pipenv shell

How to use pipenv ?

pipenv graph

Showing dependency graph:

Checking installed dependencies for security vulnerabilities:

Generate lockfile:

pipenv lock

Summary

Installing packages: pip, conda, pipenv

Managing Python installations: conda, pyenv

Environments: virtualenv, pyenv-virtualenv, conda, pipenv

Managing dependencies: pipreqs, pip-tools, pipenv

Thanks!

@p_grzesik

pj.grzesik@gmail.com