Virtual environments and dependency management in Python

Piotr Grzesik

What I do ?

What I will be talking about ?

  • How to manage multiple python versions ?
  • How to manage virtual environments ?
  • How to manage project dependencies ?

Pyenv - Python version management

(https://github.com/pyenv/pyenv)

  • Tool for managing multiple versions of Python interpreter
  • Supports CPython, PyPy, Stackless, JPython, IronPython
  • Similar to nvm (node.js) and rvm, rbenv (Ruby)
  • Does not work on Windows

How pyenv works ?

Pyenv inserts ~/.pyenv/shims at the beginning of PATH, which contains shims, small executables, which are responsible for passing commands to correct pyenv.

 

When we're invoking a command e.g. pip, following steps are performed:

  1. Search PATH for executable called pip
  2. Find pyenv shim named pip
  3. Execute shim named pip, which passes command to correct pyenv

How pyenv chooses Python version ?

  1. Environment variable PYENV_VERSION (pyenv shell command)
  2. .python-version file in current directory (pyenv local command)
  3. .python-version file in parent directories (search up to /)
  4. ~/.pyenv/version file (pyenv global command)

Pyenv update

(https://github.com/pyenv/pyenv-update)

  • Pyenv plugin that provides option to update pyenv with pyenv update command
  • Updates not only pyenv, but also other installed plugins, e.g. pyenv-virtualenv
  • Updates available Python versions

How to use pyenv ?

pyenv install 3.6.2
pyenv install anaconda3-4.4.0 

pyenv versions

pyenv uninstall 2.7

Managing available Python versions

pyenv version

pyenv shell 3.6.2
pyenv local anaconda3-4.4.0 
pyenv global 3.6.2

Managing currently used Python version

Virtual environments in Python

  • Independent, isolated environments
  • Each environment has separate dependencies (e.g. two environments with different versions of the same library)
  • Allow testing and using third-party libraries without polluting global Python installation
  • Part of standard library starting from Python 3.3 (PEP 405)

How virtualenv works ?

  • Activate - inserts /path/to/env/bin at the beginning of PATH
  • sys.path - list of search paths for modules during imports
  • 3rd party virtualenv - modified site.py, orig-prefix.txt allows to include standard library (sys.prefix)
  • venv - pyvenv.cfg file (sys.prefix i sys.base_prefix)
  • Packages are installed to /path/to/env/lib/pythonX.Y/site-packages 
home = /usr/bin
include-system-site-packages = false
version = 3.5.2

Example pyvenv.cfg file:

How to use virtualenvs ?

pip install virtualenv
virtualenv /path/to/env

Python < 3.3

pyvenv /path/to/env

Python 3.3 and higher (deprecated since version 3.6)

Recommended since version 3.5

python3 -m venv /path/to/env

Activate newly created virtualenv

source /path/to/env/bin/activate

Deactivate virtualenv

deactivate

Pyenv + virtualenv

(https://github.com/pyenv/pyenv-virtualenv)

  • Pyenv plugin for managing virtual environments with pyenv
  • Has support for managing conda environments
  • Supports third-party virtualenv as well as venv from standard library

How to use pyenv-virtualenv ?

pyenv virtualenv 3.6.2 my-venv
pyenv virtualenv my-other-venv

Creating virtualenvs with pyenv-virtualenv

pyenv activate my-venv
pyenv deactivate

Working with virtualenvs

Listing existing virtualenvs

pyenv virtualenvs

Removing virtualenvs

pyenv virtualenv-delete my-venv

Conda

(https://github.com/conda/conda)

  • Handles python versions, environments and packages management
  • Multi-platform (Win/macOS/Linux)
  • Supports pip for installing packages in conda environments
  • Installs binary packages (no need to compile during installation)
  • Possible to install packages written in other languages like C/C++, Java, Scala, JavaScript, FORTRAN, R
  • Comes with Anaconda and Miniconda

How to use conda ?

conda create --name my-env
conda create --name my-other-env keras
conda create --name my-another-env python=3.6
conda create --name cloned-env --clone my-env

Creating environments

activate my-env
source activate my-env
deactivate
source deactivate

Working with environments

Listing existing environments

conda info --envs
conda env list

Removing environments

conda remove --name my-env --all

How to use conda ?

conda install scipy
conda install scipy=0.12.0
conda install --name my-env scipy

Installing packages

conda list

List installed packages

Uninstalling packages

conda remove scipy
conda remove --name my-env scipy

Why should we care about dependency management ?

  • List of external libraries that our application/script needs to run (very often in form of requirements.txt file)
  • Logical separation of dependencies (e.g. separate list for tests, separate list for people to read, separate list for machines)
  • Stable and repeatable builds on different environments

pipreqs

(https://github.com/bndr/pipreqs)

Generates requirements.txt file with dependencies, based on imported packages in your project

 

Example usage:

 

Example result in requirements.txt:

pipreqs /path/to/project
wheel==0.23.0
Yarg==0.1.9
docopt==0.6.2

pip-tools

(https://github.com/jazzband/pip-tools)

  • Two separate lists of dependencies - for humans and for machines
  • Tool that let's you "pin" your dependencies
  • pip-compile + pip-sync

How to use pip-tools ?

Flask
requests

requirements.in file:

pip-compile
pip-compile --output-file requirements.txt requirements.in

Using pip-compile to produce requirements.txt:

requirements.txt file:

certifi==2017.7.27.1      # via requests
chardet==3.0.4            # via requests
click==6.7                # via flask
flask==0.12.2
idna==2.6                 # via requests
itsdangerous==0.24        # via flask
jinja2==2.9.6             # via flask
markupsafe==1.0           # via jinja2
requests==2.18.4
urllib3==1.22             # via requests
werkzeug==0.12.2          # via flask

How to use pip-tools ?

pip-compile --upgrade-package flask

Update specific package:

pip-compile --generate-hashes

Using pip-compile with generating hashes:

requirements.txt file with hashes:

certifi==2017.7.27.1 \
    --hash=sha256:40523d2efb60523e113b44602298f0960e900388cf3bb6043f645cf57ea9e3f5 \
    --hash=sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704 \
    # via requests
(...)

Syncing current environment with requirements.txt:

pip-sync
pip-sync requirements.txt

Pipfile

(https://github.com/pypa/pipfile)

  • "Requirements.txt" 2.0 - proposition of new standard
  • Supported by PyPA (Python Packaging Authority)
  • Replace "requirements.txt" with Pipfile and Pipfile.lock
  • Pipfile - uses TOML syntax, contains top-level dependencies
  • Pipfile.lock - contains deterministic set of dependencies, used for creating repeatable environments for your application

Pipfile example

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"

[packages]
requests = "*"


[requires]
python_version = "3.6"

Pipfile.lock example

{
    "_meta": {
        "hash": {
            "sha256": "a97c6ee0bb0a72c606f78b9f7a8088b61d838cb6312a26574d2126ad09acfead"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.6.2",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "4.4.0-53-generic",
            "platform_system": "Linux",
            "platform_version": "#74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016",
            "python_full_version": "3.6.2",
            "python_version": "3.6",
            "sys_platform": "linux"
        },
        "pipfile-spec": 3,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704",
                "sha256:40523d2efb60523e113b44602298f0960e900388cf3bb6043f645cf57ea9e3f5"
            ],
            "version": "==2017.7.27.1"
        },

pipenv

(https://github.com/kennethreitz/pipenv)

  • Tool recommended by PyPA
  • Mixture of pip, Pipfile, virtualenv
  • Automatically creates virtualenvs for you
  • Automatically adds/removes dependencies from Pipfile during installation/uninstallation
  • Creates Pipfile.lock file
  • Can install dependencies from Pipfile.lock
  • By default uses hashes for packages verification
  • Integrates with pyenv for python version management

How to use pipenv ?

pipenv --python 3.6.2

Creating new project:

pipenv install requests
pipenv install pytest --dev
pipenv uninstall requests

Installing and uninstalling packages:

Installing all dependencies:

pipenv install
pipenv install --dev

Spawning a shell with virtualenv:

pipenv shell

How to use pipenv ?

pipenv graph

Showing dependency graph:

pipenv check

Checking installed dependencies for security vulnerabilities:

Generate lockfile:

pipenv lock

Summary

Installing packages: pip, conda, pipenv

Managing Python installations: conda, pyenv

Environments: virtualenv, pyenv-virtualenv, conda, pipenv

Managing dependencies: pipreqs, pip-tools, pipenv

Thanks!

@p_grzesik

pj.grzesik@gmail.com

Made with Slides.com