Untangle Python Spaghetti

Deep dive into environments and dependencies management

https://slides.com/cheukting_ho/untangle-python-spaghetti/

Hello I am Cheuk

  • Open-Source contributor


     
  • Organisers of community events


     
  • PSF director and fellow
     
  • Community manager at OpenSSF

Do you use virtual enviroments?

(Which one do you use?)

Why do we care about virtual enviroments?

Top 10 risks with OSS

drum rolls.....

Top 10 risks with OSS

  1. Known Vulnerabilities
  2. Compromise of Legitimate Package
  3. Name Confusion Attacks
  4. Unmaintained Software
  5. Outdated Software
  6. Untracked Dependencies
  7. License Risk
  8. Immature Software
  9. Unapproved Changes (mutable)
  10. Under/over-sized Dependency

Why do we care about virtual enviroments?

  • avoid conflicts
  • preserve versions
  • discard when not needed
  • improve supply chain security

Where are your package stored?

How Python finds your packages?

How Python find your packages?

import statement

-> searches for both Python code and extension modules

-> search along `sys.path`

Python's default package manager

Pip

Pip can

  • pip install
        - install packages (with dependencies)
  • pip list
        - show all the packages
  • pip uninstall
        - remove packages

It is important to keep Pip updated

It is also important to keep pip and python in sync

What if you have multiple versions of Python?

venv

venv

  • comes with CPython
  • the basic environment manager
  • others are based on it: pipenv, poetry

Using venv

Using venv

create a new environment

-> create a folder structures to...

  • store packages and extensions that you installed
  • some scripts for the functionalities

conda

conda

  • comes with Anaconda
  • environment manager that...
  • double as package manager
    (conda install)
  • Built with Python 🙌

conda is popular in the data science community

  • Data packages involve lots of binaries
  • e.g. Numpy
  • One-stop shop to install everything
  • Getting compatible binaries  rather than build from source

Using conda

Using conda

  • Similar mechanics but...
  • live by default in the envs/ folder of your Conda directory
  • Independent from venv
  • work on system level
  • works with other languages as well

Using conda

  • conda create --name {env_name}
  • conda create --name {env_name} {python==3.7.5}
  • conda activate {env_name}

 

  • conda env export -f environment.yml
  • conda env create -f environment.yml

Which one?

venv

  • if you are installing from CPython
     
  • if you want to stay in Python ecosystem
     
  • if you are using pip only to install from PyPI

conda

  • if you are using Anaconda
     
  • if you want to manage packages is other languages as well
     
  • if you want your package to come from Anaconda.org

Or both!

(let me show you)

Now you have no excuse to not manage your environment well

Reference

Untangle Python Spaghetti

By Cheuk Ting Ho

Untangle Python Spaghetti

  • 371