Cheuk Ting Ho
Developer advocate / Data Scientist - support open-source and building the community.
Untangle Python Spaghetti
Deep dive into environments and dependencies management
https://slides.com/cheukting_ho/untangle-python-spaghetti/
Hello I am Cheuk
-
Open-Source contributor
-
Organisers of community events
-
PSF director and fellow
- Community manager at OpenSSF
- Part of the Linux Foundation
- Focus on Open Source Security
- Not for Profit
- Security newsletters and blog
- Free courses:
Do you use virtual enviroments?
(Which one do you use?)
Why do we care about virtual enviroments?
Top 10 risks with OSS
drum rolls.....
Top 10 risks with OSS
- Known Vulnerabilities
- Compromise of Legitimate Package
- Name Confusion Attacks
- Unmaintained Software
- Outdated Software
- Untracked Dependencies
- License Risk
- Immature Software
- Unapproved Changes (mutable)
- Under/over-sized Dependency
Why do we care about virtual enviroments?
- avoid conflicts
- preserve versions
- discard when not needed
- improve supply chain security
Where are your package stored?
How Python finds your packages?
How Python find your packages?
import statement
-> searches for both Python code and extension modules
-> search along `sys.path`
Python's default package manager
Pip
Pip can
- pip install
- install packages (with dependencies) - pip list
- show all the packages - pip uninstall
- remove packages
It is important to keep Pip updated
It is also important to keep pip and python in sync
What if you have multiple versions of Python?
venv
venv
- comes with CPython
- the basic environment manager
- others are based on it: pipenv, poetry
Using venv
Using venv
create a new environment
-> create a folder structures to...
- store packages and extensions that you installed
- some scripts for the functionalities
conda
conda
- comes with Anaconda
- environment manager that...
- double as package manager
(conda install) - Built with Python 🙌
conda is popular in the data science community
- Data packages involve lots of binaries
- e.g. Numpy
- One-stop shop to install everything
- Getting compatible binaries rather than build from source
Using conda
Using conda
- Similar mechanics but...
- live by default in the
envs/folder of your Conda directory - Independent from venv
- work on system level
- works with other languages as well
Using conda
- conda create --name {env_name}
- conda create --name {env_name} {python==3.7.5}
- conda activate {env_name}
- conda env export -f environment.yml
- conda env create -f environment.yml
Which one?
venv
- if you are installing from CPython
- if you want to stay in Python ecosystem
- if you are using pip only to install from PyPI
conda
- if you are using Anaconda
- if you want to manage packages is other languages as well
- if you want your package to come from Anaconda.org
Or both!
(let me show you)
Now you have no excuse to not manage your environment well
Reference
Untangle Python Spaghetti
By Cheuk Ting Ho
