Python

Packaging

Jan. 2023

Content

  • Python package: Concepts/Terminology
  • Developing a package
  • Build the package for its distribution
  • Publish the package on PyPi
  • Install your package from PyPi
  • Productionize your package/packaging

Concepts

# 1. CONCEPTS
  • Python script?
  • Python module?
  • Python package?
  • Python (standard) library?

Concepts: script & module

# 1. CONCEPTS
# in mod_add.py

def add(a, b):
  return a+b
# in my_script.py

def add(a, b):
  return a+b

result = add(30,12)
print(result)

Script

  • meant to be run
  • statements outside functions or class

Module

  • meant to be imported into scripts or other modules
  • defines functions, classes, ... for use in scripts

Concepts: module called in a script

# 1. CONCEPTS
# in mod_add.py

def add(a, b):
  return a+b
# in my_script.py

from mod_add import add


result = add(30,12)
print(result)

Script

  • meant to be run
  • statements outside functions or class

Module

  • meant to be imported into scripts or other modules
  • defines functions, classes, ... for use in scripts

Concepts: {script, module} all-in-one

# 1. CONCEPTS

Script

  • meant to be run
  • statements outside functions or class

Module

  • meant to be imported into scripts or other modules
  • defines functions, classes, ... for use in scripts
# in (exec.) mod_add.py

def add(a, b):
  return a+b

if __name__ == "__main__":
  result = add(30,12)
  print(result)
# in my_script.py

from mod_add import add


result = add(30,12)
print(result)

Concepts: package

# 1. CONCEPTS
  • a collection of related modules and/or sub-packages
  • can be an individual module, but usually a collection of them

Concepts: package (example)

# 1. CONCEPTS

codetiming package (source: Real Python)

Concepts: libraries

# 1. CONCEPTS

Collection of modules & packages

  • Python standard library
  • Third-party libraries

What are Python Libraries (source)

Concepts: Python standard library

# 1. CONCEPTS
  • a collection of modules & packages
  • come bundled with every installation of python
  • examples: os, sys, math, dateutil, email, json, ...

What are Python Libraries (source)

Concepts: third-party libraries

# 1. CONCEPTS
  • requests
  • Django
  • Flask
  • SQLAlchemy
  • numpy
  • scipy
  • pandas
  • pytest
  • pylint
  • flake8
  • psycopg2
  • ...

Example 3rd-party libraries: Jupyter

# 1. CONCEPTS

Packaging Flow

# 2. PACKAGING FLOW

The Python package cycle (source: Python Packages)

Packaging Flow

# 2. PACKAGING FLOW

Develop & Configure a Python Package

  - Create your Source Tree

  - Package configuration file

Build & Distribute a Python Package

  - Build Artifacts (source distribution, built distribution)

  - Upload to the package distribution service (PyPi, Artifactory, ...)

  - Binaries & Dependencies

Download & Install

History, Tips & Challenges

# 2. PACKAGING FLOW

Packaging Demystified (source: PyCon2021 Bernát)

Developing a package

# 3. DEMO PACKAGE

The Python package cycle (source: Python Packages)

Calculating \( \pi \)

\pi = 4 (1-\frac{1}{3}+\frac{1}{5}-\frac{1}{7}+\cdots)

The Gregory-Leibniz Series
(proof: @stanford.edu)

cf. Packaging Demystified (source: PyCon2021 Bernát)

# 3. DEMO PACKAGE

Developing a package

Create your Source Tree (folders, subfolders, files) with Poetry

➜  poetry new approx_pi
Created package approx_pi in approx_pi

➜  tree approx_pi/
approx_pi
├── README.md
├── approx_pi
│   └── __init__.py
├── pyproject.toml
└── tests
    └── __init__.py

2 directories, 4 files
# 3. DEMO PACKAGE

Developing a package

# pi_approx.py
from math import pi

def approximate_pi(iteration_count: int) -> float:
  sign, result = 1, 0.0
  for at in range(iteration_count):
    result += sign / (2 * at + 1)
    sign *= -1
  return result * 4

if __name__ == "__main__":
  approx_1, approx_2 = approximate_pi(300), approximate_pi(301)
  print(f"approx 300 {approx_1} with diff {approx_1 - pi}")
  print(f"approx 301 {approx_2} with diff {approx_2 - pi}")
  

Add functionalities to your main module

pi_approx.py
# 3. DEMO PACKAGE

Developing a package

➜ # Activating the virtual environment 
➜  poetry shell
Spawning shell within /home/patricme/.cache/pypoetry/virtualenvs/approx-pi-BE-Xuvxi-py3.9
➜  . /home/patricme/.cache/pypoetry/virtualenvs/approx-pi-BE-Xuvxi-py3.9/bin/activate
(approx-pi-py3.9)

➜ # Installing dependencies
➜  poetry install
Creating virtualenv approx-pi-BE-Xuvxi-py3.9 in /home/patricme/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Create a local environment & Install dependencies (Poetry)

# 3. DEMO PACKAGE

Developing a package

# Running a python script
(approx-pi-py3.9) ➜ python3 approx_pi/pi_approx.py
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987

# Running a python module
(approx-pi-py3.9) ➜ python3 -m approx_pi.pi_approx
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987
# 3. DEMO PACKAGE

Developing a package

Running your Python package

Adding a dependency from PyPi

(approx-pi-py3.9) ➜  poetry add cowsay
Using version ^5.0 for cowsay
Updating dependencies
Resolving dependencies... (9.2s)
Writing lock file
Package operations: 1 install, 0 updates, 0 removals
  • Installing cowsay (5.0)
# pi_approx.py
...

if __name__ == "__main__":
    approx_1, approx_2 = approximate_pi(300), approximate_pi(301)
    print(f"approx 300 {approx_1} with diff {approx_1 - pi}")
    print(f"approx 301 {approx_2} with diff {approx_2 - pi}")
    cowsay.cow(f"The difference between \nthe approximations is \n{'%.2E' % Decimal(approx_1-approx_2)}")
# 3. DEMO PACKAGE
# pyproject.toml
[tool.poetry]
name = "approx-pi"
version = "0.1.0"
description = ""
authors = ["Patrick Merlot <patrick.merlot@gmail.com>"]
readme = "README.md"
packages = [{include = "approx_pi"}]

[tool.poetry.dependencies]
python = "^3.9"
cowsay = "^5.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
# 3. DEMO PACKAGE

Developing a package

Updated config file after adding the new dependency (cowsay)

(approx-pi-py3.9) ➜ python3 -m approx_pi.pi_approx
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987
  ______________________
 /                      \
| The difference between |
| the approximations is  |
| -6.66E-03              |
 \                      /
  ======================
                      \
                       \
                         ^__^
                         (oo)\_______
                         (__)\       )\/\
                             ||----w |
                             ||     ||
# 3. DEMO PACKAGE

Running your Python package

Developing a package

Build a distribution package

# 4. BUILD DISTRIBTUION PACKAGE 

The Python package cycle (source: Python Packages)

Build the package's artifacts

(approx-pi-py3.9) ➜ poetry build
Building approx-pi (0.1.0)
  - Building sdist
  - Built approx_pi-0.1.0.tar.gz
  - Building wheel
  - Built approx_pi-0.1.0-py3-none-any.whl

the source distribution (sdist) & the built distribution (wheels)

# 4. BUILD DISTRIBTUION PACKAGE 

Publishing your Python Package to PyPI

# 5. PUBLISH DISTRIBTUION PACKAGE 

The Python package cycle (source: Python Packages)

Publishing your Python Package to PyPI

PyPI (pypi.org) is the default publishing target for Poetry.

 

In order to publish a package to PyPI, you will need to create an account. Go to the official registration page in your web browser.

# 5. PUBLISH DISTRIBTUION PACKAGE 

Publishing your Python Package to PyPI

Token authentication is the recommended way to use your PyPI account in the command line.

You can use a single, automatically generated token instead of a username and password.

Enabling Token Authentication for PyPI.

 

# 5. PUBLISH DISTRIBTUION PACKAGE 

Publishing your Python Package to PyPI

Add your API token to Poetry with this command

 

➜ poetry config pypi-token.pypi <your-api-token>
Using a plaintext file to store credentials

Publishing your Python Package to PyPI

(approx-pi-py3.9) ➜  poetry publish

Publishing approx-pi (0.1.0) to PyPI
 - Uploading approx_pi-0.1.0-py3-none-any.whl 100%
 - Uploading approx_pi-0.1.0.tar.gz 100%
# 5. PUBLISH DISTRIBTUION PACKAGE 

Publishing your Python Package to PyPI

Check your published package in your PyPI projects, in the browser.

 

# 5. PUBLISH DISTRIBTUION PACKAGE 

Your package is available for everyone to use via:
Pip, Poetry, Conda, Pipenv, ...

Installing a Python Package from PyPI

# 6. INSTALLING A PACKAGE 

The Python package cycle (source: Python Packages)

Adding a dependency from PyPi

➜  poetry add cowsay
Using version ^5.0 for cowsay
Updating dependencies
Resolving dependencies... (9.2s)
Writing lock file
Package operations: 1 install, 0 updates, 0 removals
  • Installing cowsay (5.0)
➜  poetry add <any-python-package-available-on-pypi>
# 6. INSTALLING A PACKAGE 
➜  poetry add approx-pi

Version control your code on (public) repo.

# 7. VERSION CONTROL 

The Python package cycle (source: Python Packages)

Collaborate via Git repository providers

# 7. VERSION CONTROL 

Version Control with Git

# Initialize your project (ONCE!)
git init

# Tell git where is the main repository (Github? Bitbucket? GitLab? ...)
# (ONCE!)
git remote add github https://github.com/Patechoc/approx_pi.git

# Specify the files that you want to add to the next "push" to the main repository
git add pyproject.toml approx_pi.py/ README.md

# Give a description to the changes you want to push
git commit -a -m "Minimum file structure for our python package"

# Push the changes that are committed to the main repository
git push 
  • Tracking code changes (backup, undo, feature dev.)
  • Coding collaboration (simplified merge, branching)
  • Get started: www.w3schools.com/git/
# 7. VERSION CONTROL 

Version Control with Git

Create useful .gitignore files for your project (source: Toptal)

# Created by https://www.toptal.com/developers/gitignore/api/python
# Edit at https://www.toptal.com/developers/gitignore?templates=python

### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
#   For a library or package, you might want to ignore these files since the code is
#   intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
#   However, in case of collaboration, if having platform-specific dependencies or dependencies
#   having no cross-platform support, pipenv may install dependencies that don't work, or not
#   install all needed dependencies.
#Pipfile.lock

# poetry
#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
#   This is especially recommended for binary packages to ensure reproducibility, and is more
#   commonly ignored for libraries.
#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
#   in version control.
#   https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
#  and can be added to the global gitignore or merged into this file.  For a more nuclear
#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

### Python Patch ###
# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
poetry.toml

# ruff
.ruff_cache/

# End of https://www.toptal.com/developers/gitignore/api/python
n
# 7. VERSION CONTROL 

Don't save everything!

Testing

# 8. PRODUCTIONIZING YOUR PYTHON PACKAGE

Getting started with testing in Python (source: Real Python)

Testing

# 8. PRODUCTIONIZING YOUR PYTHON PACKAGE

Code Security Check

Bandit is a tool designed to find common security issues in Python code.

To do this, Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files, it generates a report.

# 8. PRODUCTIONIZING YOUR PYTHON PACKAGE

Coding standards

Use isort for sorting the library imports (yes, imports have a suggested order).
Check the existence of undesired artifacts using Flake8 and Pylint.

Keep the code within the same style using Black.
Those tools can be configured to be PEP8 compliant. PEP8 — Python Enhancement Proposal, is a style guide that provides guidelines and best practices suggestions on how to write Python code.

How to write beautiful code with PEP 8 (source: Real Python)

# 8. PRODUCTIONIZING YOUR PYTHON PACKAGE

Choose a license

# 8. PRODUCTIONIZING YOUR PYTHON PACKAGE

Documentating your package

Documenting Python Code: A Complete Guide (source: Real Python)

# 8. PRODUCTIONIZING YOUR PYTHON PACKAGE

Continuous integration & deployment

ex. CI/CD with CircleCI (source: Real Python)

Just push your code to the git repository,
let an automated pipeline test, build, publish your package!

# 8. PRODUCTIONIZING YOUR PYTHON PACKAGE

Python Package Project Template

A tool that can be configured to include all best practices
and get started with your package in seconds!

# 8. PRODUCTIONIZING YOUR PYTHON PACKAGE

References

# 15. REFERENCES

- Python.org:

   - Python Packaging User Guide: https://packaging.python.org

   - Module & Packages (tutorial chap. 6)

- Python Packaging Demystified (Bernàt, PyCon 2021)

- Publishing (Perfect) Python Packages on PyPi (Smith, EuroPython 2019)

- Python Packages by Beuzen & Timbers: https://py-pkgs.org/

- Codetiming: Good example of best practices in Python

- More complex setup: The Sheer Joy of Packaging! (multi-language)

- Python Modules and Packages by Real Python (Jan. 2020)

- Scripts, Modules, Packages, and Libraries by Real Python (May. 2020)

 

Packaging-python

By Patrick Merlot

Packaging-python

  • 15