Poetry

What is Poetry?

Wikipedia defines "poetry" as:

Poetry (derived from the Greek poiesis, "making"), also called verse, is a form of literature that uses aesthetic and often rhythmic qualities of language − such as phonaesthetics, sound symbolism, and metre − to evoke meanings in addition to, or in place of, a prosaic ostensible meaning. [1]

[1] https://en.wikipedia.org/wiki/Poetry

Famous Poets

  • William Shakespear
  • Oscar Wilde
  • Emily Dickinson
  • Maya Angelou
  • Robert Frost

What is Poetry?

In terms of why we're here:

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a lockfile to ensure repeatable installs, and can build your project for distribution. [1]

[1] https://python-poetry.org/docs/

What is Poetry?

Poetry is a tool that helps manage dependencies of a project and has been really helpful in replacing Python's antiquated requirement.txt.

 

Hopefully this talk will convince you to make the switch!

Why Poetry?

  • Python does not have a great way to have repeatable builds with lots of dependencies
  • Pyproject.toml is starting to be an accepted resource for Python projects
  • Poetry gives a nice CLI experience when dealing with Python packages

pip install -r req.txt 😭

  • running pip install -r requirements.txt updates packages that are in your requirements.txt file
  • It also updates the latest dependencies of those packages, unless they themselves are in the file
  • This can lead to some headaches with dependencies of dependencies in large projects

Example

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed.
This behaviour is the source of the following dependency conflicts.

indigoag-service-registry-cli 1.13.0 
    requires click<9.0.0,>=8.0.0,
    but you have click 7.1.2 which is incompatible.
indigoag-service-registry-cli 
    1.13.0 requires datadog>=0.44.0,
    but you have datadog 0.42.0 which is incompatible.
indigoag-service-registry-cli 
    1.13.0 requires sentry-sdk>=1.10.0,
    but you have sentry-sdk 1.5.1 which is incompatible.
indigoag-service-registry-cli 
    1.13.0 requires tenacity~=8.0.1,
    but you have tenacity 6.3.1 which is incompatible.
black 22.12.0 
    requires click>=8.0.0,
    but you have click 7.1.2 which is incompatible.

pyproject.toml is great!

  • There's been a few PEPs talking about how to manage a Python project's information
  • PEP 508 [1] discusses this thoroughly!
  • The Python community decided pyproject.toml to be the answer

[1] https://peps.python.org/pep-0508/

ADS already uses this!

  • The ADS project already has a pyproject.toml
  • It's used to give defaults to some of the tooling we use (like isort and black)

ADS's pyproject.toml

[tool.black]
skip-string-normalization = true

[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
line_length = 88

[tool.semantic_release]
# We bump both stored versions whenever either of the two packages that are published via ADS are published.
# Why? semantic-release doesn't support multiple packages from one repo super well. If we didn't
# bump both, there would be issues reconciling the current version in these files vs the version found in git.
# The downside is that either package may have some gaps in version numbers in codeartifact.
version_variable = [
    "indigoag/ads/__init__.py:__version__",
    "core/indigoag/ads_core/__init__.py:__version__",
]
upload_to_release = false
patch_without_tag = false
commit_message = "chore(release): [skip ci]"
branch = "develop"

Poetry plugs right in

  • Poetry is built to use pyproject.toml
  • It's just another tool to pyproject.toml!

Poetry in pyproject.toml

[tool.poetry]
name = "indigoag.morpheus"
version = "0.37.1"
description = "Morpheus is a framework for modeling synthetic and experimental agronomic data."

[[tool.poetry.source]]
name = "indigo"
url = "https://indigoag-217541722159.d.codeartifact.us-east-1.amazonaws.com/pypi/indigoag.repository/simple/"

[tool.poetry.dependencies]
"indigoag.ads" = ">=23.0.0"

Poetry is nice

  • The CLI tool for Poetry is second-to-none
  • It controls adding/removing packages with ease
  • It gives you insight into your installed packages

$ poetry --list

Available commands:
  about              Shows information about Poetry.
  add                Adds a new dependency to pyproject.toml.
  build              Builds a package, as a tarball and a wheel by default.
  check              Checks the validity of the pyproject.toml file.
  config             Manages configuration settings.
  export             Exports the lock file to alternative formats.
  help               Displays help for a command.
  init               Creates a basic pyproject.toml file in the current directory.
  install            Installs the project dependencies.
  list               Lists commands.
  lock               Locks the project dependencies.
  new                Creates a new Python project at <path>.
  publish            Publishes a package to a remote repository.
  remove             Removes a package from the project dependencies.
  run                Runs a command in the appropriate environment.
  search             Searches for packages on remote repositories.
  shell              Spawns a shell within the virtual environment.
  show               Shows information about packages.
  update             Update the dependencies as according to the pyproject.toml file.
  version            Shows the version of the project or bumps it when a valid bump rule is provided.

cache
  cache clear        Clears Poetry's cache.
  cache list         List Poetry's caches.

$ poetry --list (cont)

debug
  debug info         Shows debug information.
  debug resolve      Debugs dependency resolution.

env
  env info           Displays information about the current environment.
  env list           Lists all virtualenvs associated with the current project.
  env remove         Remove virtual environments associated with the project.
  env use            Activates or creates a new virtualenv for the current project.

 self
  self add           Add additional packages to Poetry's runtime environment.
  self install       Install locked packages (incl. addons) required by this Poetry installation.
  self lock          Lock the Poetry installation's system requirements.
  self remove        Remove additional packages from Poetry's runtime environment.
  self show          Show packages from Poetry's runtime environment.
  self show plugins  Shows information about the currently installed plugins.
  self update        Updates Poetry to the latest version.

 source
  source add         Add source configuration for project.
  source remove      Remove source configured for the project.
  source show        Show information about sources configured for the project.

How Poetry

We'll go over the steps required to set up Poetry for a new project, and go into some best practices.

Install Poetry

# INSTALL
$ curl -sSL https://install.python-poetry.org | python3 -

$ pipx install poetry

# AUTOCOMPLETE
$ poetry completions bash >> ~/.bash_completion

$ poetry completions zsh > ~/.zfunc/_poetry

# Then modify ~/.zshrc file to have:

```
fpath+=~/.zfunc
autoload -Uz compinit && compinit
```

Use Poetry

$ poetry new indigo-project

indigo-project
├── pyproject.toml
├── README.md
├── indigo_project
│   └── __init__.py
└── tests
    └── __init__.py

Use Poetry (Cont)

[tool.poetry]
name = "indigo-project"
version = "0.1.0"
description = ""
authors = ["Indigo Engineering <engineering@indigoag.com>"]
readme = "README.md"
packages = [{include = "indigo_project"}]

[tool.poetry.dependencies]
python = "^3.7"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Use Poetry in Existing Proj

$ cd existing-indigo-project

$ poetry init

$ cat requirements.txt | xargs poetry add

$ poetry install

Adding Dependencies

$ cd existing-indigo-project

$ poetry add flask

$ poetry add --group dev pytest==5.8.2

$ poetry add --group docs mkdocs

[tool.poetry.dependencies]
python = "^3.7"
flask = "*"

[tool.poetry.group.dev.dependencies]
pytest = "5.4.3"

[tool.poetry.group.docs.dependencies]
mkdocs = "^1.4.2"

Note: [tool.poetry.dev-dependencies] is deprecated as of version 1.2. Though it's still supported...

 

Also, for other groups will need `poetry install --with docs`

Removing Dependencies

$ cd existing-indigo-project

$ poetry remove mkdocs --group docs

[tool.poetry.dependencies]
python = "^3.7"
flask = "*"

[tool.poetry.group.dev.dependencies]
pytest = "5.4.3"

Poetry Lock File

  • It keeps a list of all versions of package dependencies
  • Also keeps a list of your package dependencies dependencies so you get a more consistent rebuild experience

Poetry lock file 🥺

[[package]]
name = "atomicwrites"
version = "1.4.1"
description = "Atomic file writes."
category = "dev"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"

[[package]]
name = "attrs"
version = "22.2.0"
description = "Classes Without Boilerplate"
category = "dev"
optional = false
python-versions = ">=3.6"

[package.extras]
cov = ["attrs[tests]", "coverage-enable-subprocess", "coverage[toml] (>=5.3)"]
dev = ["attrs[docs,tests]"]
docs = ["furo", "myst-parser", "sphinx", "sphinx-notfound-page", "sphinxcontrib-towncrier", "towncrier", "zope.interface"]
tests = ["attrs[tests-no-zope]", "zope.interface"]
tests-no-zope = ["cloudpickle", "hypothesis", "mypy (>=0.971,<0.990)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "pytest-xdist[psutil]"]
tests_no_zope = ["cloudpickle", "hypothesis", "mypy (>=0.971,<0.990)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "pytest-xdist[psutil]"]

[[package]]
name = "click"
version = "8.1.3"
description = "Composable command line interface toolkit"
category = "main"
optional = false
python-versions = ">=3.7"

Poetry Lock File

  • Luckily it's handled exclusively by Poetry
  • It keeps all versions and the hash of the version for extra resilience
  • Causes some headaches with git merge conflicts.

Conclusion

  • Poetry is great
  • We should be using it everywhere
  • It is not that hard to convert and adopt Poetry
  • It will save us in the long-run from having issues with package dependencies of dependencies