Poetry

What is Poetry?

Wikipedia defines "poetry" as:

Poetry (derived from the Greek poiesis, "making"), also called verse, is a form of literature that uses aesthetic and often rhythmic qualities of language − such as phonaesthetics, sound symbolism, and metre − to evoke meanings in addition to, or in place of, a prosaic ostensible meaning. [1]

[1] https://en.wikipedia.org/wiki/Poetry

Famous Poets

William Shakespear
Oscar Wilde
Emily Dickinson
Maya Angelou
Robert Frost

What is Poetry?

In terms of why we're here:

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a lockfile to ensure repeatable installs, and can build your project for distribution. [1]

[1] https://python-poetry.org/docs/

What is Poetry?

Poetry is a tool that helps manage dependencies of a project and has been really helpful in replacing Python's antiquated requirement.txt.

Hopefully this talk will convince you to make the switch!

Why Poetry?

Python does not have a great way to have repeatable builds with lots of dependencies
Pyproject.toml is starting to be an accepted resource for Python projects
Poetry gives a nice CLI experience when dealing with Python packages

pip install -r req.txt 😭

running pip install -r requirements.txt updates packages that are in your requirements.txt file
It also updates the latest dependencies of those packages, unless they themselves are in the file
This can lead to some headaches with dependencies of dependencies in large projects

Example

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed.
This behaviour is the source of the following dependency conflicts.

indigoag-service-registry-cli 1.13.0 
    requires click<9.0.0,>=8.0.0,
    but you have click 7.1.2 which is incompatible.
indigoag-service-registry-cli 
    1.13.0 requires datadog>=0.44.0,
    but you have datadog 0.42.0 which is incompatible.
indigoag-service-registry-cli 
    1.13.0 requires sentry-sdk>=1.10.0,
    but you have sentry-sdk 1.5.1 which is incompatible.
indigoag-service-registry-cli 
    1.13.0 requires tenacity~=8.0.1,
    but you have tenacity 6.3.1 which is incompatible.
black 22.12.0 
    requires click>=8.0.0,
    but you have click 7.1.2 which is incompatible.

pyproject.toml is great!

There's been a few PEPs talking about how to manage a Python project's information
PEP 508 [1] discusses this thoroughly!
The Python community decided pyproject.toml to be the answer

[1] https://peps.python.org/pep-0508/

ADS already uses this!

The ADS project already has a pyproject.toml
It's used to give defaults to some of the tooling we use (like isort and black)

ADS's pyproject.toml

[tool.black]
skip-string-normalization = true

[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
line_length = 88

[tool.semantic_release]
# We bump both stored versions whenever either of the two packages that are published via ADS are published.
# Why? semantic-release doesn't support multiple packages from one repo super well. If we didn't
# bump both, there would be issues reconciling the current version in these files vs the version found in git.
# The downside is that either package may have some gaps in version numbers in codeartifact.
version_variable = [
    "indigoag/ads/__init__.py:__version__",
    "core/indigoag/ads_core/__init__.py:__version__",
]
upload_to_release = false
patch_without_tag = false
commit_message = "chore(release): [skip ci]"
branch = "develop"

Poetry plugs right in

Poetry is built to use pyproject.toml
It's just another tool to pyproject.toml!

Poetry in pyproject.toml

[tool.poetry]
name = "indigoag.morpheus"
version = "0.37.1"
description = "Morpheus is a framework for modeling synthetic and experimental agronomic data."

[[tool.poetry.source]]
name = "indigo"
url = "https://indigoag-217541722159.d.codeartifact.us-east-1.amazonaws.com/pypi/indigoag.repository/simple/"

[tool.poetry.dependencies]
"indigoag.ads" = ">=23.0.0"

Poetry is nice

The CLI tool for Poetry is second-to-none
It controls adding/removing packages with ease
It gives you insight into your installed packages

$ poetry --list

Available commands:
  about              Shows information about Poetry.
  add                Adds a new dependency to pyproject.toml.
  build              Builds a package, as a tarball and a wheel by default.
  check              Checks the validity of the pyproject.toml file.
  config             Manages configuration settings.
  export             Exports the lock file to alternative formats.
  help               Displays help for a command.
  init               Creates a basic pyproject.toml file in the current directory.
  install            Installs the project dependencies.
  list               Lists commands.
  lock               Locks the project dependencies.
  new                Creates a new Python project at <path>.
  publish            Publishes a package to a remote repository.
  remove             Removes a package from the project dependencies.
  run                Runs a command in the appropriate environment.
  search             Searches for packages on remote repositories.
  shell              Spawns a shell within the virtual environment.
  show               Shows information about packages.
  update             Update the dependencies as according to the pyproject.toml file.
  version            Shows the version of the project or bumps it when a valid bump rule is provided.

cache
  cache clear        Clears Poetry's cache.
  cache list         List Poetry's caches.

$ poetry --list (cont)

debug
  debug info         Shows debug information.
  debug resolve      Debugs dependency resolution.

env
  env info           Displays information about the current environment.
  env list           Lists all virtualenvs associated with the current project.
  env remove         Remove virtual environments associated with the project.
  env use            Activates or creates a new virtualenv for the current project.

 self
  self add           Add additional packages to Poetry's runtime environment.
  self install       Install locked packages (incl. addons) required by this Poetry installation.
  self lock          Lock the Poetry installation's system requirements.
  self remove        Remove additional packages from Poetry's runtime environment.
  self show          Show packages from Poetry's runtime environment.
  self show plugins  Shows information about the currently installed plugins.
  self update        Updates Poetry to the latest version.

 source
  source add         Add source configuration for project.
  source remove      Remove source configured for the project.
  source show        Show information about sources configured for the project.

How Poetry

We'll go over the steps required to set up Poetry for a new project, and go into some best practices.

Install Poetry

# INSTALL
$ curl -sSL https://install.python-poetry.org | python3 -

$ pipx install poetry

# AUTOCOMPLETE
$ poetry completions bash >> ~/.bash_completion

$ poetry completions zsh > ~/.zfunc/_poetry

# Then modify ~/.zshrc file to have:

```
fpath+=~/.zfunc
autoload -Uz compinit && compinit
```

Use Poetry

$ poetry new indigo-project

indigo-project
├── pyproject.toml
├── README.md
├── indigo_project
│   └── __init__.py
└── tests
    └── __init__.py

Use Poetry (Cont)

[tool.poetry]
name = "indigo-project"
version = "0.1.0"
description = ""
authors = ["Indigo Engineering <engineering@indigoag.com>"]
readme = "README.md"
packages = [{include = "indigo_project"}]

[tool.poetry.dependencies]
python = "^3.7"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Use Poetry in Existing Proj

$ cd existing-indigo-project

$ poetry init

$ cat requirements.txt | xargs poetry add

$ poetry install

Adding Dependencies

$ cd existing-indigo-project

$ poetry add flask

$ poetry add --group dev pytest==5.8.2

$ poetry add --group docs mkdocs

[tool.poetry.dependencies]
python = "^3.7"
flask = "*"

[tool.poetry.group.dev.dependencies]
pytest = "5.4.3"

[tool.poetry.group.docs.dependencies]
mkdocs = "^1.4.2"

Note: [tool.poetry.dev-dependencies] is deprecated as of version 1.2. Though it's still supported...

Also, for other groups will need `poetry install --with docs`

Removing Dependencies

$ cd existing-indigo-project

$ poetry remove mkdocs --group docs

[tool.poetry.dependencies]
python = "^3.7"
flask = "*"

[tool.poetry.group.dev.dependencies]
pytest = "5.4.3"

Poetry Lock File

It keeps a list of all versions of package dependencies
Also keeps a list of your package dependencies dependencies so you get a more consistent rebuild experience

Poetry lock file 🥺

[[package]]
name = "atomicwrites"
version = "1.4.1"
description = "Atomic file writes."
category = "dev"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"

[[package]]
name = "attrs"
version = "22.2.0"
description = "Classes Without Boilerplate"
category = "dev"
optional = false
python-versions = ">=3.6"

[package.extras]
cov = ["attrs[tests]", "coverage-enable-subprocess", "coverage[toml] (>=5.3)"]
dev = ["attrs[docs,tests]"]
docs = ["furo", "myst-parser", "sphinx", "sphinx-notfound-page", "sphinxcontrib-towncrier", "towncrier", "zope.interface"]
tests = ["attrs[tests-no-zope]", "zope.interface"]
tests-no-zope = ["cloudpickle", "hypothesis", "mypy (>=0.971,<0.990)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "pytest-xdist[psutil]"]
tests_no_zope = ["cloudpickle", "hypothesis", "mypy (>=0.971,<0.990)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "pytest-xdist[psutil]"]

[[package]]
name = "click"
version = "8.1.3"
description = "Composable command line interface toolkit"
category = "main"
optional = false
python-versions = ">=3.7"

Poetry Lock File

Luckily it's handled exclusively by Poetry
It keeps all versions and the hash of the version for extra resilience
Causes some headaches with git merge conflicts.

Conclusion

Poetry is great
We should be using it everywhere
It is not that hard to convert and adopt Poetry
It will save us in the long-run from having issues with package dependencies of dependencies

Poetry

By Josh Finnie

Poetry

Josh Finnie PRO

Full stack staff engineer. Lives in the DMV.

joshfinnie.com

Poetry

What is Poetry?

Famous Poets

What is Poetry?

What is Poetry?

Why Poetry?

pip install -r req.txt 😭

Example

pyproject.toml is great!

ADS already uses this!

ADS's pyproject.toml

Poetry plugs right in

Poetry in pyproject.toml

Poetry is nice

$ poetry --list

$ poetry --list (cont)

How Poetry

Install Poetry

Use Poetry

Use Poetry (Cont)

Use Poetry in Existing Proj

Adding Dependencies

Removing Dependencies

Poetry Lock File

Poetry lock file 🥺

Poetry Lock File

Conclusion

Poetry

More from Josh Finnie