Python - (hopefully) good standards

Adam Glos

Why I think this is important

  • keeping high code quality, simplifies simpler to understand in the future (also by you)
  • It saves your time (you don't need to check if your changes broke something manually)
  • Some of the tools presented requires only a little work only at the very beginning
  •  

pip

  • it is good (especially for science) to fix the version of the packages to improve reproducibility (gives a chance to blame a package if something goes wrong
  • pip is responsible for installing python packages. It can install package of appropriate version
  • pip can create requirements.txt file which is a description of all packages used

Careful - OS dependent

conda

  • pip is dealing with the versions of python packages. But what will take care of python, pip, other things? - conda
  • You can create a environment which you can install on other systems

OS dependent, because pip is system dependent

conda + code

Code formatters

  • isort - sorting imports
  • black - PEP8
  • autopep8 - PEP8 (style guide, like how the code should look like, like space, lowercase-uppercase, etc)
  • flake8 - PEP8
  • mypy - static typing
  • spelling - word checking (in comments and docs)
  • other (pylint, yapf, ...)

code formatters may conflict - it is important to choose a bunch of code formatters which can "cooperate"

Here: isort + black + flake8 + mypy + spelling

isort

isort                  # for sorting import s
isort --check          # for checking if files are sorted
isort --profile black  # for consistency with black

black

black                  # for formatting
black --check          # for checking if files are formatted

flake8

flake8 # list all of issues

flake8 only complains - it does not change the code! but looks for fancier issues than black

It's better to first fix with isort and black

mypy

mypy only complains - it does not change the code! but looks for fancier issues than black

Spelling

python -m spelling

One can add file wordlist.txt which will have correct words unknown to spelling

https://github.com/resplendent-dev/spelling

Which should we choose?

For scientific project:

  • isort and black are must-have
  • flake8 is nice addend
  • mypy is overkill

For python library project:

  • isort, black, flake8, mypy are must-have
  • spelling is a nice addend

What if I'm lazy?

You're in the good neighborhood

configuration files

Flake8 - .flake8

Usually we will need to pass some parameters to formatters - instead one can create configuration files

black and sort - pyproject.toml

  • It is important to at least unify line-length (at most 100, 80 is also good choice)
  • these files should be pushed to git server
  • configuration is loaded automatically

Source: https://github.com/euro-hpc-pl/omniqubo (may change over time)

Visual code config.cont.

Text

Hide following files from file manager  (dangerous)

Text

run black and isort on saving

pre-commit

pre-commit is a tool for verifying the code right before commit

pip install pre-commit

1.

2.  Create configuration file .pre-commit-config.yaml

Do on push and commit

isort --check + config file

black --check

remove if not used

3. Install required formatters

pre-commit install

4. run on all files (optional)

pre-commit run --all-files

pre-commit

Cautious - pre-commit DOES NOT use the version installed by you through pip. make sure you have the same version, for example:

 

pre-commit - example

Hint: it seems pre-commit can be configured with vsode, but currently it conflicts with conda

pre-commit - example

Line 3 needs to be removed manually (and isort, black should be run again)

isort

black

commit with pre-commit

manual fixes

pytest

Unit test checks a single functionality. In python they should be located in a separate test directory

pytest is a software for automatically running all unit tests

Tests are done through assert if no error is thrown by a function, then the code is "correct"

Note that class and methods begins with test

pytest cont.

  • unit tests should be small, fast and test single functionality
  • they're extremely important for code which are
    • large, or expected to be large
    • maintained for a long time
  • in the first place it helps you verify whether you broke something when changing the code

Hint: "pytest -x" run until the first error

pytest cont.

  • unit tests should be small and test single functionality
  • they're extremely important for code which are
    • large, or expected to be large
    • maintained for a long time
  • in the first place it helps you verify whether you broke something when changing the code

Hint: "pytest -x" run until the first error

Visual code config. (pytest)

Text

Remote actions

  • Everything presented so far was done locally on the computer
  • However, we can use external services to do this for us (important for packages)
  • key examples: GitHub actions and Codecov

GitHub actions

Text

test and coverage checking

isort/black/flake8/mypy

GitHub actions - web interface

codecov.io - how many lines are tested

Data from GitHub actions

Conclusions

Small projects (<3 small, files mostly self-checking)

medium projects (long-term or big code)

python project

  • conda + pip
  • pre-commit (isort, black, flake8?, mypy?)
  • pytest?
  • conda + pip
  • pre-commit (isort, black, flake8, mypy?)
  • pytest
  • GitHub Actions? + codecov.io?
  • conda + pip
  • pre-commit (isort, black, flake8, mypy)
  • pytest
  • GitHub Actions + codecov.io

deck

By Adam Glos

Made with Slides.com

deck

  • 230

More from Adam Glos