modern Python developMENT

⚔️ vs 🚀

Python Meetup Grenoble
2019-03-28

About me

Romain Clement

  • CTO @ Sylha
  • Freelance Software Engineer
  • Open-source software contributor 
contact@romain-clement.net
https://romain-clement.net
https://github.com/rclement

Summary

  • Dependencies management
    House keeping

  • Development environment
    Make yourself at home

  • Continuous integration
    Use a safety net, always

  • Versatile deployment
    Be a shape-shifter

Warning

This material is very opinionated!

All presented tools and methods are working for me, maybe it will for you, maybe it won't.

Tooling is no religion, just a matter of personal preference and experience.

Dependencies management

How can I make sure the packages my software requires are the ones IN use?

Dependencies management

⚔️ Ancient Times

  • virtualenv
  • pip
  • requirements.txt
  • requirements-dev.txt
  • ...
> venv myapp
> cd myapp
> source venv/bin/activate
> pip install -r requirements.txt
> pip install another-package
> pip freeze > requirements.txt
> pip uninstall previous-package
> pip freeze > requirements.txt
> pip list --outdated
> pip install another-package --upgrade
> pip freeze > requirements.txt

Dependencies management

⚔️ Ancient Times

  • Tedious (plumbing level)
  • Manual package pinning
  • How to update all packages consistently given a few constraints?
  • How to sync my venv?
  • Hell breaks lose when uninstalling packages!
  • Horror stories for cross-package sub-dependencies
  • Environment isolation
  • Quick-start

Good stuff

Not so good

Dependencies management

  • pipenv
  • Pipfile
  • Pipfile.lock
> pipenv install
> pipenv shell
> pipenv install another-package
> pipenv uninstall previous-package
> pipenv update --outdated
> pipenv update
> pipenv sync
> pipenv rm

🚀 Modern Times

Dependencies management

  • Dependencies & constraints specifications (versions, extras, package sources, PyPI source)
  • Handled by pipenv
  • Required Python version
  • Packages pinning
  • À la package.json, cargo.toml etc.

🚀 Modern Times

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
celery = {extras = ["redis"]}
flask = "*"
redis = "==2.10.6"

[dev-packages]
pytest = "*"

[requires]
python_version = "3.6"

Pipfile + Pipfile.lock

Dependencies management

  • Porcelain for venv + pip under the hood

  • Specify dependencies contraints with Pipfile

  • Deterministic environment for dev and production

  • Updating packages a breeze

🚀 Modern Times

  • Designed mostly for apps
  • Not a replacement of setup.py for libraries
  • Too opinionated for some people

Improvements

Limitations

Development environment

How can I develop Python software with confidence and respect 12-factors principles?

Development environment

  • App configuration

  • .env, .env.example
  • pipenv shell
  • python-dotenv
FLASK_APP=myapp.wsgi:app
FLASK_ENV=development
SECRET_KEY=supersecretkey
API_KEY=apisupersecretkey

Environment variables

import os


SECRET_KEY = os.environ.get('SECRET_KEY')
API_KEY = os.environ.get('API_KEY')

Development environment

  • DO NOT USE MAKEFILES!

  • pyinvoke
  • tasks.py

  • inv <task> <args>
from invoke import task


@task
def tests(ctx):
    ctx.run('py.test tests')


@task
def lint(ctx):
    ctx.run('flake8 myapp')


@task
def safety(ctx):
    ctx.run('safety check')


@task(tests, lint, safety)
def qa(ctx):
    pass

Tasks

Development environment

  • External services (databases, analytics, sentry)

  • Development and testing

  • docker-compose

version: '3'

services:
  postgres:
    image: 'postgres:10.2'
    environment:
      POSTGRES_USER: "${POSTGRES_USER}"
      POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
      POSTGRES_DB: "${POSTGRES_DB}"
    volumes:
     - './data/postgres:/var/lib/postgresql/data'
    ports:
      - "${POSTGRES_PORT_HOST}:${POSTGRES_PORT}"

  redis:
    image: 'redis:4.0.8'
    ports:
      - "${REDIS_PORT_HOST}:${REDIS_PORT}"

Docker

Development environment

  • python-dotenv
  • pyinvoke
  • tox
  • pytest
  • flake8
  • black
  • safety
  • liccheck
  • honcho
  • ...

Various Tools

CONTINUOUS INTEGRATION

I need a safety-net to catch me if I fall

CONTINUOUS INTEGRATION

Goals

  • Quality assurance automation
  • After each commit/push
  • Non-regression testing
  • Code coverage
  • Lint check
  • Vulnerabilities check
  • Dependencies licenses check
  • Automatic deployment (CD)
  • ...

CONTINUOUS INTEGRATION

CI as a Service

  • Travis-CI
  • Circle-CI
  • Azure Pipelines
  • GitLab CI
  • BitBucket Pipelines
  • Jenkins
  • ...

CONTINUOUS INTEGRATION

Configuration

  • Most of them: single YAML config file
  • Language
  • Environment
  • Dependencies
  • Stages (test, lint, vuln, deploy)
dist: xenial

language: python
python:
    - "3.7"

install:
  - pipenv install --dev

script:
  - pipenv run inv qa

Versatile Deployment

How do I make sure to be able to deploy my software with various environmentS?

Versatile Deployment

Structure

  • No "vendor lock-in" syndrome
  • Modularize properly the project
  • A proper entry-point (WSGI, ASGI, etc.)
  • Configurable with env vars + secrets
  • Deterministic dependencies
  • Buildable in single command
  • Runable in single command

Versatile Deployment

Docker (IaaS)

  • Dockerfile
  • docker-compose
  • Docker Swarm
  • Kubernetes
FROM python:3.7-alpine

RUN set -ex \
    && pip install --upgrade pip \
    && pip install pipenv

WORKDIR /app

COPY Pipfile Pipfile
COPY Pipfile.lock Pipfile.lock

RUN set -ex \
    && pipenv install --deploy --system \

COPY . /app

EXPOSE 5000

ENTRYPOINT ["honcho", "start"]
CMD ["web"]

Versatile Deployment

BuildpackS (PaaS)

  • Heroku / Dokku

  • Build a Docker image automatically

  • Define multiple types of tasks

  • Procfile
  • honcho
web: gunicorn --name=myapp --worker-class=gevent --error-logfile=- myapp.wsgi:app
worker: celery worker --app celery_worker:celery --config celery_config --loglevel=info
beat: celery beat --app celery_worker:celery --config celery_config --loglevel=info

Versatile Deployment

Serverless (FaaS)

  • Zeit Now, AWS Lambda, Azure functions

  • Stateless app automatic scaling

  • Short-lived app instances

  • Run a single entry-point handler function

  • Can be a WSGI / ASGI app

  • Not appropriate for everything

Demo

That's great and all, but can you show me a real-world example?

DeMO

Mailer

Dead-simple mailer micro-service for static websites

https://github.com/rclement/mailer

  • Python + Flask
  • Pipenv
  • 12-factor app with secrets
  • Travis-CI CI/CD
  • Docker image
  • Docker Swarm / Buildpack / Serverless

Questions?
Comments?
Tomatoes?

Thanks!

References

The 12-factor app: https://12factor.net
Pipenv: https://pipenv.readthedocs.io
Travis-CI: https://travis-ci.org
Docker: https://docs.docker.com
Kubernetes: https://kubernetes.io
Zeit Now: https://zeit.co/docs
Heroku: https://www.heroku.com
Dokku: http://dokku.viewdocs.io/dokku