Building reproducible Python 🐍🐍🐍applications for secured environments

Kushal Das

@kushaldas

CC BY-ND 4.0

PyCon US 2019

@kushaldas

$ whoami

@kushaldas

@kushaldas

@kushaldas

Git

.deb

.rpm

@kushaldas

@kushaldas

@kushaldas

@kushaldas

@kushaldas

Get Professional UX help

@kushaldas

https://qubes-os.org

@kushaldas

What could go wrong security wise?

@kushaldas

@kushaldas

@kushaldas

  • Source contains malware (or changed)
  • Binary replaced with malware
  • Mitm while downloading the source/binary
  • Storage/web server compromised

 

Threats

@kushaldas

Review your dependencies

@kushaldas

$ pipenv lock -r > requirements.txt

-i https://pypi.org/simple
alembic==1.0.2
arrow==0.12.1
certifi==2018.10.15
chardet==3.0.4
idna==2.7
mako==1.0.7
markupsafe==1.0
pathlib2==2.3.2
python-dateutil==2.7.5
...

 

@kushaldas

alembic==1.0.2 --hash=sha256:14024bd47f71d8b51920721dcd63248d07d370fbd0f6afa9bec67b9edaf71f36
arrow==0.12.1 --hash=sha256:5ef4a593615dc61ed85e62070b1bd27c71f7266233f0f9f385b651370e8c6760
certifi==2018.10.15 --hash=sha256:a5471c55b011bd45d6155f5c3629310c1d2f1e1a5a899b7e438a223343de583d
chardet==3.0.4 --hash=sha256:9f178988ca4c86e8a319b51aac1185b6fe5192328eb5a163c286f4bf50b7b3d8

But, we want to use our own wheels

@kushaldas

Makefile

+

a few Python and Bash scripts

@kushaldas

.PHONY: build-wheels
build-wheels: fetch-wheels
        ./scripts/verify-sha256sum-signature
        ./scripts/build-sync-wheels -p ${PKG_DIR}
        ./scripts/sync-sha256sums
        ./scripts/createdownloadurls.py > wheelsurls.txt

 

@kushaldas

Pipfile.lock

  • Has the source hashes
  • All recursive dependencies listed

@kushaldas

cmd = [
            "pip3",
            "download",
            "--no-binary",
            ":all:",
            "--require-hashes",
            "-d",
            tmpdir,
            "-r",
            newreq_path,
        ]
        subprocess.check_call(cmd)

 

@kushaldas

cmd = [
            "pip3",
            "wheel",
            "--no-binary",
            ":all:",
            "-f",
            tmpdir,
            "-w",
            tmpdir,
            "-r",
            newreq_path,
        ]
subprocess.check_call(cmd)

@kushaldas

alembic==1.0.2 --hash=sha256:14024bd47f71d8b51920721dcd63248d07d370fbd0f6afa9bec67b9edaf71f36
arrow==0.12.1 --hash=sha256:5ef4a593615dc61ed85e62070b1bd27c71f7266233f0f9f385b651370e8c6760
certifi==2018.10.15 --hash=sha256:a5471c55b011bd45d6155f5c3629310c1d2f1e1a5a899b7e438a223343de583d
chardet==3.0.4 --hash=sha256:9f178988ca4c86e8a319b51aac1185b6fe5192328eb5a163c286f4bf50b7b3d8

requirements.txt

@kushaldas

$ python3 setup.py sdist

@kushaldas

@kushaldas

diffoscope

@kushaldas

Get at least 2 humans verify the source updates of all the dependencies

@kushaldas

sha256sums.txt

sha256sums.txt.asc

14024bd47f71d8b51920721dcd63248d07d370fbd0f6afa9bec67b9edaf71f36  alembic-1.0.2-py2.py3-none-any.whl
04bcb970ca8659c3607ddd8ffd86cc9d6a99661c9bc590955e8813c66bfa582b  alembic-1.0.2.tar.gz
5ef4a593615dc61ed85e62070b1bd27c71f7266233f0f9f385b651370e8c6760  arrow-0.12.1-py2.py3-none-any.whl
a558d3b7b6ce7ffc74206a86c147052de23d3d4ef0e17c210dd478c53575c4cd  arrow-0.12.1.tar.gz
a5471c55b011bd45d6155f5c3629310c1d2f1e1a5a899b7e438a223343de583d  certifi-2018.10.15-py2.py3-none-any.whl
6d58c986d22b038c8c0df30d639f23a3e6d172a05c3583e766f4c0b785c0986a  certifi-2018.10.15.tar.gz

@kushaldas

@kushaldas

@kushaldas

<!DOCTYPE html>
<html>
  <head>
    <title>Simple index</title>
  </head>
  <body>
    <a href="/simple/alembic/">alembic</a>
    <a href="/simple/arrow/">arrow</a>
    <a href="/simple/atomicwrites/">atomicwrites</a>
    <a href="/simple/attrs/">attrs</a>
  </body>
</html>

@kushaldas

<!DOCTYPE html>
<html>
  <head>
    <title>Links for alembic</title>
  </head>
  <body>
    <h1>Links for alembic</h1>
    <!-- Keep adding all source and wheel urls below follwoing the format-->
    <!-- <a href="">FILENAME</a><br/> -->
    <a href="/localwheels/alembic-1.0.1.tar.gz">alembic-1.0.1.tar.gz</a><br/>
    <a href="/localwheels/alembic-1.0.1-py2.py3-none-any.whl">alembic-1.0.1-py2.py3-none-any.whl</a><br/>
    <a href="/localwheels/alembic-1.0.2.tar.gz">alembic-1.0.2.tar.gz</a><br/>
    <a href="/localwheels/alembic-1.0.2-py2.py3-none-any.whl">alembic-1.0.2-py2.py3-none-any.whl</a><br/>
  </body>
</html>

@kushaldas

@kushaldas

@kushaldas

@kushaldas

dh-virtualenv

https://dh-virtualenv.readthedocs.io

@kushaldas

#!/usr/bin/make -f

%:
	dh $@ --with python-virtualenv

override_dh_virtualenv:
	dh_virtualenv --python /usr/bin/python3.5 --setuptools -S --index-url https://dev-bin.ops.securedrop.org/simple

override_dh_strip_nondeterminism:
	find ./debian/ -type f -name '*.pyc' -delete
	find ./debian/ -type f -name 'pip-selfcheck.json' -delete
	find -type f -name RECORD -exec sed -i -e '/.*\.pyc.*/d' {} +
	dh_strip_nondeterminism $@

@kushaldas

https://reproducible-builds.org/

@kushaldas

@kushaldas

  • Source contains malware (or changed) sha256sums stored with gpg signature + human review of dependency diffs
  • Binary replaced with malware sha256sums stored with gpg signature, we build wheels from source ourselves
  • Mitm while downloading the source/binary https + sha256sums
  • Storage/web server compromised sha256sums stored with gpg signature + no key present on network connected boxes
  • Reproducible builds to verify the final debian packages

Threats Mitigation

@kushaldas

Links

https://securedrop.org

https://freedom.press

@kushaldas

Thank you