Let's shadow a Python release: insights from packaging, distribution, and supply-chain security

Agriya Khetarpal

Agenda

1. The release process for the Python programming language
2. The software supply chain in the Python ecosystem

CPython 🐍

• The most common (and the reference) implementation of Python
• Most likely the one you use on a daily basis
• Other implementations: PyPy, RustPython, Jython, GraalPy, IronPython, Brython, MicroPython, Pyodide, etc.

Python 3.13 🪩

• Will be released in October, 2024

• Shining new features

  • New, updated REPL

  • A new JIT compiler based on LLVM (experimental)

  • Free-threaded (no-GIL) builds

How does the Python release process work?

• Alphas, betas, and release candidates
• Maintenance and security updates
• Coordinated through Python Enhancement Proposals (PEPs) by core developers
  • PEP 719: Python 3.13
  • PEP 745: Python 3.14
• Yes, Python 3.14 is already in the feature development stage, while Python 3.13 is getting ready for releases!

Support cycles and backports

• 1.5 years of backports and 3.5 years of security updates
• This will be 2 + 3 for Python 3.13 and later
• End-of-life: after five years of the first release

Stakeholders involved

• Python core developers
• Release Manager
• The Python Steering Council
• The Python Software Foundation
• Community contributors, (potentially) like you

Visualising the CPython release process

Seth Larson (Security Developer-in-Residence, PSF)

How releases are made

• https://github.com/python/release-tools repository
• Important to note: everything runs in CI (GitHub Actions and Azure Pipelines)!
• Several scripts and YAML workflows to facilitate every aspect of the release: compilation, signage, binary uploads, documentation, certificates, and more

How do you usually install Python?

Windows

• Package managers (Chocolatey, NuGet, Scoop)
• Microsoft Store
• Official https://python.org/download/ installation wizards offered in .msi or .exe format

macOS

• Package managers (Homebrew, MacPorts, Fink)
• Official .pkg or .dmg installers

How do you usually install Python?

GNU/Linux

• Package managers (apt, yum, dpkg, dnf, Linuxbrew, Spack)
• Official source tarballs

How do you usually install Python?

Downstream packaging and distribution

• OS-specific package maintainers (Red Hat, Gentoo, etc.)
• Maintainers of Docker images
• Cloud provider distributions (AWS, Azure, GCP)
• Embedded systems and IoT devices

Build provenance

Provenance

Let's compare two photos. Are they the same photo?

Chris Curry on Unsplash

They are not :P

How about these two?

Francesco Ungaro on Unsplash

They are the same, but don't be too quick to judge!

Images contain metadata

FileSize: 3.5 MiB
FileModifyDate: 2024-07-19T18:50:31.000+00:00
FileAccessDate: 2024-07-19T18:50:31.000+00:00
FileInodeChangeDate: 2024-07-19T18:50:31.000+00:00
FileType: JPEG
FileTypeExtension: jpg
MIMEType: image/jpeg
JFIFVersion: 1.02
ResolutionUnit: inches
XResolution: 72
YResolution: 72
ProfileCMMType: Linotronic
ProfileVersion: 2.1.0
ProfileClass: Display Device Profile
ColorSpaceData: RGB
ProfileConnectionSpace: XYZ
ProfileDateTime: 1998-02-09T06:49:00.000+00:00
ProfileFileSignature: acsp
PrimaryPlatform: Microsoft Corporation
CMMFlags: Not Embedded, Independent
DeviceManufacturer: Hewlett-Packard
DeviceModel: sRGB
DeviceAttributes: Reflective, Glossy, Positive, Color
RenderingIntent: Perceptual
ConnectionSpaceIlluminant: 0.9642 1 0.82491
ProfileCreator: Hewlett-Packard
ProfileID: 0
ProfileCopyright: Copyright (c) 1998 Hewlett-Packard Company
ProfileDescription: sRGB IEC61966-2.1
MediaWhitePoint: 0.95045 1 1.08905
MediaBlackPoint: 0 0 0
 

RedMatrixColumn: 0.43607 0.22249 0.01392
GreenMatrixColumn: 0.38515 0.71687 0.09708
BlueMatrixColumn: 0.14307 0.06061 0.7141
DeviceMfgDesc: IEC http://www.iec.ch
DeviceModelDesc: IEC 61966-2.1 Default RGB colour space - sRGB
ViewingCondDesc: Reference Viewing Condition in IEC61966-2.1
ViewingCondIlluminant: 19.6445 20.3718 16.8089
ViewingCondSurround: 3.92889 4.07439 3.36179
ViewingCondIlluminantType: D50
Luminance: 76.03647 80 87.12462
MeasurementObserver: CIE 1931
MeasurementBacking: 0 0 0
MeasurementGeometry: Unknown
MeasurementFlare: 0.999%
MeasurementIlluminant: D65
Technology: Cathode Ray Tube Display
RedTRC: (Binary data 2060 bytes, use -b option to extract)
GreenTRC: (Binary data 2060 bytes, use -b option to extract)
BlueTRC: (Binary data 2060 bytes, use -b option to extract)
ImageWidth: 4160
ImageHeight: 6240
EncodingProcess: Progressive DCT, Huffman coding
BitsPerSample: 8
ColorComponents: 3
YCbCrSubSampling: YCbCr4:2:0 (2 2)
ImageSize: 4160x6240
Megapixels: 26

Provenance is

security

metadata

reproducibility

verifiability

evidence

identifiability

confidence

origin

trust

SLSA: Supply-chain Levels for Software Artifacts

The software supply chain

• Source code
• Build systems
• Package registries
• Distribution channels
• End users

Exhibit: dependency confusion

• PyTorch (nightly release) was compromised between December 26–31, 2022

• Malicious miscreants added torchtriton as a dependency to PyPI, from where pip downloads packages at a priority in comparison to other indices

• Downloaded ~2717 times in total, with 2500 on 26 December 2022 alone

Similar attacks in the wild

Typosquatting

This isn't endemic to the Python ecosystem alone

SLSA provenance

Level 1: Provides supply chain visibility

• Automated version control.
• Automated build process.
• Generate Provenance.
• Provenance contains information

Level 4: Assurance of build integrity + dependency management

• 2 person review of all changes.
• Hermetic/Reproducible builds.
• Hardened build service.
• Signed & Non-falsifiable provenance

Level 3: Harden build infrastructure, integrate trust

• Source/build platform meet standards.
• Auditability of source/build.
• Guaranteed integrity of provenance

Level 2: Protect against tampering, provide integrity of builds

• Version control.
• Hosted Build.
• Signed provenance

SBOM: Software Bill-of-Materials 🧾

• Comprehensive inventory of software components
• Includes dependencies, versions, and licensing info
• Crucial for vulnerability management and compliance

The Python SBOM 📜

• Included since CPython 3.12.2 and later releases
• JSON file containing all SHA-256 checksums for all files
• Names and versions of all software components
• Dependency relationships between software components
• Software identifiers (like CPE and Package URLs)
• Download URLs for source code with checksums

The Python SBOM 📜

• Since OpenSSL is built differently across Windows and Linux/macOS, the Python Windows installers need different SBOMs
• i.e., different SBOMs correspond to different sources

How are the Scientific Python and the Python packaging ecosystems faring?

• Sigstore: provides verification and signing binaries
• Coupled with GitHub: Artifact Attestations
• Scientific Python Ecosystem Coordination (SPEC)-8: "Securing the Release Process" is underway

What you can do ✅

"The locus of control" maxim

Internal

• Learn to write secure code (don't store anything in plaintext, inculcate an adversarial mindset, and more)

• Vet your dependencies - Package your code properly - Check for wheels (and not sdists) to help mitigate(!) RCEs

• Sign your binaries and releases
• In some cases, even limit such processes to dedicated release managers

What you can do ✅

"The locus of control" maxim

External

• Always build your binaries on ephemeral systems (CI providers) – never do so on your own system

• Aim for higher levels of SLSA provenance

• Use tooling that your code hosting solution provides (GitHub/GitLab/BitBucket/etc.) for provenance

• Try to score on the OpenSSF scorecards and follow up on the OpenSSF's best practices

About me 😁

Agriya Khetarpal

• Software engineer at Quansight
• Privileged to contribute to the open source
  Scientific Python and Pyodide ecosystems
• Interested in
  • Python packaging 📦🐍
  • Scientific computing ➗🧪
  • Documentation 📝🌉
  • ...and more 👾

Thank you for your time!

Please feel free to say hello!

Need these slides?

in/agriyakhetarpal

agriyakhetarpal

agriyakhetarpal

agriyakhetarpal [at] outlook [dot] com

Content licensed under CC-by-SA Attribution-ShareAlike Version 4.0 International License

Further readings