Markus Kitsinger

github.com/SwooshyCueb

Software Developer, iRODS Consortium

Build and Packaging Update

Build and Packaging Update

July 5-8, 2022

iRODS User Group Meeting 2022

Leuven, Belgium

  • External dependencies
  • External dependency packaging
  • iRODS Buildsystem
    • Compiler and C++ Standard Library
    • Dependency management
  • iRODS packaging

Areas of Interest

  • Current state of affairs, rationale, and caveats
    • Externals
    • libc++
    • Packaging
  • The new approach
  • All the friends we'll meet along the way

Brief Overview

  • External dependencies packaged with fpm
  • iRODS built with CMake and packaged with CPack
  • Everything is built with clang and libc++ that we provide
  • Two flavors of packages supported: dpkg (deb) and rpm

The Current State of Affairs - Overview

What

Why

  • Distributions do not have all our dependencies in their package repositories
  • Distributions tend to have older versions of our dependencies

The Current State of Affairs - Externals

  • Externals are not well-integrated into system
  • Currently not set up to provide different sets of externals for different distros
  • Current iRODS buildsystem relies pretty heavily on how our externals are packaged

 

More on this later...

The Current State of Affairs - Externals - Caveats

What

  • iRODS and most of our externals are built with clang
  • All C++ built against libc++
  • Using clang and libc++ from our externals

Why

  • Newer clang and newer libc++ than is in distribution repositories
  • Much of our code is not gcc-friendly
  • At one point, clang/libc++ adopted new features faster than gcc/libstdc++

The Current State of Affairs - libc++

The Current State of Affairs - libc++ - Caveats

  • Mixing binaries built against libc++ and libstdc++ is problematic
  • Distro-provided packages generally use libstdc++
  • Increases the number of externals we must provide
  • Makes building against iRODS more complicated

What

  • iRODS built using CMake and packaged with CPack
    • Buildsystem does a lot of platform-specific heavy lifting
    • Most packaging defined in CMake
    • File/directory ownership handled programmatically with postinst scripts
    • Libraries in /usr/lib, regardless of what the distro expects
  • Externals packaged with fpm

Why

  • CPack and fpm are one-size-fits-all solutions, easier to wield than dpkg-buildpackage and rpmbuild
  • The approach at the time was lazy-but-sufficient

The Current State of Affairs - Packaging

The Current State of Affairs - Packaging - Caveats

  • Cannot provide debian or rpm source packages
  • Service account shenanigans
    • More on this later
  • Using system-provided dependencies in lieu of externals we provide is tricky
    • May require buildsystem changes
  • No package linting
  • No "start from zero" package builds (no pbuilder)

 

More on next slide...

"Lazy but Sufficient" is Neither

  • We need to provide debian and rpm source packages
  • Service account hot-potato means no systemd unit files
  • CMake has to know a lot about the target distros to produce usable packages
  • Adding support for another distro requires more work and a new release
    • Likewise for a new version of an already supported distro
  • We want to reduce the number of externals packages we provide
  • No automated symbol tracking

The Future of iRODS Build and Packaging: "Normal and Boring"

  • We will shift to using the standard tools (dpkg-buildpackage and rpmbuild) for packaging
    • git-buildpackage will be used to maintain debian packages, Salsa-style
      • Possibly rpm packages as well, still investigating
    • We will not provide an externals package if the distribution already provides a usable package
    • Debian and rpm source packages will be provided in our repositories
    • We will follow established patterns for setting up service accounts
    • We will install our libraries in the normal locations
    • We will provide default systemd unit(s)
  • We will build against libstdc++
  • We will decouple the iRODS buildsystem from externals packaging implementation details

 

Clearing Hurdles and Shaving Yaks

  • I am still familiarizing myself with rpmbuild and friends. Most of my packaging experience is with dpkg, PKGBUILD, and Wix.
  • New workflow and instrumentation for building packages.
    • Separate workflows for "from zero" builds and routine development builds.
  • Service account hot-potato is actually part of a larger issue that must be solved with care as part of this transition.
  • Distros without a new-enough libstdc++ will need a libstdc++ externals package.
  • We will have to write CMake find modules for non-CMake dependencies that do not already have them.
    • We may have to also write CMake find module wrappers to work around bugs and oversights in the CMake-provided find modules, such as FindODBC.cmake.
  • This transition cannot be easily broken up into stages.

Timeline (or lack thereof)

We don't know.

  • Yak shaving
  • Known unknowns

 

We may have a better idea of the time table at the next UGM.

Questions?

UGM 2022 - iRODS Build and Packaging Update

By iRODS Consortium

UGM 2022 - iRODS Build and Packaging Update

  • 726