July 5-8, 2022
iRODS User Group Meeting 2022
Leuven, Belgium
Markus Kitsinger
github.com/SwooshyCueb
Software Engineer, iRODS Consortium
iRODS Build and Packaging Update
- External dependencies
- External dependency packaging
- iRODS Buildsystem
- Compiler and C++ Standard Library
- Dependency management
- iRODS packaging
Areas of Interest
- Current state of affairs, rationale, and caveats
- Externals
libc++
- Packaging
- The new approach
- All the friends we'll meet along the way
Brief Overview
- External dependencies packaged with
fpm
- iRODS built with CMake and packaged with CPack
- Everything is built with
clang
andlibc++
that we provide - Two flavors of packages supported:
dpkg
(deb
) andrpm
The Current State of Affairs - Overview
What
- A set of separately packaged dependencies
- Not our code
- Live in
/opt/irods-externals
(by default) - https://github.com/irods/externals
Why
- Distributions do not have all our dependencies in their package repositories
- Distributions tend to have older versions of our dependencies
The Current State of Affairs - Externals
- Externals are not well-integrated into system
- Currently not set up to provide different sets of externals for different distros
- Current iRODS buildsystem relies pretty heavily on how our externals are packaged
More on this later...
The Current State of Affairs - Externals - Caveats
What
- iRODS and most of our externals are built with
clang
- All C++ built against
libc++
- Using
clang
andlibc++
from our externals
Why
- Newer
clang
and newerlibc++
than is in distribution repositories - Much of our code is not
gcc
-friendly - At one point,
clang
/libc++
adopted new features faster thangcc
/libstdc++
The Current State of Affairs - libc++
The Current State of Affairs - libc++
- Caveats
- Mixing binaries built against
libc++
andlibstdc++
is problematic - Distro-provided packages generally use
libstdc++
- Increases the number of externals we must provide
- Makes building against iRODS more complicated
What
- iRODS built using CMake and packaged with CPack
- Buildsystem does a lot of platform-specific heavy lifting
- Most packaging defined in CMake
- File/directory ownership handled programmatically with postinst scripts
- Libraries in
/usr/lib
, regardless of what the distro expects
- Externals packaged with
fpm
Why
- CPack and
fpm
are one-size-fits-all solutions, easier to wield thandpkg-buildpackage
andrpmbuild
- The approach at the time was lazy-but-sufficient
The Current State of Affairs - Packaging
The Current State of Affairs - Packaging - Caveats
- Cannot provide debian or rpm source packages
- Service account shenanigans
- Using system-provided dependencies in lieu of externals we provide is tricky
- May require buildsystem changes
- No package linting
- No "start from zero" package builds (no
pbuilder
)
More on next slide...
"Lazy but Sufficient" is Neither
- We need to provide debian and rpm source packages
- Service account hot-potato means no
systemd
unit files - CMake has to know a lot about the target distros to produce usable packages
- All dependencies must be specified manually (no
dpkg-shlibdeps
)
- All dependencies must be specified manually (no
- Adding support for another distro requires more work and a new release
- Likewise for a new version of an already supported distro
- We want to reduce the number of externals packages we provide
- No automated symbol tracking
The Future of iRODS Build and Packaging: "Normal and Boring"
- We will shift to using the standard tools (
dpkg-buildpackage
andrpmbuild
) for packaging-
git-buildpackage
will be used to maintain debian packages, Salsa-style- Possibly rpm packages as well, still investigating
- We will not provide an externals package if the distribution already provides a usable package
- Debian and rpm source packages will be provided in our repositories
- We will follow established patterns for setting up service accounts
- We will install our libraries in the normal locations
- We will provide default systemd unit(s)
-
- We will build against
libstdc++
- We will decouple the iRODS buildsystem from externals packaging implementation details
Clearing Hurdles and Shaving Yaks
- I am still familiarizing myself with
rpmbuild
and friends. Most of my packaging experience is withdpkg
,PKGBUILD
, and Wix. - New workflow and instrumentation for building packages.
- Separate workflows for "from zero" builds and routine development builds.
- Service account hot-potato is actually part of a larger issue that must be solved with care as part of this transition.
- Distros without a new-enough
libstdc++
will need alibstdc++
externals package. - We will have to write CMake find modules for non-CMake dependencies that do not already have them.
- We may have to also write CMake find module wrappers to work around bugs and oversights in the CMake-provided find modules, such as
FindODBC.cmake
.
- We may have to also write CMake find module wrappers to work around bugs and oversights in the CMake-provided find modules, such as
- This transition cannot be easily broken up into stages.
Timeline (or lack thereof)
We don't know.
- Yak shaving
- Known unknowns
We may have a better idea of the time table at the next UGM.
Questions?
UGM 2022 - iRODS Build and Packaging Update
By Markus Kitsinger
UGM 2022 - iRODS Build and Packaging Update
- 479