Markus Kitsinger
Software Developer and Build Engineer
iRODS Consortium
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826819/hex_title_page_bg_2020_widescreen.jpg)
iRODS Build and Packaging:
2024 Update
May 28-31, 2024
iRODS User Group Meeting 2024
Amsterdam, Netherlands
- Versioning changes
- Recap of plans
- Current progress
- Externals
libstdc++
- New Features
- Version Freezes
- Other things that happened
- Notable yaks in need of shaving
- Other considerations
Overview
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
- Following the 4.3.x series, major releases of iRODS will be versioned 5.0, 6.0, 7.0, etc
- Minor releases will be versioned 5.1, 5.2, 5.3, etc
- Gnome-style development versioning - between releases, code in repository will have development version numbers
- x.9# for major releases/main branch (ex: 4.90 is development version of 5.0.0)
- x.y.9# for minor releases/stable branches (ex: 5.1.92 is development version of 5.2.0)
- Custom CMake package version file
- Takes into account development versions
- Takes into account the versioning change for 5.0
- More info in issue #7532
Versioning changes - iRODS 4.4 is now 5.0
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Recap of Plans
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
-
We will shift to using the standard tools (
dpkg-buildpackage
andrpmbuild
) for packaging-
git-buildpackage
will be used to maintain debian packages, Salsa-style- Possibly rpm packages as well, still investigating
- We will not provide an externals package if the distribution already provides a usable package
- Debian and rpm source packages will be provided in our repositories
- We will follow established patterns for setting up service accounts
- We will install our libraries in the normal locations
- We will provide default systemd unit(s)
-
- We will build against
libstdc++
- We will decouple the iRODS buildsystem from externals packaging implementation details
- We decided to transition to
libstdc++
before moving away fromfpm
/CPack.- This will allow us to reduce the number of externals packages we provide, which will facilitate the transition to standardized packaging for externals.
- This also meant putting a lot more effort into the current externals system than I really wanted to. More on this in a bit.
- iRODS 5.0 will be built against
libstdc++
. - Some externals are now used in CMake via
find_package
. -
mungefs
buildsystem completely decoupled from externals.
Current Progress
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
- In order to build iRODS against
libstdc++
, externals needs to build againstlibstdc++
as well. - iRODS 4.3.x is still built against
libc++
, so we have to support both until 4.3 is EOL. - Some packages now have two variants, one for
libstdc+++
and one forlibc++
.
Externals - libstdc++
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Externals - New and Removed Packages
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
- New package -
jwt-cpp
- Added for HTTP API provider client - Several packages removed
-
cpr
- no single version compatible with all versions of curl we must support -
elasticlient
- built on top ofcpr
-
jansson
- replaced by nlohmann-json (json
) -
pistache
- only used by one project, which has now been sunset -
libs3
- all relevant code has been merged into the s3 resource plugin -
aws-sdk-cpp
- originally added for s3 resource plugin, but was never used
-
- Exploring removing more
-
libarchive
- distro-provided packages should be sufficient once EL7 is dropped (#7286) -
redis
- distro-provided packages should be sufficient once EL7 is dropped (#7478) -
zeromq4-1
- distro-provided packages should be sufficient once EL7 is dropped (#7479) -
json
- distro-provided packages may be sufficient already (#7726)
-
Externals - New Features
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
- Each package now declares dependencies per-distribution per-version.
- New source patch system
- Initially added for Ubuntu 24.04 support, as
clang
required changes a little too complex for basic shell scripting to handle. - We now pull in some patches from distribution packages.
- Initially added for Ubuntu 24.04 support, as
- Package revisions are now properly supported, allowing for in-place upgrades.
Externals - Version Freezes
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
We want to transition to distro-provided packages where possible. In order to facilitate this, we have implemented a soft version freeze on most of our externals.
- Bumping the version of an externals package that has a distro-provided equivalent (or is likely to have one in the future) needs to be sufficiently justified.
- Externals packages unlikely to ever have a distro-provided equivalent (such as
jwt-cpp
) are free from this restriction. - This presents its own challenges we have had to overcome:
- Our
clang
externals package supports C++ coroutines, but we cannot use them due to an incompatibility withlibstdc++
. - Our
cmake
externals package does not support the newer versions of Python used on some distributions.
- Our
Other Things That Happened
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
- s3 resource plugin has absorbed
libs3
.- s3 resource plugin has been relicensed to LGPLv3+/GPLv2+.
- Python rule engine plugin build has been properly parallelized.
- Development environment
Dockerfile
s use newDockerfile
syntax. - Development environment core builder now uses
ccache
. - We now leverage CMake object library targets to improve our buildsystems in a number of ways.
Notable Yaks In Need of Shaving (that we know about)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
- Non-package installation -
make install
should be enough - File/directory ownership
- File/directory location
- Default configurations (or something along those lines)
- Unprivileged builds in CI and development environment
- CMake target names
- Removal of externals plumbing in CMake
Other Considerations
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
- How will this affect development workflows?
- How will we document this?
- How will CI need to change?
- When are cleanroom builds needed and how will we support them?
- How will we verify our dependency minimum versions?
- How often should we do this?
- Presently, a lot of functionality for standing up and cleaning up after iRODS is handled by Python scripts. How much of this should be migrated into iRODS proper, and how?
- How long should we maintain legacy CMake target aliases?
- How will the upgrade process be affected?
Thank you!
Questions?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
UGM 2024 - iRODS Build and Packaging: 2024 Update
By iRODS Consortium
UGM 2024 - iRODS Build and Packaging: 2024 Update
- 113