Alan King and Derek Dong

iRODS Consortium

November 12-17, 2023

Supercomputing 2023

Denver, CO

iRODS 4.2 Series

4.2.12 is the final release of the 4.2 series.

Limited to security fixes, bug fixes, and trivial enhancements.

https://irods.org/2023/05/irods-4-2-12-is-released

Contributors - 4.2.12

~/irods $ git shortlog --summary --numbered 4.2.11..4.2.12
    67  Alan King
    58  Kory Draughn
    13  Daniel Moore
     9  Justin James
     8  Markus Kitsinger (SwooshyCueb)
     6  Martin Jaime Flores Jr
     4  Felix A. Croes
     2  Alastair Smith
     1  Phillip Davis
     1  Terrell Russell

4.2.12 Core Server Improvements

• Microservices for read-only access to JSON objects
  • Useful in iRODS Rule Language (NREP) with JSON-based inputs/outputs
• Wider availability of admin keyword in various APIs and libraries
  • imeta
  • atomic ACLs/metadata endpoints
  • filesystem
  • msiDataObjChksum
• Improved user/group/password management
• Fixes and expansive tests for compound resource

Contributors - 4.3.1

~/irods $ git shortlog --summary --numbered 4.3.0..4.3.1
   204  Kory Draughn
   101  Alan King
    24  Markus Kitsinger (SwooshyCueb)
    15  Nishant Dash
    14  Martin Jaime Flores Jr
    12  Justin James
    10  Daniel Moore
     7  Violet White
     4  Felix A. Croes
     3  Terrell Russell
     2  Derek Dong
     2  Phillip Davis
     1  Awab Masroor
     1  Peter Verraedt
     1  June Releford
     1  Leonardo Lenoci

4.3.1 User Experience Updates

• Removed setup for rsyslog/logrotate (syslog implementation assumptions)
• Replaced log_facility with server_zone in log message output
• Deprecated SimpleQuery
• Exposed client connection information to acPreConnect()
• ichmod honors the permission model
• unixfilesystem resource plugin supports detached mode
• Additional info added to izonereport; structure flattened for clarity
• New configuration options: TCP keepalive, authentication
• Newly packaged for Ubuntu 22, Debian 12, and Enterprise Linux 9

4.3.1 Core Server Enhancements

• Approaching GCC compatibility
• Added support for Address Sanitizer
• New API plugin: rc_switch_user
• iRODS Project Templates for C++
• Improved documentation

** Bold items discussed in this talk **​​

• Library feature tests
• New API plugin: rc_check_auth_credentials
• New zone administration library for C++
• New ticket administration library for C++
• New C++ library: process_stash

https://irods.org/2023/10/irods-4-3-1-is-released

Build and Packaging

We continue to move towards a more Normal and Boring approach to build and packaging.

• clang-tidy brings us closer to building against libstdc++
• Work has begun on unprivileged build and packaging in development environment containers

As work towards this goal continues, there will be significant changes to the pre-build configuration process. We plan to use CMake presets to smooth the transition.

We still do not have a time-table. See the talk at UGM 2022 for more details on our plans.

Address Sanitizer (ASan)

A very fast memory error detector for C/C++.

It detects several different issues such as memory leaks, use-after-free bugs, heap buffer overflows, etc.

Used to track down several memory leaks in iRODS 4.3.0.

Enabled via CMake by setting IRODS_ENABLE_ADDRESS_SANITIZER to YES.

For example:

    user@sc2023:~ $ cmake ... -DIRODS_ENABLE_ADDRESS_SANITIZER=YES ...

New API Plugin - rc_switch_user

Allows the user associated with a connection to be switched to a different user.

Designed for client applications which

• act as servers (e.g. NFSRODS) and
• require a proxied connection

Benefits

• Avoids TCP connection setup and tear down
• Allows a single connection to be reused for multiple users
• Gets us closer to true connection pooling

New API Plugin - rc_switch_user (cont.)

Performance Testing Details

Setup

• Two custom client applications
• App A connects to a server N times as the same user
• App B makes one connection and calls rc_switch_user N times

Test results show a 98% performance improvement.

iRODS Project Templates for C++

Using the GitHub template repository feature, the iRODS Consortium now offers template repositories which allow C++ developers to jump directly into writing code for iRODS.

The Consortium supports five template repositories today.

• https://github.com/irods/irods_project_template_cpp_client
• https://github.com/irods/irods_project_template_cpp_api_plugin
• https://github.com/irods/irods_project_template_cpp_microservice_plugin
• https://github.com/irods/irods_project_template_cpp_resource_plugin
• https://github.com/irods/irods_project_template_cpp_rule_engine_plugin

Improved Documentation - Policy Cookbook

An online resource dedicated to providing best practices and the latest techniques to various policy-based situations encountered in the iRODS ecosystem.

The cookbook covers topics such as ...

• Synchronizing Delay Rules using Metadata
• Naming Schemes and Conventions
• Sharing data across PEPs
• Simulating User Quotas
• Implementing maintainable Policy through reusable rules

If you have suggestions on how to improve the cookbook, please reach out.

https://docs.irods.org/4.3.1/administrators/policy_cookbook

Improved Documentation - Data Objects

Information about data objects has been expanded.

Documentation for 4.3.1 includes details about ...

• The meaning of each replica status (intermediate, write-locked, etc.)
• Logical Locking
• High-Level Operations (put, get, copy, replicate, etc.)
• R_DATA_MAIN - The database table which holds all replica information

We'll continue to expand on these topics as improvements to the server are made.

https://docs.irods.org/4.3.1/system_overview/data_objects

Improved Documentation - Protocol Cookbook

Intern project documenting the iRODS protocol by demonstrating a basic client implementation of the iRODS control flow.

Meant to serve as a model for implementing new client libraries in various languages.

Implemented as a Jupyter Notebook.

More information:

https://www.youtube.com/watch?v=JL1snQbdWWQ

https://github.com/irods/iRODS-Protocol-Cookbook

UnixFileSystem Resource - Detached Mode

Allow multiple servers in an iRODS Zone to service requests made to a single UnixFileSystem resource. Only requires a mountpoint to a common backend filesystem accessible by all participating servers.

Useful for parallel and distributed filesystems.

Configure via context string:

• Add "host_mode=detached" (any other value means attached)
• Add comma-delimited list of hosts to host_list in context string (optional - all servers will service requests if host_list is excluded)

Audit AMQP Rule Engine Plugin

• Modernization
  • Refactored to use nlohmann-json instead of jansson
  • Refactored to use qpid-proton's C++ API
  • Migrated to new logging framework
  • Miscellaneous other modernization
• Housekeeping
  • Repository reorganized and code reformatted
  • RPM package installation less fussy
  • Removed unused amqp_options configuration setting
  • Miscellaneous other housekeeping
• Removed JSON wrapper tokens
• Fixed JSON types for some fields
• More AMQP message metadata set
• Better handling of default configuration

Audit AMQP Rule Engine Plugin - ELK Stack

• Modernization
  • New Dockerfile syntax
  • Updated entire software stack
    • Container base image
    • Elasticsearch, Kibana, RabbitMQ
    • Temurin JDK
• Housekeeping
  • Reduced number and size of intermediate container images
  • Excluded more unneeded files from container image
• Updated for use with new version of the rule engine plugin
  • Workarounds for use with older/current versions of the plugin are togglable
• Replaced logstash with a Python daemon using qpid-proton's Python API
• Moved as much setup as possible to container build-time
• Added argument for specifying Java heap size

iRODS Clients

Protocols

• HTTP API
• S3 API
• NFSRODS
• irodsfs (FUSE)
• SFTPGo
• k8s CSI Driver

https://irods.org/clients

Libraries

• irods-dev (C/C++)
• python-irodsclient (Python)
• go-irodsclient (Go)
• Jargon (Java)
• rirods (R)

CLIs

• iCommands
• iRODS CLI
• gocommands

GUIs

• Metalnx
• ZMT
• iBridges

iRODS CLI

Restart of a previous effort to make a modernized iRODS CLI:

• A brand new client, not just an icommands rewrite
  • The aim is to write everything with idiomatic C++
• Single executable instead of 50+ icommands

  • $ irods put testFile
    

• Optionally modular, thanks to CMake magic (and a hint of preprocessor)
  • Reads .so plugins as sub-commands via boost:dll, or
  • Sub-commands compiled into main executable
• Aims to make use of the newest libraries
  • irods::filesystem and friends for iRODS things
    • Avoiding icommand-specific libraries, too
  • A lot of boost, everywhere
• Implementations for several of the simpler commands are in a working state
  • ls, mv, put, get, etc.
• Still a WIP, first goal is to reach feature parity with icommands

https://github.com/irods/irods_client_cli

Big Picture

Core

• 4.3.x/4.4.x/5.0 - Satisfy Roadmap (Cloud-friendliness, Replace PackStruct, etc.)

https://irods.org/roadmap

Continue building out policy components (Capabilities).

We want installation and management of iRODS to become about policy design, composition, and configuration.

Please share your ...

• Use cases

• Pain points

• Hopes and dreams

Open Source Community Engagement

Get Involved

• Working Groups

• GitHub Issues

• Pull Requests

• Chat List

• Consortium Membership

Tell Others

• Publish, Cite, Advocate, Refer