Kory Draughn

Chief Technologist

iRODS Consortium

May 28-31, 2024

iRODS User Group Meeting 2024

Amsterdam, Netherlands

Contributors

$ git shortlog --summary --numbered 4.3.0..4.3.2
   228  Kory Draughn
   148  Alan King
    58  Markus Kitsinger (SwooshyCueb)
    18  Martin Jaime Flores Jr
    15  Justin James
    15  Nishant Dash
    11  Derek D
    10  d-w-moore

 
     7  Terrell Russell
     7  Violet White
     4  Felix A. Croes
     2  Phillip Davis
     1  Awab
     1  Peter Verraedt
     1  june-releford
     1  lenocil

4.3.1 Release - Updates since UGM 2023 (not exhaustive)

• New iadmin subcommands
  • get/set_grid_configuration
    • ​Manage zone-wide settings stored in R_GRID_CONFIGURATION
• New APIs
  • rc_get_library_features
  • rc_check_auth_credentials
  • rc_get_resource_info_for_operation
• Includes options for controlling TCP keepalive
• Removed size limit on PAM password
• C++ User Administration library supports changing passwords just like iadmin
• C++ Connection Pool supports automatic refreshing of connections
• Added modify_ts_millis column to R_RESC_MAIN
  • ​Known in GenQuery as RESC_MODIFY_TIME_MILLIS

https://irods.org/2023/10/irods-4-3-1-is-released/

4.3.2 Release

iRODS 4.3.2 is available!

The goals remain the same ...

• Clean implementation
• Clear semantics
• No magic
• Improved tooling
• Focus on data management

iRODS 4.3.2 is smaller than previous releases, but offers improved stability with some nice additions.

https://irods.org/2024/05/irods-4-3-2-is-released/

4.3.2 User Experience Updates

• Improved multiple icommands
  • irsync and iput correctly ignore symlinks
  • ibun now uses the pax format when handling archive files
  • iadmin rmuser/rmgroup are no longer interchangeable
  • Help text is clearer and more detailed
• Setup script correctly handles configuration of local storage for iRODS Consumer servers

4.3.2 Core Server Updates

• GenQuery2
• Replica Truncate API
• Support for MariaDB
• Serialization of more iRODS data types
• Improved server stability

** Bold items discussed in this talk **​​

https://github.com/irods/irods/milestone/40?closed=1

• Improved documentation
  • How to report security vulnerabilities
  • Updated dynamic PEP listing
• Replication
  • Good replicas overwriting good replicas is now a no-op rather than an error

GenQuery2

GenQuery2 is now shipped with the server.

It is still considered experimental, but use is encouraged.

Its implementation can change as a result of community feedback.

Not used by the server, yet.

GenQuery2 will be available in the next release of the Python iRODS Client and Jargon.

• https://github.com/irods/python-irodsclient/pull/555
• https://github.com/DICE-UNC/jargon/pull/443

https://docs.irods.org/4.3.2/doxygen/genquery2_8h.html#a2b7171a0bea5f65ab4ca7c67fc87b3fa

New API - rc_replica_truncate

Allows applications to adjust the size of a single replica.

Motivated by NFSRODS.

• https://github.com/irods/irods_client_nfsrods/issues/178

Additional Information

• Modeled after POSIX truncate(2)
• Includes a microservice for policy
• Replaces rcDataObjTruncate (deprecated)

There are plans to add support for an ftruncate-style API as well.

• https://github.com/irods/irods/issues/7559

https://docs.irods.org/4.3.2/doxygen/replica__truncate_8h.html#a982a350e8933d32b4a81ba3053269a7f

Serialization of more iRODS data types

iRODS 4.3.2 now serializes the following data types for the rule engine plugin framework.

• StructFileExtAndRegInp (ibun)
• ExecMyRuleInp (irule)
• MsParamArray (irule)

Policy implementers can now inspect the contents of requests dealing with archive files and rule execution, and react accordingly.

iRODS 5.0 - Primary Goals

Modernize the startup/shutdown process for the iRODS server.

Things we are going to address ...

• Startup/Shutdown sequence is more deterministic
  • No mixing of fork[-exec] and multi-threading logic
  • No dependency on external Python code
• Server configuration is only loaded on startup
  • Supports live reloads via SIGHUP signal
  • Agents no longer reload the configuration
• Non-package installation is supported by design

We are considering ...

• Decoupling of hostnames from servers
  • Possibly the removal of hostnames altogether
    • May not be possible due to server-to-server redirects

iRODS 5.0 - Early Design and Implementation

Experimentation is now in progress. It's too early to assert anything about the design/implementation. Initial design is heavily influenced by the iRODS HTTP API.

At the moment, the plan is to honor the following ...

• Clear separation of concerns between components
• Plugins handle their own shared memory
• Multi-threaded code is constrained to leaf processes
• Runs as a normal application
  • Accepts a path to directory containing configuration files
  • Can be launched as a daemon process
• Control plane runs as a separate process
• Meets all requirements for working with systemd

Core Development Team Talks

• Not in This Talk / Separate Talks

  • Phil Owen

    • iRODS Build and Test v9: Automation via GitHub and Kubernetes

  • Markus Kitsinger

    • iRODS Build and Packaging: 2024 Update

  • Kory Draughn and Martin Flores

    • iRODS HTTP API v0.3.0 with OpenID Connect

  • Justin James

    • iRODS S3 API v0.2.0 with Multipart

  • Terrell Russell

    • DAViDD: Initial data management solution for UNC's READDI AViDD Center

    • iRODS Metadata Templates Working Group: Building Blocks and Lessions Learned

• Included in This Talk

  • Kory Draughn

    • Indexing Capability Plugin

    • Python Rule Engine Plugin

  • Derek Dong

    • Metadata Guard Rule Engine Plugin

  • Justin James

    • S3 Resource Plugin

    • Globus Connector

  • Daniel Moore

    • Python iRODS Client

  • Kory Draughn

    • NFSRODS

Indexing Capability Plugin

• Requires Elasticsearch 7 or later
• Uses Boost.Beast library for communication with Elasticsearch
  • elasticlient and CPR replaced by Boost.Beast
  • No change in behavior
• Removed document type rule engine plugin
  • No longer serves a purpose due to changes in Elasticsearch
    • Removal of mapping types
• Supports TLS communication with Elasticsearch
• Supports Basic Authentication
• Modernized CMake

Python Rule Engine Plugin

irods_rule_vars now available in more contexts

• Accessible in imported python modules
• irods_rule_vars formerly known as global_vars
• Deprecated global_vars

Improved documentation

• Covers how to pass values back to the caller, across rule engine plugin boundaries
• Covers what happens to the output of print()

Metadata Guard Rule Engine Plugin

Resolved some holes in the implementation.

• Handles atomic metadata operations API PEPs
• Disallows renaming non-protected AVU to existing protected AVU
• Disallows use of deprecated imeta rmw
  • Wildcard operations can bypass plugin logic (e.g. "irod%")
  • Wildcard operations make it difficult to cover all metadata cases

S3 Resource Plugin and Globus Connector

Python iRODS Client - from 1.1.9 to 2.0.1

$ git shortlog --summary --numbered v1.1.9..v2.0.1
    25  d-w-moore
     7  Terrell Russell
     1  Raoul Schram

Python iRODS Client - from 1.1.9 to 2.0.1 (cont.)

Bug fixes

• Tickets now work with parallel transfer
• Tickets are now applied to all connections within a session
• Regression in which ssl_verify_server = 'none' stopped working
• Data objects opened with append flag now seek to end

New features

• Now redirects to resource server for improved transfer performance
• Client-side configuration can now be loaded/saved
• Added auto-close option of data objects
• Added ability to manipulate and query user/group quotas
• Added support for case-insensitive queries

Python iRODS Client - from 1.1.9 to 2.0.1 (cont.)

Deprecation and Removal

• permissions manager has been removed (originally deprecated)
  • Replaced by acls manager
    • Defaults to only raw listing of ACLs, per ils -A

Compatibility Updates

• 2.0.1 brings full compatibility with 4.3.2
• pam_password support since 2.0.0
  • Allows working alongside icommands environments in 4.3+

NFSRODS - from 2.1.0 to 2.3.0

Updates

• Reports its name to the iRODS server for ips
• Resilient to restarts of the iRODS server
  • No longer requires remounting
• Supports resizing of data objects (requires iRODS 4.3.2)
• Improved stability

Ongoing Investigations

• Improve performance of list operation

iRODS Internships - Summer 2024

Convert existing web applications to our new HTTP API (ReactJS + HTTP)
The relatively new iRODS Zone Management Tool is due for its first refactor. Originally built to talk with a REST API, it needs to be converted to talk to the new iRODS HTTP API. If this work on the administrator tool proves pretty straightforward, we are interested in evaluating a similar refactor for our user-level GUI, Metalnx (or to even start designing a new webapp from scratch).

Create new client libraries around our new HTTP API (Various Languages)
Our new iRODS HTTP API is making it easier for developers to interact with the iRODS ecosystem. We would like to help them even more by providing new client libraries in various languages that wrap their native or library-provided http calls. We are most interested in Java, Python, and Javascript, but any language will provide a learning opportunity and help map out the space for other languages.

https://irods.org/2024/01/irods-internship-summer-2024/

Big Picture

Core

• 4.3 - Focus on stability, bug fixes, plugins, clients

• 5.0 - Modernize the deployment process, improve determinism, libstdc++

Clients

• GUIs (ZMT, Metalnx, et al.)

• Onboarding and Syncing (Automated Ingest)

• File System Integration (NFSRODS, SFTP)

• iRODS HTTP API

• iRODS S3 API

Continue building out policy components (Capabilities).

We want installation and management of iRODS to become about policy design, composition, and configuration.

Please share your ...

• Use cases

• Pain points

• Hopes and dreams

Open Source Community Engagement

Get Involved

• Working Groups

• GitHub Issues

• Pull Requests

• Chat List

• Consortium Membership

Tell Others

• Publish, Cite, Advocate, Refer