Kory Draughn
Chief Technologist
iRODS Consortium
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826819/hex_title_page_bg_2020_widescreen.jpg)
Technology Update
May 28-31, 2024
iRODS User Group Meeting 2024
Amsterdam, Netherlands
Contributors
iRODS Release | Issues Closed |
---|---|
4.3.1 | 236 |
4.3.2 | 154 |
$ git shortlog --summary --numbered 4.3.0..4.3.2 228 Kory Draughn 148 Alan King 58 Markus Kitsinger (SwooshyCueb) 18 Martin Jaime Flores Jr 15 Justin James 15 Nishant Dash 11 Derek D 10 d-w-moore
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
7 Terrell Russell 7 Violet White 4 Felix A. Croes 2 Phillip Davis 1 Awab 1 Peter Verraedt 1 june-releford 1 lenocil
4.3.1 Release - Updates since UGM 2023 (not exhaustive)
- New
iadmin
subcommands-
get/set_grid_configuration
-
Manage zone-wide settings stored inR_GRID_CONFIGURATION
-
-
- New APIs
- Includes options for controlling TCP keepalive
- Removed size limit on PAM password
- C++ User Administration library supports changing passwords just like
iadmin
- C++ Connection Pool supports automatic refreshing of connections
- Added
modify_ts_millis
column toR_RESC_MAIN
-
Known in GenQuery asRESC_MODIFY_TIME_MILLIS
-
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
4.3.2 Release
iRODS 4.3.2 is available!
The goals remain the same ...
- Clean implementation
- Clear semantics
- No magic
- Improved tooling
- Focus on data management
iRODS 4.3.2 is smaller than previous releases, but offers improved stability with some nice additions.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
4.3.2 User Experience Updates
- Improved multiple icommands
-
irsync
andiput
correctly ignore symlinks -
ibun
now uses the pax format when handling archive files -
iadmin rmuser/rmgroup
are no longer interchangeable - Help text is clearer and more detailed
-
- Setup script correctly handles configuration of local storage for iRODS Consumer servers
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
4.3.2 Core Server Updates
- GenQuery2
- Replica Truncate API
- Support for MariaDB
- Serialization of more iRODS data types
- Improved server stability
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
** Bold items discussed in this talk **
- Improved documentation
- How to report security vulnerabilities
- Updated dynamic PEP listing
- Replication
- Good replicas overwriting good replicas is now a no-op rather than an error
GenQuery2
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
GenQuery2 is now shipped with the server.
It is still considered experimental, but use is encouraged.
Its implementation can change as a result of community feedback.
Not used by the server, yet.
GenQuery2 will be available in the next release of the Python iRODS Client and Jargon.
New API - rc_replica_truncate
Allows applications to adjust the size of a single replica.
Motivated by NFSRODS.
Additional Information
- Modeled after POSIX
truncate(2)
- Includes a microservice for policy
- Replaces
rcDataObjTruncate
(deprecated)
There are plans to add support for an ftruncate
-style API as well.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Serialization of more iRODS data types
iRODS 4.3.2 now serializes the following data types for the rule engine plugin framework.
-
StructFileExtAndRegInp
(ibun
) -
ExecMyRuleInp
(irule
) -
MsParamArray
(irule
)
Policy implementers can now inspect the contents of requests dealing with archive files and rule execution, and react accordingly.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
iRODS 5.0 - Primary Goals
Modernize the startup/shutdown process for the iRODS server.
Things we are going to address ...
- Startup/Shutdown sequence is more deterministic
- No mixing of fork[-exec] and multi-threading logic
- No dependency on external Python code
- Server configuration is only loaded on startup
- Supports live reloads via
SIGHUP
signal - Agents no longer reload the configuration
- Supports live reloads via
- Non-package installation is supported by design
We are considering ...
- Decoupling of hostnames from servers
- Possibly the removal of hostnames altogether
- May not be possible due to server-to-server redirects
- Possibly the removal of hostnames altogether
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
iRODS 5.0 - Early Design and Implementation
Experimentation is now in progress. It's too early to assert anything about the design/implementation. Initial design is heavily influenced by the iRODS HTTP API.
At the moment, the plan is to honor the following ...
- Clear separation of concerns between components
- Plugins handle their own shared memory
- Multi-threaded code is constrained to leaf processes
- Runs as a normal application
- Accepts a path to directory containing configuration files
- Can be launched as a daemon process
- Control plane runs as a separate process
- Meets all requirements for working with systemd
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Core Development Team Talks
-
Not in This Talk / Separate Talks
-
Phil Owen
-
iRODS Build and Test v9: Automation via GitHub and Kubernetes
-
-
Markus Kitsinger
-
iRODS Build and Packaging: 2024 Update
-
-
Kory Draughn and Martin Flores
-
iRODS HTTP API v0.3.0 with OpenID Connect
-
-
Justin James
-
iRODS S3 API v0.2.0 with Multipart
-
-
Terrell Russell
-
DAViDD: Initial data management solution for UNC's READDI AViDD Center
-
iRODS Metadata Templates Working Group: Building Blocks and Lessions Learned
-
-
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
-
Included in This Talk
-
Kory Draughn
-
Indexing Capability Plugin
-
Python Rule Engine Plugin
-
-
Derek Dong
-
Metadata Guard Rule Engine Plugin
-
-
Justin James
-
S3 Resource Plugin
-
Globus Connector
-
-
Daniel Moore
-
Python iRODS Client
-
-
Kory Draughn
-
NFSRODS
-
-
Indexing Capability Plugin
- Requires Elasticsearch 7 or later
- Uses Boost.Beast library for communication with Elasticsearch
- elasticlient and CPR replaced by Boost.Beast
- No change in behavior
- Removed document type rule engine plugin
- No longer serves a purpose due to changes in Elasticsearch
- Supports TLS communication with Elasticsearch
- Supports Basic Authentication
- Modernized CMake
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Python Rule Engine Plugin
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
irods_rule_vars
now available in more contexts
- Accessible in imported python modules
-
irods_rule_vars
formerly known asglobal_vars
- Deprecated
global_vars
Improved documentation
- Covers how to pass values back to the caller, across rule engine plugin boundaries
- Covers what happens to the output of
print()
Metadata Guard Rule Engine Plugin
Resolved some holes in the implementation.
- Handles atomic metadata operations API PEPs
- Disallows renaming non-protected AVU to existing protected AVU
- Disallows use of deprecated
imeta rmw
- Wildcard operations can bypass plugin logic (e.g.
"irod%"
) - Wildcard operations make it difficult to cover all metadata cases
- Wildcard operations can bypass plugin logic (e.g.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
S3 Resource Plugin and Globus Connector
S3 Resource Plugin
- libs3 dependency absorbed into repository
- libs3 no longer shipped as externals package
- Fixed streaming with decoupled mode after server redirect
- Fixed setting of open flags on PUT operation
Globus Connector
- Supports rename functionality
- Supports preservation of file modification time
- Added Docker-based testing framework
-
O_CREAT
flag is now set on the first open for write operations
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Python iRODS Client - from 1.1.9 to 2.0.1
$ git shortlog --summary --numbered v1.1.9..v2.0.1 25 d-w-moore 7 Terrell Russell 1 Raoul Schram
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Release | Issues Affected |
---|---|
1.1.9 | 6 |
2.0.0 | 20 |
2.0.1 | 9 |
Total | 35 |
Python iRODS Client - from 1.1.9 to 2.0.1 (cont.)
Bug fixes
- Tickets now work with parallel transfer
- Tickets are now applied to all connections within a session
- Regression in which
ssl_verify_server = 'none'
stopped working - Data objects opened with append flag now seek to end
New features
- Now redirects to resource server for improved transfer performance
- Client-side configuration can now be loaded/saved
- Added auto-close option of data objects
- Added ability to manipulate and query user/group quotas
- Added support for case-insensitive queries
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Python iRODS Client - from 1.1.9 to 2.0.1 (cont.)
Deprecation and Removal
-
permissions
manager has been removed (originally deprecated)- Replaced by
acls
manager- Defaults to only raw listing of ACLs, per
ils -A
- Defaults to only raw listing of ACLs, per
- Replaced by
Compatibility Updates
- 2.0.1 brings full compatibility with 4.3.2
-
pam_password
support since 2.0.0- Allows working alongside icommands environments in 4.3+
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
NFSRODS - from 2.1.0 to 2.3.0
Updates
- Reports its name to the iRODS server for
ips
-
Resilient to restarts of the iRODS server
- No longer requires remounting
- Supports resizing of data objects (requires iRODS 4.3.2)
- Improved stability
Ongoing Investigations
- Improve performance of list operation
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Release | Issues Closed |
---|---|
2.2.0 | 14 |
2.3.0 | 6 |
iRODS Internships - Summer 2024
Convert existing web applications to our new HTTP API (ReactJS + HTTP)
The relatively new iRODS Zone Management Tool is due for its first refactor. Originally built to talk with a REST API, it needs to be converted to talk to the new iRODS HTTP API. If this work on the administrator tool proves pretty straightforward, we are interested in evaluating a similar refactor for our user-level GUI, Metalnx (or to even start designing a new webapp from scratch).
Create new client libraries around our new HTTP API (Various Languages)
Our new iRODS HTTP API is making it easier for developers to interact with the iRODS ecosystem. We would like to help them even more by providing new client libraries in various languages that wrap their native or library-provided http calls. We are most interested in Java, Python, and Javascript, but any language will provide a learning opportunity and help map out the space for other languages.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Big Picture
Core
-
4.3 - Focus on stability, bug fixes, plugins, clients
-
5.0 - Modernize the deployment process, improve determinism, libstdc++
Clients
-
GUIs (ZMT, Metalnx, et al.)
-
Onboarding and Syncing (Automated Ingest)
-
File System Integration (NFSRODS, SFTP)
-
iRODS HTTP API
-
iRODS S3 API
Continue building out policy components (Capabilities).
We want installation and management of iRODS to become about policy design, composition, and configuration.
Please share your ...
-
Use cases
-
Pain points
-
Hopes and dreams
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Open Source Community Engagement
Get Involved
-
Working Groups
-
GitHub Issues
-
Pull Requests
-
Chat List
-
Consortium Membership
Tell Others
-
Publish, Cite, Advocate, Refer
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
UGM 2024 - Technology Update
By iRODS Consortium
UGM 2024 - Technology Update
iRODS User Group Meeting 2024 - Technology Update
- 126