Kory Draughn
Chief Technologist
iRODS Consortium
June 13-16, 2023
iRODS User Group Meeting 2023
Chapel Hill, North Carolina
Technology Update
Technology Update
Philosophical Drivers
Plugin Architecture
Core is generic - protocol, api, bookkeeping
Plugins are specific
Policy composition
Modern core libraries
Standardized interfaces
Refactor iRODS internals
Ease of (re)use
Fewer bugs
iRODS 4.2 Series
4.2.12 likely the final release of the 4.2 series.
Limited to security fixes, bug fixes, and trivial enhancements.
iRODS 4.2.12
This marks the final release of the 4.2 series.
The Consortium is now focused on the 4.3 series
Contributors
iRODS Release | Issues Closed |
---|---|
4.2.12 | 160 |
~/irods $ git shortlog --summary --numbered 4.2.11..4.2.12 67 Alan King 58 Kory Draughn 13 Daniel Moore 9 Justin James 8 Markus Kitsinger (SwooshyCueb) 6 Martin Jaime Flores Jr 4 Felix A. Croes 2 Alastair Smith 1 Phillip Davis (2022 Summer Intern) 1 Terrell Russell
4.3.0 Core Server Enhancements
** Bold items discussed in this talk **
Admin Mode for Metadata APIs
Gives administrators the power to manipulate any metadata in the zone regardless of permissions.
Simplifies implementation (e.g. clients, plugins, etc.)
Removes the need for client identity switching
Requires the ADMIN_KW keyword
imeta now supports admin mode via the -M option.
For example:
rods@ugm2022:~ $ imeta -M add -d /tempZone/home/alice/file.txt
Improved Server Durability
iRODS 4.3.0 now contains an internal CRON-like manager.
Enables tasks to be run periodically in the background.
The primary irodsServer process will respawn the Agent Factory and/or Delay Server if they aren't running.
// Setup the delay server CRON task.
// The delay server will launch just before we enter the server's main loop.
ix::cron::cron_builder delay_server;
delay_server
.interval(5)
.task([enable_test_mode, write_to_stdout] {
migrate_delay_server(enable_test_mode, write_to_stdout);
});
ix::cron::cron::instance().add_task(delay_server.build());
GenQuery Reimplementation
The goal is to provide a high quality implementation that solves the issues of the past implementation.
The new implementation is being built using flex/bison. The use of flex/bison enables the following:
Current Features:
Expected to be released in iRODS 4.3.1 or 4.3.2.
Clients
Jargon
Metalnx
Zone Management Tool (ZMT)
NFSRODS
Clients - Jargon
Current version is 4.3.2.5.
Supports Parallel Transfer over port 1247.
Managed internally by Jargon for simplicity.
Clients - Metalnx
Includes several enhancements and bug fixes since UGM 2021.
Current version is 2.6.0.
Search Enhancements:
Merged property search and AVU search interfaces
Searches the logical space only
Closer to iquest
Future Plans:
Remove all administrative capabilities
Remove dependency on separate Metalnx specific database
Clients - Zone Management Tool (ZMT)
The premier administration tool for managing an iRODS zone.
Current version is 0.2.0.
Features:
Manage users, groups, and resources
Health check files (default and custom)
Lightweight
Clients - NFSRODS
Current version is 2.1.0.
Notable Updates:
Supports large file transfer
Parallel Transfer over port 1247
Separated the compilation from building of docker image
Allows developers to iterate faster
Simplifies the development model
Includes docker-compose.yml template file
https://github.com/irods/irods_client_nfsrods/releases/tag/2.1.0
Last Year and Next Year
Not in This Talk / Separate Talks
Terrell Russell and Kory Draughn
iRODS Delay Server Migration
Justin James
iRODS S3 Resource Plugin: Glacier Support
Alan King
iRODS Development and Testing Environments (v8)
Daniel Moore
iRODS Client Library: Python iRODS Client 1.1.4
Markus Kitsinger
iRODS Build and Packaging Update
Included in This Talk
Alan King
Authentication Plugin Framework
Dan
Python Rule Engine Plugin
Indexing Capability
New Authentication Plugin Framework
Python Rule Engine Plugin
Indexing Capability
Philosophy to Policy
With the new libraries and first class replicas, we can rewrite 90% of the internals, and then fix the things that depend on them later, with little expectation of regression, because the interfaces remain the same.
Internally
We will have a new API... but not really
Instead, we stepped back and built good tools
Allows us to refactor and go faster without breaking the 4.x API
This has turned out to be more powerful than originally expected
Externally
It's a good story, the ability to compose policy into capabilities
Can build smaller pieces of functionality which can be composed to help solve larger problems
We don't have to worry about side effects
Continuation within the Rule Engine Plugin Framework allows administrators to break apart monolithic policy implementations into reusable components.
iRODS Internships - Summer 2022
Refactor to modernize irodsServer (4.3.x)
The iRODS server was originally written in C around 20 years ago. We would like to refactor the core server software with C++ to use modern, high-level techniques for purposes of maintainability and performance as well as extensibility.
Add live reload for Server configuration and Agent resource manager (4.3.x)
An iRODS server uses a process model which spawns independent agents to service incoming requests. While performant and relatively safe, the agents do not currently interact and will hold stale information when another part of the system changes. Two places where it would be beneficial for these things to be smarter would be the configuration of the storage resources in the zone or of the server itself.
Refactor Audit Plugin to fix invalid JSON (4.2.12)
The iRODS Consortium supports the AMQP Audit Rule Engine Plugin which sends information to a message broker for every policy enforcement point (PEP) in the iRODS server. We have discovered that the information we are sending to the message broker includes some slightly malformed JSON. We need to identify what is happening and make it better.
iRODS Internships - Summer 2022 (cont.)
Libraries (4.3.x)
iRODS has been around for a long time. Like other similarly sized projects, iRODS has grown several low-level libraries. We'd like to provide higher level APIs that wrap these existing low-level libraries to simplify use and enable easier maintenance of the software. The new libraries should take full advantage of what is provided by modern C++.
iRODS Testing Environment Web Application
This new application would be a front end / GUI to an existing Python command line tool. This tool helps us test a distributed system (iRODS) to provide confidence to our users around the world. Design and implementation of a new web application, its API, GUI, and data storage formats. This will require interfacing with the iRODS team and coding in public on open source software.
Big Picture
Core
4.3.x - Harden and Polish
Clients
GUIs (Metalnx, ZMT, Kanki, et al.)
Onboarding and Syncing (Automated Ingest)
File System Integration (NFSRODS / SMBRODS, SFTP)
iRODS Console (alongside existing iCommands)
C++ REST API
Continue building out policy components (Capabilities)
We want installation and management of iRODS to become about policy design, composition, and configuration.
Please share your:
use cases
pain points
hopes and dreams
Open Source Community Engagement
Get Involved
Working Groups
GitHub Issues
Pull Requests
Chat List
Consortium Membership
Tell Others
Publish, Cite, Advocate, Refer