Technology Update
June 25-28, 2019
iRODS User Group Meeting 2019
Utrecht, Netherlands
Terrell Russell, Ph.D.
@terrellrussell
Chief Technologist, iRODS Consortium
Technology Update
In The Last Year
iRODS Release | Issues Closed |
---|---|
4.1.12 | 36 |
4.2.4 | 31 |
4.2.5 | 57 |
4.2.6 | 27 |
Plugins
Clients
~/irods $ git shortlog --summary --numbered 4.1.11..4.1.12 27 Alan King 11 Terrell Russell 1 Justin James ~/irods $ git shortlog --summary --numbered 4.2.3..4-2-stable 39 Alan King 20 Kory Draughn 20 Terrell Russell 14 Andrew Kelly 6 Jason Coposky 5 Justin James 5 Zoey Greer 5 d-w-moore 3 Hao Xu 2 Felix A. Croes 2 jkgill 1 Kyle Ferriter 1 Matt Watson
In The Last Year
Ongoing Development Work
Policy Advancement
Steadily filling out the iRODS Data Management Model...
Working Groups
Technology Working Group
Metadata Templates Working Group
Changelog Working Group (Upcoming...)
Last Year and Next Year
New Libraries, Oh My!
Goal: Provide standardized interfaces that simplify common iRODS tasks
Six new libraries (so far):
Benefits:
Originally planned for 4.3.0.
Backported to 4.2.5 and 4.2.6 due to their ease of use and immediate impact.
irodsReServer -> irodsDelayServer
Old irodsReServer (pre-4.2.5)
Fork-exec model for synchronous work distribution
Maximum of 256 rules processed per wake-up
Rules to be run later may block other rules
Long-running rules may block entire RE server process
New irodsReServer (4.2.5+)
Rename to irodsDelayServer (4.3.0)
iRODS Query Processor, distributed rule execution, ...
The Missing Link: Intermediate Replicas
Intermediate replica
Putting a data object into iRODS
# Intermediate state of all replicas - an iput to a replication resource with 3 leaves $ ils -l /tempZone/home/rods: rods 0 repl;ufs0 0 2019-04-08.15:38 ? foo rods 1 repl;ufs1 0 2019-04-08.15:38 ? foo rods 2 repl;ufs2 0 2019-04-08.15:38 ? foo # After initial put is complete and before synchronous replication has completed $ ils -l /tempZone/home/rods: rods 0 repl;ufs0 12345 2019-04-08.15:38 & foo rods 1 repl;ufs1 0 2019-04-08.15:38 ? foo rods 2 repl;ufs2 0 2019-04-08.15:38 ? foo # After replication has succeeded $ ils -l /tempZone/home/rods: rods 0 repl;ufs0 12345 2019-04-08.15:38 & foo rods 1 repl;ufs1 12345 2019-04-08.15:38 & foo rods 2 repl;ufs2 12345 2019-04-08.15:38 & foo
$ ils -l
/tempZone/home/rods:
rods 0 resc1 54321 2019-04-08.15:38 & bar
rods 1 resc2 12345 2019-04-08.15:38 X bar
Stale replicas will now be indicated with 'X'
iRODS Build and Test - History
July 2011
Python → Node.js → RabbitMQ → Celery → Eucalyptus
October 2012
January 2013
October 2013
Spring 2015
Spring 2017
iRODS Build and Test - 2018 Promises
iRODS Build and Test - Reality
iRODS Build and Test - 2019 Architecture
iRODS Build and Test - Demo
OS | Database | Containers | Total | |
---|---|---|---|---|
Core | 2 | 1 | 2 test suites | 4 |
Plugins | 2 | 1 | 2 plugins (1 suite each) | 4 |
Federation | 2 | 1 | 2 providers (1 suite each) | 4 |
Topology | 2 | 1 | 4 (1 provider + 3 consumers) | 8 |
TOTAL | 20 |
iRODS Build and Test - Future
4.3.0 Update
4.3.0 Logging Update
Today
Design Goals
Local Files (rsyslog) | Remote (rsyslog) | --stdout | |
---|---|---|---|
Packaged | default | centralized logging | Docker-friendly |
Non-Package Install | probably n/a |
probably n/a |
HPC and development |
Policy Composition
With the new libraries, we can rewrite 90% of the internals, and then fix the things that depend on them later, with little expectation of regression, because the interfaces remain the same.
Internally
Externally
Continuation within the Rule Engine Plugin Framework allows administrators to break apart monolithic PEP implementations into reusable components.
Big Picture
Core
Clients
Continue building out policy components (Capabilities)
We want installation and management of iRODS to become about policy design, composition, and configuration.
Please share your:
Open Source Community Engagement
Get Involved
Tell Others