Terrell Russell, Ph.D.
@terrellrussell
Interim Executive Director, iRODS Consortium
November 14-19, 2021
Supercomputing 2021
St. Louis, MO
Research Community Requirements
Drive Expanded Scale
Data Management Features
Research Community Requirements
Drive Expanded Scale
Data Management Features
The iRODS Consortium
Our Mission
- Continuous Improvement
- Grow the Community
- Standardization
- Show value to our Membership
Our Membership
Consortium
Member
Consortium
Member
Consortium
Member
Consortium
Member
Research Community Requirements Drive Features
- History
- Philosophy
- Process
- Results
History
- 1995 - SRB started (grid storage)
- 2004 - iRODS started (added rule engine / policy)
- 2013 - Consortium founded by RENCI, DICE, and DDN
- 2014 - Consortium accepted the code base
- 42 releases of iRODS to date
Why use iRODS?
People need a solution for:
- Managing large amounts of data across various storage technologies
- Controlling access to data
- Searching their data quickly and efficiently
- Automation
The larger the organization, the more they need software like iRODS.
iRODS as the Integration Layer
iRODS Core Competencies
- Packaged and supported solutions
- Require configuration not code
- Derived from the majority of use cases observed in the user community
iRODS Capabilities
The Data Management Model
Research Community Requirements Drive Features
- History
- Philosophy
- Process
- Results
Philosophical Drivers
-
100-year view
-
Plugin Architecture
-
core is generic - protocol, api, bookkeeping
-
plugins are specific
-
policy composition
-
-
Modern core libraries
-
standardized interfaces
-
refactored iRODS internals
-
ease of (re)use
-
fewer bugs
-
-
- Configuration, Not Code
Research Community Requirements Drive Features
- History
- Philosophy
-
Process
- Google Group
- GitHub
- Working Groups
- Results
Working Groups
Technology Working Group
- Goal: To keep everyone up to date, provide a forum for roadmap discussion and collaboration opportunities
Metadata Templates Working Group
- Goal: To define a standardized process for the application and management of metadata templates by the iRODS Server
- NIEHS / Data Commons
- Utrecht / Yoda
- Maastricht / DataHub+
- Arizona / CyVerse
Authentication Working Group
- Goal: To provide a more flexible authentication mechanism to the iRODS Server
- SURF
- NIEHS
- Sanger
- CyVerse
- Utrecht
Working Groups
Imaging Working Group
- Goal: To provide a standardized suite of imaging policies and practices for integration with existing tools and pipelines
- Open Microscopy Environment (and OMERO)
- Neuroscience Microscopy Core at UNC School of Medicine
- New York University
- Santa Clara University
- UC San Diego
- CyVerse
- NIEHS
Research Community Requirements Drive Features
- History
- Philosophy
- Process
- Results
In The Last Year
Plugins
- Python Rule Engine Plugin
- Storage Tiering Rule Engine Plugin
- Auditing (AMQP) Rule Engine Plugin
- S3 Resource Plugin
- Kerberos Authentication Plugin
- Curl Microservice Plugin
- Indexing Rule Engine Plugin
- Logical Quotas Rule Engine Plugin
- Metadata Guard Rule Engine Plugin
- Policy Composition Framework
- Policy Composition Event Handlers
Clients
- Python iRODS Client
- Metalnx
- NFSRODS
- Automated Ingest Framework
- AWS Lambda for S3
- C++ REST API
- Zone Management Tool (ZMT)
- iRODS Globus Connector
- iCommands
AWS Lambda for S3
Open Source:
- iRODS Client
- Developed in collaboration with BMS
Automated Ingest Capability
- iRODS Capability
- Developed in collaboration with Roche
Storage Tiering Capability
- iRODS Capability
- Developed in collaboration with Roche
Indexing Capability
Open Source:
- iRODS Capability
- Developed in collaboration with NIEHS and BMS
Metalnx
Open Source:
- Web GUI
- Developed and Contributed by EMC
go-irodsclient
Open Source:
- Golang iRODS Client Library
- Developed by CyVerse
NFSRODS
Open Source:
- iRODS Client, NFSv4.1 Server
- Developed by iRODS Consortium in collaboration with CU Boulder
S3 Resource Plugin
Open Source:
- iRODS storage resource
- Developed Glacier-compatibility in collaboration with Fujifilm
iRODS Globus Connector
- iRODS Client, Globus Connector
- Developed in collaboration with Globus
Questions?
Proper data management requires policy enforcement.
These policies will change over time.
Open source is the best practice for a 100-year view.
Terrell Russell
@terrellrussell
iRODS Consortium
Thank you.
irods.org
SC21 - Community-Driven Features
By iRODS Consortium
SC21 - Community-Driven Features
- 699