Terrell Russell, Ph.D.

@terrellrussell

Executive Director, iRODS Consortium

January 24-27, 2022

CS3 2022

Virtual

Research Community Requirements

Drive Expanded Scale

Data Management Features

Research Community Requirements

Drive Expanded Scale

Data Management Features

The iRODS Consortium

Our Mission

• Continuous Improvement
• Grow the Community
• Standardization
• Show value to our Membership

Our Membership

Consortium

Member

Consortium

Member

Consortium

Member

Consortium

Member

Consortium

Member

Research Community Requirements Drive Features

• History
• Philosophy
• Process
• Results

History

• 1995 - SRB started (grid storage)
• 2004 - iRODS started (added rule engine / policy layer)
• 2013 - Consortium founded by RENCI, DICE, and DDN
• 2014 - Consortium accepted the code base
• 43 releases of iRODS to date

Why use iRODS?

People need a solution for:

• Managing large amounts of data across various storage technologies
• Controlling access to data
• Searching their data quickly and efficiently
• Automation

The larger the organization, the more they need software like iRODS.

iRODS as the Integration Layer

• Packaged and supported solutions
• Require configuration not code
• Derived from the majority of use cases observed in the user community

iRODS Capabilities

The Data Management Model

Research Community Requirements Drive Features

• History
• Philosophy
• Process
• Results

Philosophical Drivers

• 100-year view

• Plugin Architecture

  • core is generic - protocol, api, bookkeeping

  • plugins are specific

  • policy composition

​​

• Modern core libraries

  • standardized interfaces

  • refactored iRODS internals

    • ease of (re)use

    • fewer bugs

​

• Configuration, Not Code

Research Community Requirements Drive Features

• History
• Philosophy
• Process
  • Google Group
  • GitHub
  • Working Groups
• Results

Working Groups

Technology Working Group

• Goal: To keep everyone up to date, provide a forum for roadmap discussion and collaboration opportunities

Metadata Templates Working Group

• Goal: To define a standardized process for the application and management of metadata templates by the iRODS Server
  • NIEHS / Data Commons
  • Utrecht / Yoda
  • Maastricht / DataHub+
  • Arizona / CyVerse

Authentication Working Group

• Goal: To provide a more flexible authentication mechanism to the iRODS Server
  • SURF
  • NIEHS
  • Sanger
  • CyVerse
  • Utrecht

Working Groups

Imaging Working Group

• Goal: To provide a standardized suite of imaging policies and practices for integration with existing tools and pipelines
  • Open Microscopy Environment (and OMERO)
  • Neuroscience Microscopy Core at UNC School of Medicine
  • New York University
  • Santa Clara University
  • UC San Diego
  • UC Santa Cruz
  • UMass
  • Harvard
  • Maastricht University
  • Wellcome Sanger Institute
  • CyVerse
  • NIEHS
  • Netherlands Cancer Institute (NKI)
  • Francis Crick Institute
  • Fritz Lipmann Institute
  • Osnabrück University
  • RIKEN

Research Community Requirements Drive Features

• History
• Philosophy
• Process
• Results

In The Last Year

Plugins

• Python Rule Engine Plugin
• Storage Tiering Rule Engine Plugin
• Auditing (AMQP) Rule Engine Plugin
• S3 Resource Plugin
• Kerberos Authentication Plugin
• Curl Microservice Plugin
• Indexing Rule Engine Plugin
• Logical Quotas Rule Engine Plugin
• Metadata Guard Rule Engine Plugin
• Policy Composition Framework
• Policy Composition Event Handlers

Clients

• Python iRODS Client
• Metalnx
• NFSRODS
• Automated Ingest Framework
• AWS Lambda for S3
• C++ REST API
• Zone Management Tool (ZMT)
• iRODS Globus Connector
• iCommands

AWS Lambda for S3

Open Source:

    https://github.com/irods/irods_client_aws_lambda_s3

• iRODS Client
• Developed in collaboration with BMS

Automated Ingest Capability

Open Source:

    https://github.com/irods/irods_capability_automated_ingest

• iRODS Capability
• Developed in collaboration with Roche

Storage Tiering Capability

Open Source:

    https://github.com/irods/irods_capability_storage_tiering

• iRODS Capability
• Developed in collaboration with Roche

Indexing Capability

Open Source:

    https://github.com/irods/irods_capability_indexing

• iRODS Capability
• Developed in collaboration with NIEHS and BMS

Metalnx

Open Source:

    https://github.com/irods-contrib/metalnx-web

• Web GUI
• Developed and Contributed by EMC

go-irodsclient

Open Source:

    https://github.com/cyverse/go-irodsclient

• Golang iRODS Client Library
• Developed by CyVerse

NFSRODS

Open Source:

    https://github.com/irods/irods_client_nfsrods

• iRODS Client, NFSv4.1 Server
• Developed in collaboration with CU Boulder and BMS

S3 Resource Plugin

Open Source:

    https://github.com/irods/irods_resource_plugin_s3

• iRODS storage resource
• Developed Glacier-compatibility in collaboration with Fujifilm

iRODS Globus Connector

Open Source:

    https://github.com/irods/irods_client_globus_connector

• iRODS Client, Globus Connector
• Developed in collaboration with Globus

Questions?

Proper data management requires policy enforcement.

These policies will change over time.

Open source is the best practice for a 100-year view.

Terrell Russell

@terrellrussell

iRODS Consortium

Thank you.

irods.org