Terrell Russell, Ph.D.

@terrellrussell

Interim Executive Director, iRODS Consortium

November 14-19, 2021

Supercomputing 2021

St. Louis, MO

Research Community Requirements

Drive Expanded Scale

Data Management Features

Research Community Requirements

Drive Expanded Scale

Data Management Features

The iRODS Consortium

Our Mission

  • Continuous Improvement
  • Grow the Community
  • Standardization
  • Show value to our Membership

Our Membership

Consortium

Member

Consortium

Member

Consortium

Member

Consortium

Member

Research Community Requirements Drive Features

  • History
  • Philosophy
  • Process
  • Results

History

  • 1995 - SRB started (grid storage)
  • 2004 - iRODS started (added rule engine / policy)
  • 2013 - Consortium founded by RENCI, DICE, and DDN
  • 2014 - Consortium accepted the code base
  • 42 releases of iRODS to date

Why use iRODS?

People need a solution for:

  • Managing large amounts of data across various storage technologies
  • Controlling access to data
  • Searching their data quickly and efficiently
  • Automation

 

The larger the organization, the more they need software like iRODS.

iRODS as the Integration Layer

iRODS Core Competencies

  • Packaged and supported solutions
  • Require configuration not code
  • Derived from the majority of use cases observed in the user community

iRODS Capabilities

The Data Management Model

Research Community Requirements Drive Features

  • History
  • Philosophy
  • Process
  • Results

Philosophical Drivers

  • 100-year view

 

  • Plugin Architecture

    • core is generic - protocol, api, bookkeeping

    • plugins are specific

    • policy composition

​​

  • Modern core libraries

    • standardized interfaces

    • refactored iRODS internals

      • ease of (re)use

      • fewer bugs

  • Configuration, Not Code

Research Community Requirements Drive Features

  • History
  • Philosophy
  • Process
    • Google Group
    • GitHub
    • Working Groups
  • Results

Working Groups

Technology Working Group

  • Goal: To keep everyone up to date, provide a forum for roadmap discussion and collaboration opportunities

 

Metadata Templates Working Group

  • Goal: To define a standardized process for the application and management of metadata templates by the iRODS Server
    • NIEHS / Data Commons
    • Utrecht / Yoda
    • Maastricht / DataHub+
    • Arizona / CyVerse

 

Authentication Working Group

  • Goal: To provide a more flexible authentication mechanism to the iRODS Server
    • SURF
    • NIEHS
    • Sanger
    • CyVerse
    • Utrecht

Working Groups

Imaging Working Group

  • Goal: To provide a standardized suite of imaging policies and practices for integration with existing tools and pipelines
    • Open Microscopy Environment (and OMERO)
    • Neuroscience Microscopy Core at UNC School of Medicine
    • New York University
    • Santa Clara University
    • UC San Diego
    • CyVerse
    • NIEHS

Research Community Requirements Drive Features

  • History
  • Philosophy
  • Process
  • Results

In The Last Year

Plugins

  • Python Rule Engine Plugin
  • Storage Tiering Rule Engine Plugin
  • Auditing (AMQP) Rule Engine Plugin
  • S3 Resource Plugin
  • Kerberos Authentication Plugin
  • Curl Microservice Plugin
  • Indexing Rule Engine Plugin
  • Logical Quotas Rule Engine Plugin
  • Metadata Guard Rule Engine Plugin
  • Policy Composition Framework
  • Policy Composition Event Handlers

Clients

  • Python iRODS Client
  • Metalnx
  • NFSRODS
  • Automated Ingest Framework
  • AWS Lambda for S3
  • C++ REST API
  • Zone Management Tool (ZMT)
  • iRODS Globus Connector
  • iCommands

AWS Lambda for S3

  • iRODS Client
  • Developed in collaboration with BMS

Automated Ingest Capability

  • iRODS Capability
  • Developed in collaboration with Roche

Storage Tiering Capability

  • iRODS Capability
  • Developed in collaboration with Roche

Indexing Capability

  • iRODS Capability
  • Developed in collaboration with NIEHS and BMS

Metalnx

  • Web GUI
  • Developed and Contributed by EMC

go-irodsclient

  • Golang iRODS Client Library
  • Developed by CyVerse

NFSRODS

  • iRODS Client, NFSv4.1 Server
  • Developed by iRODS Consortium in collaboration with CU Boulder

S3 Resource Plugin

  • iRODS storage resource
  • Developed Glacier-compatibility in collaboration with Fujifilm

iRODS Globus Connector

  • iRODS Client, Globus Connector
  • Developed in collaboration with Globus

Questions?

Proper data management requires policy enforcement.

 

These policies will change over time.

 

Open source is the best practice for a 100-year view.

Terrell Russell

@terrellrussell

iRODS Consortium

Thank you.

 

irods.org

SC21 - Community-Driven Features

By iRODS Consortium

SC21 - Community-Driven Features

  • 699