BioIT World 2025 - Building an Approachable Cost-Effective Data Management Platform

April 2-4, 2025

BioIT World 2025

Boston, MA

Building an

Approachable Cost-Effective Data Management Platform

Kory Draughn

Chief Technologist

iRODS Consortium

Our Membership

Consortium

Member

Consortium

Member

Consortium

Member

Partners and Users: Past and Present

Supercomputing Centers
Physics
Library / Archives
Genomics
Bio / Pharmaceutical
Hydrology / Weather
Medical
Manufacturing
Shipping / Logistics
Automotive

Mission

Long-term data management is best executed when policies are clear and infrastructure is abstracted and swappable.

iRODS has a desire to be normal and boring for the administrator and approachable and powerful for the user.

This talk will cover recent advances and interfaces which allow companies to sustain FAIR data practices, enforce consistency and reproducibility, and realize cost-savings through open source software.

What is iRODS?

Open Source

C++ client-server architecture
iRODS Protocol and RPC API
BSD-3 Licensed

Distributed

Runs on a laptop, a cluster, on premises or geographically distributed

Data Centric & Metadata Driven

Insulate both your users and your data from your infrastructure over time

iRODS as the Integration Layer

Why use iRODS?

People need a solution for:

Managing large amounts of data across various storage technologies
Controlling access to data
Searching their data quickly and efficiently
Automation

The larger the organization, the more they need software like iRODS.

Where is iRODS used?

Research, Commercial, and Governmental Organizations

Largest userbase in Europe
- Strict data protection laws
- Data sovereignty
- Federation with other entities for collaboration

iRODS is designed for long-term storage needs - 100+ years
- Keeps your infrastructure flexible

The Data Management Model

Our Approach

Focus on what we do best
- Bookkeeping, abstracting storage, enforcing policy
- Providing good building blocks
- Being the glue between software
Meet people where they are
- Protocol Plumbing
Listen to users and address pain points
Enable the community to build their own tools easily

https://irods.org/roadmap/

Flexible Authentication

iRODS 4.3 introduced a new authentication plugin framework which enables client-driven authentication flows.

This effort led to the development of the PAM Interactive Authentication plugin - a community-built plugin enabling dynamic authentication flows.

This plugin pushes the details of authentication from the server to the PAM stack, enabling various authentication schemes and flows.

Protocol Plumbing - Presenting iRODS as other Protocols

Over the last few years, the ecosystem around the iRODS server has continued to expand.

Integration with other types of systems is a valuable way to increase accessibility without teaching existing tools about the iRODS protocol or introducing new tools to users.

With some plumbing, existing tools get the benefit of visibility into an iRODS deployment.

iRODS + HTTP

The iRODS HTTP API is a client application designed to make iRODS approachable by presenting a cohesive representation of the iRODS API over the HTTP protocol.

It supports OpenID Connect, allowing adminstrators to leverage existing authentication services.

It's on track to be absorbed into the iRODS server.

Future

iRODS 5 is the next major release of the software and represents three years of significant effort towards our goal of being normal and boring.

Improved process model
Improved packaging and version handling
Improved cloud compatibility
Improved support for:
- Time-series Data / Statistics
- Dashboarding

Thank you!

Questions?

https://irods.org/ugm2025

June 17-20, 2025

Ingest to Institutional Repository

As data matures and reaches a broader community, data management policy must also evolve to meet these additional requirements.

Data Virtualization

Combine various distributed storage technologies into a Unified Namespace

Existing file systems
Cloud storage
On premises object storage
Archival storage systems

iRODS provides a logical view into the complex physical representation of your data, distributed geographically, and at scale.

Data Discovery

Attach metadata to any first class entity within the iRODS Zone

Data Objects
Collections
Users
Storage Resources
The Namespace

iRODS supports automated and user-provided metadata which makes your data and infrastructure more discoverable, operational, and valuable.

Workflow Automation

Policy Enforcement Points (PEPs) are triggered by every operation within the framework

Authentication
Storage Access
Database Interaction
Network Activity
Extensible RPC API

The iRODS rule engine framework provides the ability to capture real world policy as computer actionable rules which may allow, deny, or add context to operations within the system.

Dynamic Policy Enforcement

The iRODS rule may:

restrict access
log for audit and reporting
provide additional context
send a notification

Dynamic Policy Enforcement

A single API call expands to many plugin operations all of which may invoke policy enforcement.

Plugin Interfaces:

Authentication
Database
Storage
Network
Rule Engine
Microservice
RPC API

Secure Collaboration

iRODS allows for collaboration across administrative boundaries after deployment

No need for common infrastructure
No need for shared funding
Affords temporary collaborations

iRODS provides the ability to federate namespaces across organizations without pre-coordinated funding or effort.

iRODS Clients

https://irods.org/clients

iRODS S3 Functionality

https://github.com/irods/irods_resource_plugin_s3

The iRODS S3 storage resource plugin allows iRODS to use any S3-compatible storage device or service to hold iRODS Data Objects, on-premises or in the cloud.

This plugin can work as a standalone "cacheless" resource or as an archive resource under the iRODS compound resource. Either configuration provides a POSIX interface to data held on an object storage device or service.

The following S3 services and appliances (in no particular order) have been tested: