SC24 - Building an Approachable Cost-Effective Data Management Platform

Terrell Russell, Ph.D

Executive Director, iRODS Consortium

Director of Data Management, RENCI

November 17-22, 2024

Supercomputing 2024

Atlanta, GA

Building an

Approachable Cost-Effective Data Management Platform

Our Membership

Consortium

Member

Consortium

Member

Consortium

Member

Mission

Long-term data management is best executed when policies are clear and infrastructure is abstracted and swappable.

iRODS has a desire to be normal and boring for the administrator and approachable and powerful for the user.

This talk will cover recent advances and interfaces which allow companies to sustain FAIR data practices, enforce consistency and reproducibility, and realize cost-savings through open source software.

Partners and Users: Past and Present

Supercomputing Centers
Physics
Library / Archives
Genomics
Bio / Pharmaceutical
Hydrology / Weather
Medical
Manufacturing
Shipping / Logistics
Automotive

Working with DDN on the AI Stack

What is iRODS

Open Source

C++ client-server architecture
iRODS Protocol and RPC API
BSD-3 Licensed

Distributed

Runs on a laptop, a cluster, on premises or geographically distributed

Data Centric & Metadata Driven

Insulate both your users and your data from your infrastructure over time

iRODS as the Integration Layer

Why use iRODS?

People need a solution for:

Managing large amounts of data across various storage technologies
Controlling access to data
Searching their data quickly and efficiently
Automation

The larger the organization, the more they need software like iRODS.

Ingest to Institutional Repository

As data matures and reaches a broader community, data management policy must also evolve to meet these additional requirements.

Data Virtualization

Combine various distributed storage technologies into a Unified Namespace

Existing file systems
Cloud storage
On premises object storage
Archival storage systems

iRODS provides a logical view into the complex physical representation of your data, distributed geographically, and at scale.

Data Discovery

Attach metadata to any first class entity within the iRODS Zone

Data Objects
Collections
Users
Storage Resources
The Namespace

iRODS supports automated and user-provided metadata which makes your data and infrastructure more discoverable, operational, and valuable.

Workflow Automation

Policy Enforcement Points (PEPs) are triggered by every operation within the framework

Authentication
Storage Access
Database Interaction
Network Activity
Extensible RPC API

The iRODS rule engine framework provides the ability to capture real world policy as computer actionable rules which may allow, deny, or add context to operations within the system.

Dynamic Policy Enforcement

The iRODS rule may:

restrict access
log for audit and reporting
provide additional context
send a notification

Dynamic Policy Enforcement

A single API call expands to many plugin operations all of which may invoke policy enforcement.

Plugin Interfaces:

Authentication
Database
Storage
Network
Rule Engine
Microservice
RPC API

Secure Collaboration

iRODS allows for collaboration across administrative boundaries after deployment

No need for common infrastructure
No need for shared funding
Affords temporary collaborations

iRODS provides the ability to federate namespaces across organizations without pre-coordinated funding or effort.

The Data Management Model

Protocol Plumbing - Presenting iRODS as other Protocols

WebDAV
FUSE
HTTP
NFS
SFTP
K8s CSI
S3

Over the last few years, the ecosystem around the iRODS server has continued to expand.

Integration with other types of systems is a valuable way to increase accessibility without teaching existing tools about the iRODS protocol or introducing new tools to users.

With some plumbing, existing tools get the benefit of visibility into an iRODS deployment.

Future

Towards Cloud-Native processes and bookkeeping
Vertical Integrations in various domains
Timeseries Data / Statistics
Dashboarding
- Visibility
- Costs

Thank you

Terrell Russell

RENCI Booth #3923

iRODS Clients

https://irods.org/clients

iRODS S3 Functionality

https://github.com/irods/irods_resource_plugin_s3

The iRODS S3 storage resource plugin allows iRODS to use any S3-compatible storage device or service to hold iRODS Data Objects, on-premises or in the cloud.

This plugin can work as a standalone "cacheless" resource or as an archive resource under the iRODS compound resource. Either configuration provides a POSIX interface to data held on an object storage device or service.

The following S3 services and appliances (in no particular order) have been tested: