Terrell Russell, Ph.D.

@terrellrussell

Executive Director, iRODS Consortium

Protocol Plumbing:

Presenting iRODS as WebDAV, FUSE, REST, NFS, SFTP, K8s CSI, and S3

Protocol Plumbing:

Presenting iRODS as WebDAV, FUSE, REST, NFS, SFTP, K8s CSI, and S3

November 9, 2022

BioTechX 2022

Basel, Switzerland

Our Membership

Consortium

Member

Consortium

Member

Consortium

Member

Consortium

Member

The iRODS Consortium

Our Mission

  • Continuous Improvement
  • Grow the Community
  • Standardization
  • Show value to our Membership

Our Business Model

Start with proof of concept

  • Use Case Driven
  • Hands on
  • Service and Support Contracts

 

Consortium Membership

  • Four Levels - $10k to $150k
  • 10 to 300 hours of support
  • Participation in Software roadmap
  • Discounted hourly rate

 

Tier 3 Support

  • Systems Integrators
  • Compute Vendors
  • Storage Vendors

History

  • 1995 - SRB started (grid storage)
  • 2004 - iRODS started (added rule engine / policy)
  • 2013 - Consortium founded by RENCI, DICE, and DDN
  • 2014 - Consortium accepted the code base
  • 42 releases of iRODS to date

Why use iRODS?

People need a solution for:

  • Managing large amounts of data across various storage technologies
  • Controlling access to data
  • Searching their data quickly and efficiently
  • Automation

 

The larger the organization, the more they need software like iRODS.

iRODS as the Integration Layer

  • Packaged and supported solutions
  • Require configuration not code
  • Derived from the majority of use cases observed in the user community

iRODS Capabilities

The Data Management Model

Protocol Plumbing

Over the last few years, the ecosystem around the iRODS server has continued to expand.

 

Integration with other types of systems is a valuable way to increase accessibility without teaching existing tools about the iRODS protocol or introducing new tools to users.

 

With some plumbing, existing tools get the benefit of visibility into an iRODS deployment.

Protocol Plumbing

  • WebDAV
  • FUSE
  • REST
  • NFS
  • SFTP
  • K8s CSI
  • S3

Protocol Plumbing - WebDAV

"Davrods provides access to iRODS servers using the WebDAV protocol. It is a bridge between the WebDAV protocol and the iRODS API, implemented as an Apache HTTPD module."

 

Designed, developed, and maintained by

Utrecht University

 

Open Source:

    https://github.com/UtrechtUniversity/davrods

Protocol Plumbing - FUSE

"FUSE implementation of iRODS Client written in Golang"

(2nd generation)

 

Designed, developed, and maintained by

CyVerse at the University of Arizona

 

 

Open Source:

    https://github.com/cyverse/irodsfs

Protocol Plumbing - REST

"This REST API is designed to be deployed in front of an iRODS Server to provide an HTTP REST interface into the iRODS protocol."

(2nd generation)

 

Designed, developed, and maintained by

the iRODS Consortium

 

Open Source:

    https://github.com/irods/irods_client_rest_cpp

Protocol Plumbing - NFS

"A standalone NFSv4.1 server (via nfs4j) with a Virtual File System implementation supporting the iRODS Data Management Platform."

 

Designed, developed, and maintained by

the iRODS Consortium

 

Open Source:

    https://github.com/irods/irods_client_nfsrods

Protocol Plumbing - SFTP

"Support for serving local filesystem, encrypted local filesystem, S3 Compatible Object Storage, Google Cloud Storage, Azure Blob Storage, iRODS Storage or other SFTP accounts over SFTP/SCP/FTP/WebDAV."

 

Designed, developed, and maintained by

CyVerse at the University of Arizona

 

Open Source:

    https://github.com/cyverse/sftpgo

Protocol Plumbing - K8s CSI

"iRODS Container Storage Interface (CSI) Driver implements the CSI Specification to provide container orchestration engines (like Kubernetes) iRODS access."

 

Designed, developed, and maintained by

CyVerse at the University of Arizona

 

Open Source:

    https://github.com/cyverse/irods-csi-driver

Protocol Plumbing - S3

"C++ S3 API for iRODS"

 

Designed, developed, and maintained by

the iRODS Consortium

 

 

Open Source:

    https://github.com/irods/irods_client_s3_cpp

Protocol Plumbing

Lessons Learned

  • shims are never perfect - always some impedance mismatch
  • they are still very valuable and worth the effort
  • maintenance burden can be minimal

 

Future Work

  • continued identification of useful candidate protocols
  • ongoing effort to stabilize existing implementations
  • consideration of the iRODS server offering REST natively

Questions?

Thank you.

Terrell Russell

@terrellrussell

iRODS Consortium

BioTechX 2022 - Protocol Plumbing

By iRODS Consortium

BioTechX 2022 - Protocol Plumbing

  • 419

More from iRODS Consortium