SC19 - iRODS Workshop

Terrell Russell, Ph.D.

@terrellrussell

Chief Technologist, iRODS Consortium

SC19 - iRODS Workshop

November 18, 2019

Supercomputing 2019

Denver, CO

Our Membership

Consortium

Member

Consortium

Member

iRODS is

• Open Source
• Distributed
• Data Centric
• Metadata Driven

A flexible framework for the abstraction of infrastructure

iRODS as the Integration Layer

In The Last Year

Plugins

• Python Rule Engine Plugin
• Storage Tiering Rule Engine Plugin
• Auditing (AMQP) Rule Engine Plugin
• Update Collection Mtime Rule Engine Plugin
• S3 Resource Plugin
• GSI Authentication Plugin
• Kerberos Authentication Plugin
• Curl Microservice Plugin

Clients

• Python iRODS Client
• Cloud Browser
• Metalnx
• NFSRODS
• Automated Ingest Framework

Core Update

• New Libraries:

  • iRODS Query Iterator

  • iRODS Thread Pool

  • iRODS Connection Pool

  • iRODS Filesystem (experimental)

  • iRODS IOStreams (experimental)

  • iRODS Query Processor

• Refactored Delay Server

  • Rebuilt using new libraries

  • Single-Producer/Multi-Consumer

  • irodsReServer -> irodsDelayServer

• Intermediate Replicas

  • Stale replicas will now be indicated with 'X'

  • Unifying replication codepaths

• Dockerized Build and Test
  • Jenkins
  • Launches sibling containers

Ongoing Development Work

• iRODS 4.2.7
• iRODS 4.3.0
• Automated Ingest Capability
• Storage Tiering Capability
• Indexing Capability
• Publishing Capability
• Python iRODS Client (PRC)
• Metalnx
• NFSRODS
• Lustre Integration
• NetCDF Extraction
• Ceph RADOS Resource Plugin
• Cacheless S3 Resource Plugin
• Multipart Transfer, v5 API
• Testing Infrastructure

Questions?

• Today's Workshop
  • Beyond Discoverability: Metadata to drive your data management
  • iRODS S3 Resource Plugin: Cacheless and Detached Mode
  • NFSRODS: Presenting iRODS as NFSv4.1