iRODS Overview

 

February 2016

Introduction

Dan Bedard

Director, iRODS Consortium

danb@renci.org

+1-919-445-0632

 

Why are we here?

Agenda

• How iRODS Is Used

 

• How It Does It

 

• Where It's Going

 

• Getting Started

 

• Discussion

 

 

How iRODS Is Used

          Policy-Based Data Management

The Integrated Rule-Oriented Data System:

• Open Source

• Manages data for indefinite lifespan

• Metadata and event-triggered rules

 

Designed to work with big, important, and/or complex data

 

Example applications:

   • Combining data from multiple sources

   • Annotation and search

   • Tiered storage

   • Preservation

   • Sharing in place

 

 

 

Data

Virtualization

Data

Discovery

Workflow

Automation

Secure

Collaboration

User Profile:

National Institute of Environmental Health Sciences

• Viral Vector Core creates designer viruses:

    request⟶transfection and amplification⟶sample delivery⟶reports

 

• Uses iRODS to combine, organize, and analyze sets of requests and instrument results

   • Produces packaged results in response to researcher requests

   • Quarterly cost reports for chargeback and trend analysis for quality control

User Profile: Wellcome Trust Sanger Institute

• Key genomics research centre

• 7 PB of storage managed by iRODS

Rich Metadata

 

attribute: library

attribute: total_reads

attribute: type

attribute: lane

attribute: is_paired_read

attribute: study_accession_number

attribute: library_id

attribute: sample_accession_number

attribute: sample_public_name

attribute: manual_qc

attribute: tag

attribute: sample_common_name

attribute: md5

 

Replication and Federation

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

User Profile: NASA Atmospheric Science Data Center

• 2 PB of archived satellite data

• Publicly available, subsetting on demand

• In-house ingest and archiving software: ANGe (Archive Next Generation)

User Profile: NASA Atmospheric Science Data Center

Federation

 

 

 

 

 

 

 

 

Virtual Collections

 

ls –l

/CER_100100.2012053100
/CER_100100.2012053100.met
/CER_100100.2012053101
/CER_100100.2012053101.met
/CER_100100.2012053102
/CER_100100.2012053102.met

Visibility determined by "visibility attribute"

Logical collection of files spread across physical storage resources.

Single Interface to Multiple Clients

 

WebDAV, FUSE, Web UI, Cyberduck

REST, Python, R, Java C++

(And more!)

User Profile: University College London

• Repository for research data that spans social science, physics, and genomics

• UK sponsored research requirements: last date of access request plus 10 years

• iRODS spans storage technologies and enables federated access from other centres

How iRODS Does It

  Data Virtualization

 

 

 

  Data Discovery

 

 

 

  Workflow Automation

 

 

 

  Secure Collaboration

          The Four Pillars

← all your storage in a single namespace

← system and user-generated metadata

← event-driven and scheduled cron  policies

 federation

iRODS Clients

• Web-based and Standalone GUIs

  - iRODS Cloud Browser, MetaLnx, Kanki, Cyberduck

 

• Portals, External Systems

  - iPlant Discovery Environment, Islandora, Fedora Commons

 

• WebDAV for drag-and-drop access built in to the OS

• APIs: Python, REST, Qt, Java, C++

• Command Line Interface

iRODS is free, open source software owned by a foundation called the iRODS Consortium.

  • Goal is to sustain iRODS as free open source software by:

    ▹ Building good software.  ▹ Growing the iRODS community.  ▹ Demonstrating value.

 

  • Funds a team of 10+ developers, application engineers, documentation, support staff

The iRODS Consortium and Sustainability

Contract Customers

and more ...

Meet the Team

Where We're Going

In Parallel

• Reference implementations, plugins, clients

 

iRODS 4.2

• Pluggable Rule Engine: Automation in Python, any language; provability

• Cleaner install and configuration

• Continued advancement of enterprise message bus

 

iRODS 4.3

• Universal Metadata Access (GQV2)

• Next Generation API

• Pluggable Transport

 

iRODS 5

• Advanced Zone Management

• Plugin Registry

 

Getting Started

User Group Meeting

June 7-9, 2016

Chapel Hill, NC

ugm2016.irods.org

Initial Trial

  • Google Group
  • Blog posts, social media
  • Cloud images
  • Documentation
  • Training workshops
  • iRODS Hub: The iRODS App Store

Proof of Concept to Pilot

  • Occasional 1-on-1 Support
  • iRODS Consortium Members
  • iRODS Partners
  • iRODS Consortium Service Contracts

Production

  • iRODS Consortium Membership

Building Community and Demonstrating Value

https://irods.org/documentation/

Getting Plugged In with iRODS...

Discussion

Thank You!

iRODS Overview

By iRODS Consortium

iRODS Overview

Meet the iRODS team. Learn how we're talking about iRODS. Learn what we're doing. Learn where we're going.

  • 1,799