EPA / RENCI

Demo and Discussion

 

Terrell Russell, Ph.D.

@terrellrussell

Senior Data Scientist, iRODS Consortium

May 2, 2016

Renaissance Computing Institute

UNC-Chapel Hill

           Overview

iRODS is open source

data management software.

What is iRODS?

Storage Virtualization: iRODS is Middleware

User Application

"Logical" Layer

Storage Environment

"Physical" Layer

storagecluster.example.org:/managed

s3.amazonaws.com:/example/bitbucket

Storage Virtualization: iRODS Clients

• Web-based and Standalone GUIs

  - iRODS Cloud Browser, MetaLnx, Kanki, Cyberduck

 

• Portals, External Systems

  - iPlant Discovery Environment, Islandora, Fedora Commons

 

• WebDAV for drag-and-drop access built in to the OS

• APIs: Python, REST, Qt, Java, C++

• Command Line Interface

"Logical" Layer

"Physical" Layer

storagecluster.example.org:/managed/Vault/home/alice/training_jpgs/clown_fish.jpg

/tempZone/home/alice/training_jpgs/clown_fish.jpg

s3.amazonaws.com:/example/bitbucket/Vault/home/alice/training_jpgs/clown_fish.jpg

Storage Virtualization: Objects and Collections

Storage Virtualization: iRODS is Distributed

Storage Virtualization: Composable Resources

replResc

demoResc

s3Resc

archiveResc

cacheResc

Data Discovery

Attribute: filename      | Value: seal.jpg

Attribute: animal         | Value: seal

Attribute: photo_color | Value: gray and brown

Attribute: file_size        | Value: 362833                | Units: bytes

acPostProcForPut {
    if ($filePath like "*.jpg" || $filePath like "*.jpeg" || $filePath like "*.bmp" || $filePath like "*.tif" || $filePath like "*.tiff" || $filePath like "*.rif" || $filePath like "*.gif" || $filePath like "*.png" || $filePath like "*.svg" || $filePath like "*.xpm") { 
    msiget_image_meta($filePath, *meta);
    msiString2KeyValPair(*meta, *meta_kvp);
    msiAssociateKeyValuePairsToObj(*meta_kvp, $objPath, "-d"); 
    } # if 
} # acPostProcForPut 

Policy Enforcement Points = Triggers

Microservices = Actions

Workflow Automation: Rules

Secure Collaboration: Federation

Some History

The Team

Static Analysis and Continuous Integration

https://jenkins.irods.org

Initial Trial

  • Google Group
  • Blog posts, social media
  • Cloud images
  • Documentation
  • Training workshops
  • iRODS Hub: The iRODS App Store

Proof of Concept to Pilot

  • Occasional 1-on-1 Support
  • iRODS Consortium Members
  • iRODS Partners
  • iRODS Consortium Service Contracts

Production

  • iRODS Consortium Membership

Building Community and Demonstrating Value

https://irods.org/documentation/

Getting Plugged In with iRODS...

  • Goal is to sustain iRODS as free open source software by:

    ▹ Building good software.  ▹ Growing the iRODS community.  ▹ Demonstrating value.

 

  • Funds a team of 10+ developers, application engineers, documentation, support staff

The iRODS Consortium and Sustainability

Contract Customers

and more ...

  Storage Virtualization

 

 

 

  Data Discovery

 

 

 

  Workflow Automation

 

 

 

  Secure Collaboration

          The Four Pillars

all your storage in a single namespace

← system and user-generated metadata

← event-driven and scheduled cron  policies

 federation

Building Good Software: Plugins

Microservices

• Storage Resources

Authentication

Network

• Database

API

• Rule Engine (iRODS 4.2)

Transport (iRODS 4.3)

Modern, modular, maintainable code

Thank You!

 

 

irods.org

 

 

 

 

Terrell Russell

tgr@renci.org

EPA / RENCI Demo

By iRODS Consortium

EPA / RENCI Demo

An Overview of iRODS and the Consortium.

  • 1,837