Executive Overview
and
Demonstration
Jason Coposky
@jason_coposky
Executive Director, iRODS Consortium
Executive Overview
and
Demonstration

January 30, 2020
DiRAC-3
Zoom Presentation




Our Membership

























Consortium
Member

Consortium
Member
What is iRODS
Distributed - runs on a laptop, a cluster, on premises or geographically distributed
Open Source - BSD-3 Licensed, install it today and try before you buy
Metadata Driven & Data Centric - Insulate both your users and your data from your infrastructure

iRODS as the Integration Layer


iRODS Core Competencies
The underlying technology categorized into four areas






Data Virtualization

Combine various distributed storage technologies into a Unified Namespace
- Existing file systems
- Cloud storage
- On premises object storage
- Archival storage systems
iRODS provides a logical view into the complex physical representation of your data, distributed geographically, and at scale.


Projection of the Physical into the Logical

Logical Path
Physical Path(s)

Data Discovery
Attach metadata to any first class entity within the iRODS Zone
- Data Objects
- Collections
- Users
- Storage Resources
- The Namespace
iRODS provides automated and user-provided metadata which makes your data and infrastructure more discoverable, operational and valuable.



Metadata Everywhere


Workflow Automation
Integrated scripting language which is triggered by any operation within the framework
- Authentication
- Storage Access
- Database Interaction
- Network Activity
- Extensible RPC API

The iRODS rule engine provides the ability to capture real world policy as computer actionable rules which may allow, deny, or add context to operations within the system.


Dynamic Policy Enforcement

- restrict access
- log for audit and reporting
- provide additional context
- send a notification
The iRODS rule may:

Dynamic Policy Enforcement

A single API call expands to many plugin operations all of which may invoke policy enforcement
- Authentication
- Database
- Storage
- Network
- Rule Engine
- Microservice
- RPC API
Plugin Interfaces:

Secure Collaboration
iRODS allows for collaboration across administrative boundaries after deployment
- No need for common infrastructure
- No need for shared funding
- Affords temporary collaborations
iRODS provides the ability to federate namespaces across organizations without pre-coordinated funding or effort.



iRODS as a Service Interface


Federation - Shared Data and Services


Ingest to Institutional repository
As data matures and reaches a broader community, data management policy must also evolve to meet these additional requirements.


iRODS Policies


- Replication
- Versioning
- Fixity Checking
- Retention
- Verification
- Logical Quotas
- Hard Links
- Metadata Access Control
- Metadata Template Enforcement
- ...

- Packaged and supported solutions
- Require configuration not code
- Derived from the majority of use cases observed in the user community








iRODS Capabilities

Automated Ingest - Landing Zone



Automated Ingest - Filesystem Scanning


Storage Tiering



Core Competencies
Policy
Capabilities
Indexing


Core Competencies
Policy
Capabilities

Publishing

Deployment Patterns



Data to Compute
Compute to Data

Data Transfer Nodes

Filesystem Synchronization

Filesystem Synchronization



Data to Compute



Compute to Data



Data Transfer Nodes



The Data Management Model



Use Cases
iRODS

The Wellcome Sanger Institute


Sanger - Replication

- Data preferentially placed on resource servers in the green data center (fallback to red)
- Data replicated to the other room.
- Checksums applied
- Green and red centers both used for read access.

Sanger - Metadata
attribute: library
attribute: total_reads
attribute: type
attribute: lane
attribute: is_paired_read
attribute: study_accession_number
attribute: library_id
attribute: sample_accession_number
attribute: sample_public_name
attribute: manual_qc
attribute: tag
attribute: sample_common_name
attribute: md5
attribute: tag_index
attribute: study_title
attribute: study_id
attribute: reference
attribute: sample
attribute: target
attribute: sample_id
attribute: id_run
attribute: study
attribute: alignment
- Example metadata attributes
- Users query and access data from local compute clusters
- Users access iRODS locally via the command line interface

Sanger - Federation


Maastricht DataHub
Maastricht DataHub
SURF Scale Out Pilot



University Zone
Catalog



University Zone
Catalog


Server Hosting Environment
Resource Server
Resource Server

Tape Archive
Disk Storage
Object Storage
SURF EUDAT CDI

External Community Zones
Catalog

Zone
Catalog
Local Storage
CXFS

Tape Library



EUDAT University Zone
Catalog



EUDAT University Zone
Catalog
B2SAFE iRODS Federation
EUDAT Centers
iRODS Federation
ARCHIVE
GridFTP Data Movement
Overview
iRODS Proof of Concept

Initial Goals

- Automatically Ingest data from a 'Landing Zone'
- Extract salient metadata - eg EXIF tags
- Tag Data Objects and Collections to make them Actionable and Discoverable
- Discover and interact with data on the command line
- Discover and interact with data via Metalnx
- Share data via Metalnx
- Interact with data via NFS and WebDAV
Automated Ingest

Any data that is discovered during a scan
- Automatically registered to a storage resource
- Metadata extracted and applied to the object in the catalog
- Event possibly generated for audit trail
Users can view and access data and metadata from any client
Data Discovery with Metalnx

Automated Ingest has provided metadata for data discovery
The metadata can be directly inspected in Metalnx
The query builder can be used to identify data sets of interest via Attribute, Value, Unit matches
Queries to the system metadata may also be performed, searching on values such as file name, collection path, user, etc.
File System Presentations: DAVRods

DAVRods provides both a simple web based interface as well as the ability to mount a folder on the desktop
DAVRods is an Apache Module implemented in C using the native iRODS POSIX API
DAVRods can be used to edit data in-place, or to copy data to/from a users collections
File System Presentations: NFSRODS

NFSRODS leverages the Java iRODS Client 'Jargon' and is implemented with NFS4J
NFSRODS acts as a Mid-Tier client to iRODS
NFSRODS projects iRODS ACLs into NFS extended ACLs
NFSRODS can also be used to edit data in-place, or to copy data to/from a users collections with
Data Discovery with Command Line

Query using imeta, a command-line iRODS client utility:
imeta qu -d "Image Make" = Apple
iquest "%s/%s" "SELECT COLL_NAME, DATA_NAME WHERE META_DATA_ATTR_NAME = 'Image Make' AND META_DATA_ATTR_VALUE = 'Apple'"
Query using iquest, a command-line iRODS client utility:
Questions?

DiRAC-3 - iRODS Executive Overview and Demonstration
By jason coposky
DiRAC-3 - iRODS Executive Overview and Demonstration
An executive overview of iRODS, its technology, capabilities and deployment patterns as well as a demonstration of capabilities.
- 1,239