Executive Overview
and
Demonstration
Jason Coposky
@jason_coposky
Executive Director, iRODS Consortium
Executive Overview
and
Demonstration
October 15, 2019
Agriculture Victoria
Melbourne, Australia
What is iRODS
Distributed - runs on a laptop, a cluster, on premises or geographically distributed
Open Source - BSD-3 Licensed, install it today and try before you buy
Metadata Driven & Data Centric - Insulate both your users and your data from your infrastructure
iRODS as the Integration Layer
iRODS Core Competencies
The underlying technology categorized into four areas
Data Virtualization
Combine various distributed storage technologies into a Unified Namespace
iRODS provides a logical view into the complex physical representation of your data, distributed geographically, and at scale.
Projection of the Physical into the Logical
Logical Path
Physical Path(s)
Data Discovery
Attach metadata to any first class entity within the iRODS Zone
iRODS provides automated and user-provided metadata which makes your data and infrastructure more discoverable, operational and valuable.
Metadata Everywhere
Workflow Automation
Integrated scripting language which is triggered by any operation within the framework
The iRODS rule engine provides the ability to capture real world policy as computer actionable rules which may allow, deny, or add context to operations within the system.
Dynamic Policy Enforcement
The iRODS rule may:
Dynamic Policy Enforcement
A single API call expands to many plugin operations all of which may invoke policy enforcement
Plugin Interfaces:
Secure Collaboration
iRODS allows for collaboration across administrative boundaries after deployment
iRODS provides the ability to federate namespaces across organizations without pre-coordinated funding or effort.
iRODS as a Service Interface
Federation - Shared Data and Services
Ingest to Institutional repository
As data matures and reaches a broader community, data management policy must also evolve to meet these additional requirements.
iRODS Capabilities
Automated Ingest - Landing Zone
Automated Ingest - Filesystem Scanning
Storage Tiering
Core Competencies
Policy
Capabilities
Indexing
Core Competencies
Policy
Capabilities
Publishing
Deployment Patterns
Data to Compute
Compute to Data
Filesystem Synchronization
Filesystem Synchronization
Data to Compute
Compute to Data
The Data Management Model
Use Cases
iRODS
The Wellcome Sanger Institute
Sanger - Replication
Sanger - Metadata
attribute: library
attribute: total_reads
attribute: type
attribute: lane
attribute: is_paired_read
attribute: study_accession_number
attribute: library_id
attribute: sample_accession_number
attribute: sample_public_name
attribute: manual_qc
attribute: tag
attribute: sample_common_name
attribute: md5
attribute: tag_index
attribute: study_title
attribute: study_id
attribute: reference
attribute: sample
attribute: target
attribute: sample_id
attribute: id_run
attribute: study
attribute: alignment
Sanger - Federation
Maastricht DataHub
Maastricht DataHub
SURF Scale Out Pilot
University Zone
Catalog
University Zone
Catalog
Server Hosting Environment
Resource Server
Resource Server
Tape Archive
Disk Storage
Object Storage
SURF EUDAT CDI
External Community Zones
Catalog
Zone
Catalog
Local Storage
CXFS
Tape Library
EUDAT University Zone
Catalog
EUDAT University Zone
Catalog
B2SAFE iRODS Federation
EUDAT Centers
iRODS Federation
ARCHIVE
GridFTP Data Movement
Overview
iRODS Proof of Concept
Initial Goals
The Infrastructure
Catalog Service
Provider
Automated Ingest
Service
iRODS
Protocol
DAVRods
iRODS
Protocol
S3 Buckets
Metalnx
iRODS
Protocol
HTTP
HTTP
icommands
iRODS
Protocol
icommands
The Content
Automated Ingest
S3 buckets scanned
Any data that is discovered during a scan
Users can view and access data and metadata from any client
Data Discovery with Metalnx
Automated Ingest has provided metadata for data discovery
The metadata can be directly inspected in Metalnx
The query builder can be used to identify data sets of interest via Attribute, Value, Unit matches
Queries to the system metadata may also be performed, searching on values such as file name, collection path, user, etc.
File System Presentations: DAVRods
DAVRods provides both a simple web based interface as well as the ability to mount a folder on the desktop
DAVRods is an Apache Module implemented in C using the native iRODS POSIX API
DAVRods can be used to edit data in-place, or to copy data to/from a users collections
Data Discovery with Command Line
Query using imeta, a command-line iRODS client utility:
imeta qu -C TYPE = pix4d and EXPER like '%wonwondah%' and YMD '>=' 2018-09 and YMD '<' 2019-01 and STRUCT like '%ortho%mosaic%' | grep -v "\-\-\-\-"
Upcoming Goals
Upcoming Goals
Questions?