Data Management Design Patterns

Terrell Russell, Ph.D.

@terrellrussell

Chief Technologist, iRODS Consortium

Data Management Design Patterns

November 14-16, 2017

Supercomputing 2017

Denver, CO

Why Data Management Matters

As data matures and reaches a broader community, data management policy must also evolve to meet these additional requirements.

iRODS is

  • Open Source
  • Distributed
  • Data Centric
  • Metadata Driven

 

A flexible framework for the abstraction of infrastructure

iRODS as the Integration Layer

iRODS Build and Test - Today

Spring 2015 - onwards

  • Jenkins → Python → Ansible → zone_bundles → vSphere dynamic VMs

 

Changes Since 2017

  • Centos 6 and Ubuntu 12 no more supported

  • irods build logic moved out of ansible

  • workflow to test all plugins

  • run-script-on-vms

  • run-script-on-irods-zone

History

20+ year legacy

  • 10 years of federal funding for grid storage research
  • 10 years of federal funding for policy engine research
  • iRODS Consortium founded in 2013

Our Membership

Community Driven

Input from the Open Source Community

  • Support Requests
  • Community Feedback
  • Working Groups
  • Use Cases
  • Proofs of Concept

 

All with the Expectation of Public Discourse and Disclosure

Discovered a common enabling practice...

(aka metadata)

Annotation with meaning

Annotation is both descriptive and prescriptive.

 

It is useful

  • for discovery of the past and the present
  • to direct the future

 

Metadata Everywhere

With the appropriate abstractions, everything in a system can be described with metadata and therefore, all actions within a system can be driven by that metadata.

 

Metadata Driven Patterns:

  • Good Metadata (Templates)
  • Landing Zone / Ingest
  • Replication
  • Tiering
  • Archiving
  • Auditing
  • Data to Compute
  • Compute to Data

 

Metadata Templates

iRODS Capabilities

From Prototype to Production

Provenance and Reporting

Data to Compute Pattern

Compute to Data Pattern

An open community-driven process

  • is hard
  • is slow

 

But, it also

  • sets clear expectations
  • generates a shared language
  • produces a strong culture
  • produces a better 'product'
  • is worth it

Discovering Design Patterns

Thank you!

 

 

iRODS Consortium      @iRODS

RENCI                              @RENCI

Booth #437

 

 

Terrell Russell, Ph.D.

@terrellrussell

Chief Technologist, iRODS Consortium