Technology Update

June 5-7, 2018

iRODS User Group Meeting 2018

Durham, NC

Terrell Russell, Ph.D.

@terrellrussell

Chief Technologist, iRODS Consortium

Technology Update

In The Last Year

Plugins

• Python Rule Engine Plugin
• Storage Tiering Rule Engine Plugin
• Auditing (AMQP) Rule Engine Plugin
• S3 Resource Plugin
• GSI Authentication Plugin
• Kerberos Authentication Plugin
• Curl Microservice Plugin

Clients

• Python iRODS Client
• Automated Ingest Framework
• Cloud Browser

~/irods $ git shortlog --summary --numbered 4.1.10..4-1-stable
    31  Ben Keller
    20  Alan King
    20  Terrell Russell
     7  Hao Xu
     7  Rick Skarbez
     7  jkgill
     6  Zoey Greer
     5  Jason Coposky
     2  Matthew Vernon
     1  Brian N. Makin


~/irods $ git shortlog --summary --numbered 4.2.1..4-2-stable
    51  Alan King
    27  Jason Coposky
    22  Hao Xu
    18  Terrell Russell
    17  Ben Keller
    17  Zoey Greer
    15  Kory Draughn
     9  jkgill
     8  Justin James
     5  Andrew Kelly
     5  Kyle Ferriter
     2  d-w-moore
     2  xu-hao
     1  Felix A. Croes
     1  Marco Grossi
     1  Sam Morris

In The Last Year

Ongoing Development Work

• iRODS 4.1.12
• iRODS 4.2.4
• iRODS 4.3.0
• Python iRODS Client (PRC)
• Indexing Framework
• Lustre Integration
• Metalnx
• CockroachDB Database Plugin
• Ceph RADOS Resource Plugin
• QueryArrow Database Plugin
• R Client Library (R-iRODS)
• Multipart Transfer, v5 API
• NFS4J Integration
• Metadata Templates
• Testing Infrastructure

4.1.12 Update

Replicas in a complex tree (across 2 zones)...

Not enough replicas! #3746

    Problem: 1 copy == BAD

    Cause(s): Network, storage, OS...

    Solution: Try harder

    Replication retry (tested with mungefs)

Too many replicas! #3665

    Problem: Filling up storage

    Cause: Surprise concurrency? Repl resc?

    Solution: ???

    "Enhanced logging" with PID/user info

root:passthru
└── replicate:replication
    └── green-rand:random
    │   └── green0:random
    │   │   ├── green0ufs0
    │   │   └── green0ufs1
    │   └── green1:random
    │       ├── green1ufs0
    │       └── green1ufs1
    └── red-rand:random
        └── red0:random
        │   ├── red0ufs0
        │   └── red0ufs1
        └── red1:random
            ├── red1ufs0
            └── red1ufs1

To be continued...

4.3.0 Update

• irodsMonitor process
• Checksums moving down into resource plugins
• Containerize installation
• Logging overhaul

4.3.0 Logging Update

Today

• Inconsistent
• Incomplete (syslog support)
• Not very helpful in tracking what caused an error
• Not very helpful when multiple servers are involved

 

Design Goals

• Leverage existing logging library (spdlog, etc.)
• Enable admins to easily capture, process, and analyze activity
• Consistent
• Easily track errors across multiple servers (hostname, timestamp, PID, plugin, etc.)
• Tie in existing infrastructure
• Provide more options for controlling output

 

Join Us!

• What would you like to see accomplished with the logging refactor?
• RFC: https://github.com/irods/irods_rfcs/blob/master/0001_logging.md
• Have questions/suggestions? Let's talk!

iRODS Build and Test - Yesterday

July 2011

• Python → Node.js → RabbitMQ → Celery → Eucalyptus

October 2012

• Python → Node.js → ssh → OpenStack

January 2013

• Hudson → Python → OpenStack

October 2013

• Hudson → Python → vSphere long-running VMs

Spring 2015

• Jenkins → Python → Ansible → vSphere dynamic VMs

iRODS Build and Test - Today

Spring 2015 - onwards

• Jenkins → Python → Ansible → zone_bundles → vSphere dynamic VMs

 

Changes Since 2017

• Ended support for CentOS 6 and Ubuntu 12

• Moved iRODS build/test logic from Ansible to per-repository python modules

• Unified Jenkins workflow to test all supported plugins

• Consolidated to two parameterized Jenkins jobs

  • run-script-on-vms

  • run-script-on-irods-zone

iRODS Build and Test - Today

iRODS Build and Test - Future

• Increase coverage                                         (more plugins in CI)
• Move pipeline scripts to GitHub                 (no logic in Jenkins)
• Address inconsistency                                  (false reds / pyvmomi errors)
• Containerize Jenkins                                       (easier to test / update / redeploy)
• Possibly move from VMs to containers     (speed / fewer moving parts)

 

 

Testing a distributed system is hard.

 

Input and other ideas are welcome.

Big Picture

4.2 is about sending things out

• Dynamic PEPs
• Pluggable Rule Engine
• Enterprise Messaging

 

 

4.3 (and friends) will be about bringing them back

• Facilitated by GQv2

Big Picture - GenQuery

Big Picture

Enterprise Messaging - Scatter

Big Picture

Metadata Virtualization - Gather

Big Picture

Open Source Community Engagement

• Technology Working Group
• Metadata Templates Working Group
• GitHub Issues
• Pull Requests
• Chat List
• Consortium Membership