Technology Update
June 5-7, 2018
iRODS User Group Meeting 2018
Durham, NC
Terrell Russell, Ph.D.
@terrellrussell
Chief Technologist, iRODS Consortium
Technology Update
In The Last Year
iRODS Release | Issues Closed |
---|---|
4.1.11 | 92 |
4.2.2 | 39 |
4.2.3 | 76 + 18 |
Plugins
Clients
~/irods $ git shortlog --summary --numbered 4.1.10..4-1-stable 31 Ben Keller 20 Alan King 20 Terrell Russell 7 Hao Xu 7 Rick Skarbez 7 jkgill 6 Zoey Greer 5 Jason Coposky 2 Matthew Vernon 1 Brian N. Makin ~/irods $ git shortlog --summary --numbered 4.2.1..4-2-stable 51 Alan King 27 Jason Coposky 22 Hao Xu 18 Terrell Russell 17 Ben Keller 17 Zoey Greer 15 Kory Draughn 9 jkgill 8 Justin James 5 Andrew Kelly 5 Kyle Ferriter 2 d-w-moore 2 xu-hao 1 Felix A. Croes 1 Marco Grossi 1 Sam Morris
In The Last Year
Ongoing Development Work
4.1.12 Update
Replicas in a complex tree (across 2 zones)...
Not enough replicas! #3746
Problem: 1 copy == BAD
Cause(s): Network, storage, OS...
Solution: Try harder
Replication retry (tested with mungefs)
Too many replicas! #3665
Problem: Filling up storage
Cause: Surprise concurrency? Repl resc?
Solution: ???
"Enhanced logging" with PID/user info
root:passthru
└── replicate:replication
└── green-rand:random
│ └── green0:random
│ │ ├── green0ufs0
│ │ └── green0ufs1
│ └── green1:random
│ ├── green1ufs0
│ └── green1ufs1
└── red-rand:random
└── red0:random
│ ├── red0ufs0
│ └── red0ufs1
└── red1:random
├── red1ufs0
└── red1ufs1
To be continued...
4.3.0 Update
4.3.0 Logging Update
Today
Design Goals
Join Us!
iRODS Build and Test - Yesterday
July 2011
Python → Node.js → RabbitMQ → Celery → Eucalyptus
October 2012
January 2013
October 2013
Spring 2015
iRODS Build and Test - Today
Spring 2015 - onwards
Changes Since 2017
Moved iRODS build/test logic from Ansible to per-repository python modules
Unified Jenkins workflow to test all supported plugins
Consolidated to two parameterized Jenkins jobs
run-script-on-vms
iRODS Build and Test - Today
iRODS Build and Test - Future
Testing a distributed system is hard.
Input and other ideas are welcome.
Big Picture
4.2 is about sending things out
4.3 (and friends) will be about bringing them back
Big Picture - GenQuery
Big Picture
Enterprise Messaging - Scatter
Big Picture
Metadata Virtualization - Gather
Big Picture
Open Source Community Engagement