Daniel Moore

Applications Engineer

iRODS Consortium

June 8-11, 2021

iRODS User Group Meeting 2021

Virtual Event

Python iRODS Client 1.0

Python iRODS Client 1.0

Python iRODS Client Library 1.0 - An 8-Year Journey

The original implementation was by Chris LaRose and Michael Gatto at CyVerse (then known as the iPlant Collaborative) in 2013.

 

The library was contributed in mid-2014 to the newly established iRODS Consortium and was taken over by Antoine de Torcy.  Antoine brought the library up to PyPI standards with modern packaging and maintained it through v0.7.0.

 

Daniel Moore then took the reins in 2018 and has shepherded the code from v0.8.0 to today.

 

Thank you to the 41 contributors and many users over the years.

Python iRODS Client Library 1.0 - Major Updates

Atomic Metadata

  • allows a batch of add/remove operations within one transaction 
  • increased efficiency over individual operations
    • demonstrated up to 750x speedup for large batches

 

Parallel Transfer

  • new client-controlled multi-connect with iRODS release 4.2.9+
  • data object put() and get() spawn threads for transfer
  • increased efficiency over single-threaded put/get operations
    • demonstrated 3-4x speedup

Python iRODS Client Library 1.0 - Other Improvements

 

XML Protocol Hardening

  • BinBytesBuf for JSON API payloads in iRODS 4.2.9
    • corrects cases like AVU searches containing '<', '>', '&'
  • Remaining work includes handling control characters in object names

 

New Checksum Functionality

>>> from irods.test.helpers import make_session
>>> session = make_session()
>>> d = session.data_objects.create("/some/object")
>>> d.chksum()
'sha2:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU='

Python iRODS Client Library 1.0 - Other Improvements

We can now:

  • list current ACLs, same as ils -A
  • login via PAM/SSL using client environment configuration
  • configure connection timeouts
  • issue temporary passwords
  • manage session cleanup
  • query rules/tasks in the delay queue
  • automate Python client testing in a Jenkins CI workflow

Python iRODS Client Library - Future Work

  • fix known bugs
  • prepare to drop support for Python2
  • continue working towards full coverage of the iRODS API

Questions?