Next-Generation WOS Resource Plugin

Jason Coposky, Chief Technologist

Current Architecture

Uses a Compound Resource Hierarchy

  • Places a POSIX compliant Resource Plugin in front of the Object Store
  • Necessary for several iRODS Features
    • Parallel Transfer
    • Automatic bulk put operations ( iput -r )
    • Structured File Objects: mounted collections, bundled collections
    • Offset writes and reads

A cacheless implementation would need to throw an error on these operations

Intermediary Steps

The existing WOS implementation only supports streaming

  • Immediate performance improvements from multi-part transfer 

 

Improve cache implementation and management

  • Faster / Larger caches
  • Less aggressive cache management

Potential New Architecture

iRODS parallel transfer requires the ability to write at an offset for N threads     -- some sort of cache is still required

  • We can create a new Coordinating Resource Plugin as an In-Memory Cache
  • Buffer out-of-order writes
  • Leverage the Multi-Part Put
    • drain the cache in sections when enough contiguous data is available ( user defined parameter )
    • register a multipart object when complete
  • Possibly cache a part-object(s) for offset write/reads
  • Bulk operations are still an issue

 

 

Next-Generation WOS Resource Plugin

By jason coposky

Next-Generation WOS Resource Plugin

Short overview of current WOS plugin and possible development direction

  • 1,114
Loading comments...

More from jason coposky