Next-Generation WOS Resource Plugin

Jason Coposky, Chief Technologist

Current Architecture

Uses a Compound Resource Hierarchy

• Places a POSIX compliant Resource Plugin in front of the Object Store
• Necessary for several iRODS Features
  • Parallel Transfer
  • Automatic bulk put operations ( iput -r )
  • Structured File Objects: mounted collections, bundled collections
  • Offset writes and reads

A cacheless implementation would need to throw an error on these operations

Intermediary Steps

The existing WOS implementation only supports streaming

• Immediate performance improvements from multi-part transfer 

Improve cache implementation and management

• Faster / Larger caches
• Less aggressive cache management

Potential New Architecture

iRODS parallel transfer requires the ability to write at an offset for N threads     -- some sort of cache is still required

• We can create a new Coordinating Resource Plugin as an In-Memory Cache
• Buffer out-of-order writes
• Leverage the Multi-Part Put
  • drain the cache in sections when enough contiguous data is available ( user defined parameter )
  • register a multipart object when complete
• Possibly cache a part-object(s) for offset write/reads
• Bulk operations are still an issue