June 25-28, 2019

iRODS User Group Meeting 2019

Utrecht, Netherlands

Justin James

Applications Engineer

iRODS Consortium

Administration

Resource Hierarchies

and Composition

Administration

Resource Hierarchies

and Composition

iRODS Resource Plugins

• Internally defines the interface to all storage technologies
• Loaded dynamically at runtime
• Uses a vote to advertise the ability to satisfy a given operation
• Maintains individual configuration per instance in the catalog via a 'context string'
• May exist independently or can be wired together into hierarchies

Motivation

Many iRODS users spent considerable time implementing the same basic use cases as policy in their rule base

 

• Replication
• Data distribution
• Replica synchronization
• Data archival

 

Resource hierarchies provide an out of the box means to implement the majority of the use cases, while remaining future-proof

Introduction to Resource Hierarchies

Capture the implementation of various policies as nodes in a decision tree, a well known metaphor

 

Two types of nodes:

• Coordinating - pure policy implementation
• Storage Resource - manages the interface to storage technologies

By convention Coordinating Resources do not manage storage

  - this is not enforced

Coordinating Resources - Branches

Compound - provide POSIX interface to alternative storage

Deferred - defer to children regarding voting behavior

Load Balanced - use gathered load values to determine choices

Passthru - weight, then delegate operations to a child resource

Random - randomly choose a child for a write operation

Replication - ensure all data objects are consistent across children

 

Purely virtual in memory - they are not pinned to any given server and plugins must exist on every server in the grid

Storage Resources - Leaves

Storage resources which provide POSIX semantics

   - do not require a compound resource hierarchy and a cache

 

Unix File System - surfaces any mount point

Ceph-RADOS - Ceph object storage

HPSS - access to IBM High Performance Storage System

Cacheless S3 - resource for Amazon S3 service (soon to be released)

 

Usually pinned to a given server by hostname, as they are expected to access storage provided by the server - plugins do not need to be available on every server.

Exception:  The soon to be released cacheless S3 resource implements a detached mode which is not pinned to a server.

Archive Storage Resources - Leaves

Storage Resources which participate in the role as an Archive Resource in a Compound Resource Composition

 

S3 - archive resource for Amazon S3 

WOS - DDN Web Object Scalar

Universal MSS - script based access to generic object storage

 

Must be pinned to the servers which host the cache in order to synchronize data to the archive resource

The Voting Mechanism

• Votes are by convention between 0.0 and 1.0
• All communication starts at the root of a hierarchy
  • addressing children is disallowed
• Voting follows a depth-first recursive descent
  • coordinating nodes delegate votes to children
• Storage Nodes (Leaves) vote
• Coordinating Nodes (Branches) interpret the results of its children
  • The interpretation of these votes expresses the policy encoded in Coordinating Resource plugins

Voting - Weighted Passthru

For both Put and Get operations

• Delegate the vote to the child
• Multiply the result by the weight
• Pass the result to the calling resource

 

This resource has many uses, such as disabling writes or reads to a given resource, or providing an abstraction to the users

 

The weights also may be overridden by the rule engine which allows for dynamically influencing votes based on policy

Voting - Unix File System Resource

For a Put operation

• If the resource is marked Down, vote 0.0
• If the client is connected to the server which owns the target resource, vote 1.0
• Otherwise vote 0.5

Voting - Unix File System Resource

For a Get operation

• If the resource is marked Down, vote 0.0
• If the client is connected to the server which owns the target resource, and the resource has an up-to-date replica, vote 1.0
• If the resource has an up-to-date replica, vote 0.5
• If the resource has a stale replica, vote 0.25
• Otherwise, vote 0.0

Voting - Random Resource

For a Put Operation

• Randomly select a child and delegate the vote to the child, continue until a positive non-zero vote is achieved or all children are queried
• Pass the result up to the calling resource

Voting - Random Resource

For a Get Operation

• Delegate the vote to all children
• Select the highest vote
• Pass the result up to the calling resource

Voting - Replication Resource

For a Put Operation

• Delegate voting to all children
• Select highest vote
• Pass result to calling resource
• Once Put is complete, trigger replication to all other children who accept a Put Operation

Voting - Replication Resource

For a Get Operation

• Delegate vote to all children
• Select the highest vote
• Pass result to calling resource

 

Note that given the behavior of the Unix File System, locality of reference significantly affects the behavior of reads and writes for this Resource

Voting - A Put Example

pt1 - passthru

pt2 - passthru

pt3 - passthru

rnd1 - random

rnd2 - random

repl1 - replication

ufsN

ufs0

Voting - A Put Example

Votes 1.0 Connected

Votes 0.5

Not Connected

Votes 0.0 Marked Down

Randomly chooses ufs0 for vote of 1.0

Passes ufsN-1 vote of 0.5

Write weight of 0.25, passes ufs0 as a vote of 0.25

Randomly chooses ufsN-1 vote of 0.5

Chooses ufsN-1 with a vote of 0.5 > 0.25

Passes ufsN-1

Compound Resources

Necessary for POSIX compliance of Tape, Object, etc.

 

For a Put

• Data objects are written to the cache first
• Then a replica is made to the archive

 

For a Get

• Replica is staged to cache, if required
• Reads always are made from the cache

 

By default, the compound resource

synchronously replicates to the archive

 

Compound Resources - An S3 Example

iadmin mkresc resc_name type context_string

iadmin mkresc comp_resc compound
iadmin mkresc cache_resc unixfilesystem `hostname`:/tmp/cache_resc

iadmin addchildtoresc parent_name child_name parent_child_context_string

iadmin addchildtoresc comp_resc cache_resc cache

The compound resource plugin honors two parent-child context values: "cache", and "archive"

 

These are used internally to identify the resources by role

Compound Resources - An S3 Example

iadmin mkresc arch_resc s3 `hostname`:/bucket/name <context_string>

• S3_DEFAULT_HOSTNAME - used to define regions for your S3 bucket
• S3_AUTH_FILE - fully qualified path to a file holding the access id and the access key for the given bucket name
• S3_RETRY_COUNT - number of retries before failure
• S3_WAIT_TIME_SEC - wait time before retries in seconds
• S3_PROTO - use either HTTP or HTTPS

iadmin mkresc arch_resc s3 `hostname`:/bucket/name "S3_DEFAULT_HOSTNAME=s3.amazonaws.com;S3_AUTH_FILE=/etc/irods/auth_file;S3_RETRY_COUNT=3;S3_WAIT_TIME_SEC=1;S3_PROTO=HTTP"

iadmin addchildtoresc comp_resc arch_resc archive

Add the archive resource to the compound resource

Compound Resources - Delayed Replication

Set the compound resource context string to "auto_repl=off"

iadmin modresc comp_resc context "auto_repl=off"

Leverage the rule engine for replication via acPostProcForPut

acPostProcForPut() {
    if("ufs_cache" == $KVPairs.rescName) {
        delay("<PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>") {
            *CacheRescName = "comp_resc;ufs_cache";
            msisync_to_archive("*CacheRescName", $filePath, $objPath);
        }
    }
}

Excellent examples of delayed replication and cache purging can be found here:

https://github.com/trel/irods-compound-resource/blob/master/rules/SaraRules.re

Demonstration

http://slides.com/irods/ugm2017-composable-resources#/5