Advanced Training:
Resource Hierarchies
and Composition
June 13-16, 2023
iRODS User Group Meeting 2023
Chapel Hill, NC
Alan King, Senior Software Developer
Justin James, Senior Applications Engineer
iRODS Consortium
iRODS Resource Plugins
- Internally defines the interface to all storage technologies
- Loaded dynamically at runtime
- Uses a vote to advertise the ability to satisfy a given operation
- Maintains individual configuration per instance in the catalog via a 'context string'
- May exist independently or can be wired together into hierarchies
Motivation
Many iRODS users spent considerable time implementing the same basic use cases as policy in their rule base
- Replication
- Data distribution
- Replica synchronization
- Data archival
Resource hierarchies provide an out of the box means to implement the majority of the use cases, while remaining future-proof
Introduction to Resource Hierarchies
Capture the implementation of various policies as nodes in a decision tree, a well known metaphor
Two types of nodes:
- Coordinating - pure policy implementation
- Storage Resource - manages the interface to storage technologies
By convention Coordinating Resources do not manage storage
- this is not enforced
Coordinating Resources - Branches
Compound - provide POSIX interface to alternative storage
Deferred - defer to children regarding voting behavior
Load Balanced - use gathered load values to determine choices
Passthru - weight, then delegate operations to a child resource
Random - randomly choose a child for a write operation
Replication - ensure all data objects are consistent across children
Purely virtual in memory - they are not pinned to any given server and plugins must exist on every server in the grid
Storage Resources - Leaves
Storage resources which provide POSIX semantics
- do not require a compound resource hierarchy and a cache
Unix File System - surfaces any mount point
Cacheless S3 - resource for S3-compatible storage service
Usually pinned to a given server by hostname, as they are expected to access storage provided by the server - plugins do not need to be available on every server.
The cacheless S3 resource and Unix File System resources implement a detached mode, which is not pinned to a server.
Archive Storage Resources - Leaves
Storage Resources which participate in the role as an Archive Resource in a Compound Resource Composition
S3 - archive resource for S3-compatible storage
Universal MSS - script based access to generic mass storage (tape)
Must be pinned to the servers which host the cache in order to synchronize data to the archive resource
The Voting Mechanism
- Votes are by convention between 0.0 and 1.0
- All communication starts at the root of a hierarchy
- addressing children is disallowed
- Voting follows a depth-first recursive descent
- coordinating nodes delegate votes to children
- Storage Nodes (Leaves) vote
- Coordinating Nodes (Branches) interpret the results of its children
- The interpretation of these votes expresses the policy encoded in Coordinating Resource plugins
Voting - Weighted Passthru
For both Put and Get operations
- Delegate the vote to the child
- Multiply the result by the weight
- Pass the result to the calling resource
This resource has many uses, such as disabling writes or reads to a given resource, or providing an abstraction to the users
The weights also may be overridden by the rule engine which allows for dynamically influencing votes based on policy
Voting - Unix File System Resource
For a Put operation
- If the resource is marked Down, vote 0.0
- If the client is connected to the server which owns the target resource, vote 1.0
- Otherwise vote 0.5
Voting - Unix File System Resource
For a Get operation
- If the resource is marked Down, vote 0.0
- If the client is connected to the server which owns the target resource, and the resource has an up-to-date replica, vote 1.0
- If the resource has an up-to-date replica, vote 0.5
- If the resource has a stale replica, vote 0.25
- Otherwise, vote 0.0
Voting - Random Resource
For a Put Operation
- Randomly select a child and delegate the vote to the child
- Continue until a positive non-zero vote is achieved or all children are queried
- Pass the result up to the calling resource
Voting - Random Resource
For a Get Operation
- Delegate the vote to all children
- Select the highest vote
- Pass the result up to the calling resource
Voting - Replication Resource
For a Put Operation
- Delegate voting to all children
- Select highest vote
- Pass result to calling resource
- Once Put is complete, trigger replication to all other children that accept a Put Operation
Voting - Replication Resource
For a Get Operation
- Delegate vote to all children
- Select the highest vote
- Pass result to calling resource
Note that given the behavior of the unixfilesystem, locality of reference significantly affects the behavior of reads and writes for this Resource
Voting - A Put Example
pt1 - passthru
pt2 - passthru
pt3 - passthru
rnd1 - random
rnd2 - random
repl1 - replication
ufsN
ufs0
Voting - A Put Example
Votes 1.0 Connected
Votes 0.5
Not Connected
Votes 0.0 Marked Down
Randomly chooses ufs0 for vote of 1.0
Passes ufsN-1 vote of 0.5
Write weight of 0.25, passes ufs0 as a vote of 0.25
Randomly chooses ufsN-1 vote of 0.5
Chooses ufsN-1 with a vote of 0.5 > 0.25
Passes ufsN-1
Compound Resources
Necessary for POSIX compliance of Tape, Object, etc.
For a Put
- Data objects are written to the cache first
- Then a replica is made to the archive
For a Get
- Replica is staged to cache, if required
- Reads are always made from the cache
By default, the compound resource
synchronously replicates to the archive
Compound Resources - An S3 Example
iadmin mkresc resc_name type context_string
iadmin mkresc comp_resc compound iadmin mkresc cache_resc unixfilesystem $(hostname):/tmp/cache_resc
iadmin addchildtoresc parent_name child_name parent_child_context_string
iadmin addchildtoresc comp_resc cache_resc cache
The compound resource plugin honors two parent-child context values: "cache", and "archive"
These are used internally to identify the resources by role
Compound Resources - An S3 Example
iadmin mkresc arch_resc s3 $(hostname):/bucket/name <context_string>
- S3_DEFAULT_HOSTNAME - used to define regions for your S3 bucket
- S3_AUTH_FILE - absolute path to file holding the access id and access key for the given bucket
- S3_RETRY_COUNT - number of retries before failure
- S3_WAIT_TIME_SEC - wait time before retries in seconds
- S3_PROTO - use either HTTP or HTTPS
iadmin mkresc arch_resc s3 $(hostname):/bucket/name "S3_DEFAULT_HOSTNAME=s3.amazonaws.com;S3_AUTH_FILE=/etc/irods/auth_file;S3_RETRY_COUNT=3;S3_WAIT_TIME_SEC=1;S3_PROTO=HTTP"
iadmin addchildtoresc comp_resc arch_resc archive
Add the archive resource to the compound resource
Building a Compound Resource
iadmin mkresc comp_resc compound
iadmin mkresc ufs_cache unixfilesystem $(hostname):/tmp/irods/ufs_cache
iadmin mkresc ufs_arch unixfilesystem $(hostname):/tmp/irods/ufs_arch
iadmin mkresc resc_name resc_type context_string
iadmin addchildtoresc parent_name child_name context
iadmin addchildtoresc comp_resc ufs_cache cache
iadmin addchildtoresc comp_resc ufs_arch archive
Review Compound Resource Configuration
irods@example:~$ ilsresc
comp_resc:compound
├── ufs_arch
└── ufs_cache
demoResc
irods@example:~$ ilsresc -l comp_resc resource name: comp_resc id: 10001 zone: tempZone type: compound location: EMPTY_RESC_HOST vault: EMPTY_RESC_PATH free space: free space time: : Never status: info: comment: create time: 01685984596: 2023-06-05.17:03:16 modify time: 01685984596: 2023-06-05.17:03:16 context: parent: parent context:
Review the child resources
irods@example:~$ ilsresc -l ufs_cache resource name: ufs_cache id: 10017 zone: tempZone type: unixfilesystem location: example vault: /tmp/irods/ufs_cache free space: free space time: : Never status: info: comment: create time: 01685984596: 2023-06-05.17:03:16 modify time: 01685984602: 2023-06-05.17:03:22 context: parent: 10001 parent context: cache
irods@example:~$ ilsresc -l ufs_arch resource name: ufs_arch id: 10018 zone: tempZone type: unixfilesystem location: example vault: /tmp/irods/ufs_arch free space: free space time: : Never status: info: comment: create time: 01685984597: 2023-06-05.17:03:17 modify time: 01685984602: 2023-06-05.17:03:22 context: parent: 10001 parent context: archive
Test Put
irods@example:~$ truncate --size 10M test_file
irods@example:~$ ls -l test_file
-rw-rw-r-- 1 irods irods 10485760 Jun 5 17:05 test_file
irods@example:~$ iput -R comp_resc test_file
irods@example:~$ ils -l
/tempZone/home/rods:
rods 0 comp_resc;ufs_cache 10485760 2023-06-05.18:18 & test_file
rods 1 comp_resc;ufs_arch 10485760 2023-06-05.18:18 & test_fileBy default, the archive is immediately replicated after the cache replica is at rest and registered in the catalog
ufs_cache has replica number 0 (written first)
ufs_arch has replica number 1 (written second)
Delayed Replication to an Archive Resource
Set the compound resource context string to "auto_repl=off"
iadmin modresc comp_resc context "auto_repl=off"
Leverage the rule engine for replication via pep_api_data_obj_put_post
pep_api_data_obj_put_post(*INSTANCE_NAME, *COMM, *DATAOBJINP, *BUFFER, *PORTAL_OPR_OUT) {
*cache_resc_hier = "comp_resc;ufs_cache";
*resc_hier = *DATAOBJINP.resc_hier;
if("*cache_resc_hier" == "*resc_hier") {
delay("<PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>") {
*unused_param = "";
*obj_path = *DATAOBJINP.obj_path;
msisync_to_archive("*cache_resc_hier", "*unused_param", "*obj_path");
}
}
}
Prepping for delayed replication
"rule_engines": [
{
"instance_name": "irods_rule_engine_plugin-irods_rule_language-instance",
"plugin_name": "irods_rule_engine_plugin-irods_rule_language",
"plugin_specific_configuration": {
"re_data_variable_mapping_set": [
"core"
],
"re_function_name_mapping_set": [
"core"
],
"re_rulebase_set": [
"training",
"core"
],
Add a custom rulebase to /etc/irods/server_config.json
Create a new Rulebase
Edit /etc/irods/training.re and add our new Policy Enforcement Point
pep_api_data_obj_put_post(*INSTANCE_NAME, *COMM, *DATAOBJINP, *BUFFER, *PORTAL_OPR_OUT) {
*cache_resc_hier = "comp_resc;ufs_cache";
*resc_hier = *DATAOBJINP.resc_hier;
if("*cache_resc_hier" == "*resc_hier") {
delay("<PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>") {
*unused_param = "";
*obj_path = *DATAOBJINP.obj_path;
msisync_to_archive("*cache_resc_hier", "*unused_param", "*obj_path");
}
}
}
Test Put, Delayed
irods@example:~$ iput -R comp_resc test_file test_file2 ; ils -l /tempZone/home/rods: rods 0 comp_resc;ufs_cache 10485760 2023-06-05.18:18 & test_file rods 1 comp_resc;ufs_arch 10485760 2023-06-05.18:18 & test_file rods 0 comp_resc;ufs_cache 10485760 2023-06-05.18:25 & test_file2
irods@example:~$ ils -l /tempZone/home/rods: rods 0 comp_resc;ufs_cache 10485760 2023-06-05.18:18 & test_file rods 1 comp_resc;ufs_arch 10485760 2023-06-05.18:18 & test_file rods 0 comp_resc;ufs_cache 10485760 2023-06-05.18:25 & test_file2 rods 1 comp_resc;ufs_arch 10485760 2023-06-05.18:25 & test_file2
Wait for it...
Questions?