Storage Tiering

November 12-15, 2018

Supercomputing 2018

Dallas, TX

Justin James

Applications Engineer, iRODS Consortium

Storage Tiering

iRODS Capabilities

  • Packaged and supported solutions
  • Require configuration not code
  • Derived from the majority of use cases observed in the user community

Storage Tiering Overview

Getting a Docker Image with iRODS

docker pull justinkylejames/irods-storage-tiering-demo:latest

Get the docker image with iRODS 4.2.

Run the instance

docker run -ti justinkylejames/irods-storage-tiering-demo:latest /bin/bash

Connect a terminal to the instance

$ docker ps --format "{{.Image}} {{.Names}}"
justinkylejames/irods-storage-tiering-demo:latest cranky_villani

 

$ docker exec -ti cranky_villani /bin/bash

Installing Tiered Storage Plugin

wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add -
echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" | \
  sudo tee /etc/apt/sources.list.d/renci-irods.list
sudo apt-get update

Install the package repository

sudo apt-get install irods-rule-engine-plugin-storage-tiering

Install the storage tiering package

Configuring Tiered Storage Plugin

Plugin is configured in  /etc/irods/server_config.json

"rule_engines": [
    {
         "instance_name": "irods_rule_engine_plugin-storage_tiering-instance",
         "plugin_name": "irods_rule_engine_plugin-storage_tiering",
         "plugin_specific_configuration": {
         }
    },
    {    
        "instance_name": "irods_rule_engine_plugin-irods_rule_language-instance",
        "plugin_name": "irods_rule_engine_plugin-irods_rule_language",
        "plugin_specific_configuration": {  
            <snip>
        },
        "shared_memory_instance": "irods_rule_language_rule_engine"
    },
]

Tiering Example

Three Tier Groups with Common Archive

We will demonstrate data flow from instrument to archive

Storage Tiering Demo

We will illustrate storage tiering with three storage groups (A, B, and C) each with three storage tiers (0, 1, and 2).

In this example, a common resource is used for the archive tier (2).

 

Creating the resources:

iadmin mkresc tier2 unixfilesystem `hostname`:/tmp/irods/tier2
iadmin mkresc tier0_A unixfilesystem `hostname`:/tmp/irods/tier0_A
iadmin mkresc tier1_A unixfilesystem `hostname`:/tmp/irods/tier1_A

iadmin mkresc tier0_B unixfilesystem `hostname`:/tmp/irods/tier0_B
iadmin mkresc tier1_B unixfilesystem `hostname`:/tmp/irods/tier1_B

iadmin mkresc tier0_C unixfilesystem `hostname`:/tmp/irods/tier0_C
iadmin mkresc tier1_C unixfilesystem `hostname`:/tmp/irods/tier1_C

Configuring the Tiering Policy

The tiering policy can be configured  simply by adding metadata to our resources.  

 

  • irods::storage_tiering::group - Used to assign a resource to a tier group
     
  • irods::storage_tiering::time - Used to set the time violation policy for objects within the resource
 

 

Assigning resources to a storage tiering group

imeta set -R tier0_A irods::storage_tiering::group tier_group_A 0

imeta set -R tier1_A irods::storage_tiering::group tier_group_A 1

imeta add -R tier2 irods::storage_tiering::group tier_group_A 2

Tier Group A

Tier Group B

imeta set -R tier0_B irods::storage_tiering::group tier_group_B 0

imeta set -R tier1_B irods::storage_tiering::group tier_group_B 1

imeta add -R tier2 irods::storage_tiering::group tier_group_B 2

Tier Group C

imeta set -R tier0_C irods::storage_tiering::group tier_group_C 0

imeta set -R tier1_C irods::storage_tiering::group tier_group_C 1

imeta add -R tier2 irods::storage_tiering::group tier_group_C 2

Notice that the archive tier (tier 2) belongs to all three tier groups.

Set Tier Time Constraints

imeta set -R tier0_A irods::storage_tiering::time 30

imeta set -R tier1_A irods::storage_tiering::time 60

Tier Group A

Tier Group B

imeta set -R tier0_B irods::storage_tiering::time 45

imeta set -R tier1_B irods::storage_tiering::time 120

The archive tier has no time constraints.

Tier Group C

imeta set -R tier0_C irods::storage_tiering::time 100
imeta set -R tier1_C irods::storage_tiering::time 180

Running a Storage Tiering Rule

{
   "rule-engine-instance-name": "irods_rule_engine_plugin-storage_tiering-instance",
   "rule-engine-operation": "apply_storage_tiering_policy",
   "delay-parameters": "<PLUSET>1s</PLUSET><EF>REPEAT FOR EVER</EF>",
   "storage-tier-groups": [
       "tier_group_A",
       "tier_group_B",
       "tier_group_C"
   ]
}
INPUT null
OUTPUT ruleExecOut

Now we will create a rule file (/var/lib/irods/foo.r) which describes our storage tiering policy for tier groups A, B, C.

Executing this rule with the irule command will put the storage tiering policy on the delay execution queue.

irods@hostname:~$ irule -r irods_rule_engine_plugin-storage_tiering-instance -F /var/lib/irods/foo.r

irods@hostname:~$ iqstat
id     name
10096 {"rule-engine-operation":"apply_storage_tiering_policy","storage-tier-groups":["tier_group_A","tier_group_B","tier_group_C"]}

 

Reminder of What We Have Configured

Stage data into all three groups and watch

irods@hostname:~$ iput -R tier0_A 10Mfile 10MfileA && iput -R tier0_B 10Mfile 10MfileB && iput -R tier0_C 10Mfile 10MfileC
irods@hostname:~$ ils -l
/tempZone/home/rods:
  rods              0 tier0_A     10485760 2018-11-07.14:15 & 10MfileA
  rods              0 tier0_B     10485760 2018-11-07.14:15 & 10MfileB
  rods              0 tier0_C     10485760 2018-11-07.14:16 & 10MfileC

All newly ingested files reside in tier 0.

Create a test file

truncate -s 10M 10Mfile

Put the test file at tier 0 in tier groups A, B, and C.

Wait for it...

irods@hostname:~$ ils -l
/tempZone/home/rods:
  rods              2 tier2       10485760 2018-11-07.15:26 & 10MfileA
  rods              1 tier1_B     10485760 2018-11-07.15:26 & 10MfileB
  rods              0 tier0_C     10485760 2018-11-07.15:25 & 10MfileC

After one minute...

irods@hostname:~$ ils -l
/tempZone/home/rods:
  rods              2 tier2     10485760 2018-11-07.15:26 & 10MfileA
  rods              2 tier2     10485760 2018-11-07.15:27 & 10MfileB
  rods              2 tier2     10485760 2018-11-07.15:28 & 10MfileC​​

All newly ingested files now reside in tier 2.

After three minutes...

Restage to Lowest Tier

imeta add -R tier1_A irods::storage_tiering::minimum_restage_tier true

irods@hostname:~$ iget 10MfileA
irods@hostname:~$ ils -l
/tempZone/home/rods:
  rods              6 tier0_A     10485760 2018-11-07.15:40 & 10MfileA
  rods              2 tier2       10485760 2018-11-07.15:27 & 10MfileB
  rods              2 tier2       10485760 2018-11-07.15:28 & 10MfileC

Now retrieve 10MfileA again

Now let's say we want tier1 to be the minimum tier from group A

Now retrieve 10MfileA

irods@hostname:~$ iget -f 10MfileA
irods@hostname:~$ ils -l
/tempZone/home/rods:
  rods             12 tier1_A     10485760 2018-11-07.15:54 & 10MfileA
  rods              5 tier2       10485760 2018-11-07.15:46 & 10MfileB
  rods              2 tier2       10485760 2018-11-07.15:28 & 10MfileC

Summary

  • We set up three storage tiering groups with three tiers for each group.
     
  • We assigned resources to the groups and used a single archive resource as the final tier for each group.
     
  • We demonstrated the flow of files through the various tiers.
     
  • All of this was done by setting metadata on the storage resources.

Configuring Storage Tiering

Let's now discuss all the various ways that the storage tiering framework can be configured and customized...

Data Object Access Time

The default policy for tiering is based on the last time of access for a given data object which is applied as metadata

pep_api_data_obj_close_post
pep_api_data_obj_put_post
pep_api_data_obj_get_post
pep_api_phy_path_reg_post
irods::access_time <unix timestamp>

Dynamic Policy Enforcement Points for RPC API are used to apply the metadata

Configuring a Tier Group

imeta set -R <resc0> irods::storage_tiering::group example_group 0
imeta set -R <resc1> irods::storage_tiering::group example_group 1
imeta set -R <resc2> irods::storage_tiering::group example_group 2

Tier groups are entirely driven by metadata

  • The attribute identifies the resource as a tiering group participant
  • The value defines the group name
  • The unit defines the position within the group
  • Tier position, or index, can be any value - order will be honored
  • Configuration must be performed at the root of a resource composition
  • A resource may belong to many tiering groups

Configuring Tiering Time Constraints

Tiering violation time is configured in seconds

imeta set -R <resc> irods::storage_tiering::time 2592000

The final tier in a group does not have a storage tiering time

  - it will hold data indefinitely

imeta set -R <resc> irods::storage_tiering::time 30

Configure a tier to hold data for 30 seconds

Configure a tier to hold data for 30 days

Verification of Data Migration

When data is found to be in violation:

  • Data object is replicated to the next tier
  • New replica integrity is verified (in one of three ways)
  • Source replica is trimmed

 

'catalog' is the default verification for all resources

imeta set -R <resc> irods::storage_tiering::verification catalog

Other verification settings:

  • filesystem - performs a stat of the file, more expensive
  • checksum - verifies by computing the checksum on the file at rest, most expensive

Configuring the restage resource

imeta add -R <resc> irods::storage_tiering::minimum_restage_tier true

When data is in a tier other than the lowest tier, upon access the data is restaged back to the lowest tier.

 

This flag identifies the tier for restage:

Users may not want data restaged back to the lowest tier, should that tier be very remote or not appropriate for analysis.

 

Consider a storage resource at the edge serving as a landing zone for instrument data.

Preserving Replicas

Some users may not wish to trim a replica from a tier when data is migrated, such as to allow data to be archived and also still available on fast storage.

 

 

To preserve a replica on any given tier, attach the following metadata flag to the root resource.

imeta set -R <resc> irods::storage_tiering::preserve_replicas true

Custom Violation Query

Admins may specify a custom query which identifies violating data objects

imeta set -R <resc> irods::storage_tiering::query "SELECT DATA_NAME, COLL_NAME, DATA_RESC_ID WHERE META_DATA_ATTR_NAME = 'irods::access_time' AND META_DATA_ATTR_VALUE < 'TIME_CHECK_STRING' AND DATA_RESC_ID IN ('10021', '10022')"

In this case 'TIME_CHECK_STRING' is a macro which is replaced by: now - irods::storage_tiering::time.

Any number of queries may be attached to a resource in order provide a range of criteria by which violating data may be identified

  • could include user applied metadata
  • could include externally harvested metadata

Storage Tiering Metadata Vocabulary

"plugin_specific_configuration": {
    "access_time_attribute" : "irods::access_time",
    "storage_tiering_group_attribute" : "irods::storage_tiering::group",
    "storage_tiering_time_attribute" : "irods::storage_tiering::time",
    "storage_tiering_query_attribute" : "irods::storage_tiering::query",
    "storage_tiering_verification_attribute" : "irods::storage_tiering::verification",
    "storage_tiering_restage_delay_attribute" : "irods::storage_tiering::restage_delay",
    "default_restage_delay_parameters" : "<PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>",
    "time_check_string" : "TIME_CHECK_STRING"
}

All default metadata attributes are configurable

Should there be a preexisting vocabulary in your organization,

it can be leveraged by redefining the metadata attributes used by the storage tiering framework.

Custom violation queries

Let's say that instead of a time based violation policy, we only want to trigger violation of an object if it has the metadata of the form archive_object=true.  The following custom query could be assigned as the tiering query for the resource.

iadmin asq "select distinct R_DATA_MAIN.data_name, R_COLL_MAIN.coll_name, R_DATA_MAIN.resc_id from R_DATA_MAIN, R_COLL_MAIN, R_OBJT_METAMAP r_data_metamap, R_META_MAIN r_data_meta_main where R_DATA_MAIN.resc_id IN (10021, 10022) AND r_data_meta_main.meta_attr_name = 'archive_object' AND r_data_meta_main.meta_attr_value = 'true' AND R_COLL_MAIN.coll_id = R_DATA_MAIN.coll_id AND R_DATA_MAIN.data_id = r_data_metamap.object_id AND r_data_metamap.meta_id = r_data_meta_main.meta_id order by R_COLL_MAIN.coll_name, R_DATA_MAIN.data_name" archive_query
imeta set -R tier1_A irods::storage_tiering::query archive_query specific

Attach the query to the resource.

Questions?

SC18 - Storage Tiering

By iRODS Consortium

Source: justinkylejames

SC18 - Storage Tiering

  • 1,888

More from iRODS Consortium