Cacheless S3 Resource

February 20, 2019

Renaissance Computing Institute

UNC-Chapel Hill

Justin James

iRODS Consortium

Cacheless S3 Resource

Introduction (Legacy Operation)

The legacy S3 plugin must be use in conjunction with a compound resource and a unixfilesystem cache resource.

The following is a sample hierarchy of the S3 plugin.

s3compound:compound
├── s3archive:s3
└── s3cache:unixfilesystem

This required the iRODS administrator to create a cache cleanup rule.

 

The S3 plugin itself only implemented a few operations:

  • irods::RESOURCE_OP_UNLINK
  • irods::RESOURCE_OP_STAT
  • irods::RESOURCE_OP_RENAME
  • irods::RESOURCE_OP_STAGETOCACHE
  • irods::RESOURCE_OP_SYNCTOARCH

 

All of the other operations were handled by the cache resource.

Introduction (New Modes of Operation)

The newest version of the S3 resource plugin will now support three operating modes:

This mode is set using the HOST_MODE parameter in the S3 resource's context string.

 

If the HOST_MODE is not set, the existing default behavior will be used, now known as "archive_attached".

 

Note that "archive_detached" is not a valid mode.  

Archive Cacheless
Attached archive_attached
(default)		 cacheless_attached
(demonstrated today)
Detached N/A cacheless_detached
(coming soon)

Introduction (Archive vs Cacheless)

  • Archive - The S3 resource acts in the archive role behind a compound resource.
    • Requires a sibling cache resource to provide POSIX semantics.
    • Must be attached to a specific iRODS server.

 

 

  • Cacheless - The S3 resource can be standalone.
    • May be detached from any specific iRODS server (see next slide).
    • The S3 plugin provides POSIX semantics without using a cache resource and without requiring any cache management.

 

Introduction (Attached vs Detached)

  • Attached - Only the server that is defined as the host in the resource configuration will serve the request. 

 

 

  • Detached - All iRODS servers may serve a request for an object.  This is appropriate if all servers have connectivity to the S3-based backend.

 

Creating a Cacheless S3 Resource

iadmin mkresc s3resc s3 `hostname`:/irods-bucket/irods/Vault "S3_DEFAULT_HOSTNAME=s3.amazonaws.com;S3_AUTH_FILE=/var/lib/irods/s3.keypair;S3_REGIONNAME=us-east-1;S3_RETRY_COUNT=1;S3_WAIT_TIME_SEC=3;S3_PROTO=HTTP;ARCHIVE_NAMING_POLICY=consistent;HOST_MODE=cacheless_attached"

Creating a cacheless S3 resource is very similar to creating a legacy/archive S3 resource.  

 

As stated previously, the only differences are that the cacheless S3 may be a standalone resource and the HOST_MODE must be set to either "cacheless_attached" or "cacheless_detached".

 

The following is an example of creating a "cacheless_attached" S3 resource to Amazon S3.

Implementation Details

To implement the cacheless S3 resource, we added many of the operations that the existing archive S3 resource was missing.

 

These include: 

 

  • irods::RESOURCE_OP_OPEN
  • irods::RESOURCE_OP_READ
  • irods::RESOURCE_OP_WRITE
  • irods::RESOURCE_OP_CLOSE
  • irods::RESOURCE_OP_LSEEK

Implementation Details

We started with a project called S3FS which is an open source FUSE mount point for S3.  Much of the integration was translations of FUSE operations to iRODS resource operations.


However, the translation was not seamless.

 

  • Linux handles multiple opens on a single file which is transparent to FUSE.
  • Linux handles file offsets which is transparent to FUSE.  
  • The S3 plugin needed to be adjusted to handle multiple simultaneous writes.

Demonstration of the S3 Plugin

First, some assumptions:

  • The S3 credentials have been saved to /var/lib/irods/news3resc.keypair with a newline separating the Access Key ID and Secret Access Key.
  • The S3 plugin has been installed on the server.
  • A bucket named justinkylejames-irods1 has been created with write permissions.

 

Create a cacheless S3 plugin:

iadmin mkresc news3resc s3 `hostname`:/justinkylejames-irods1/irods/Vault "S3_DEFAULT_HOSTNAME=s3.amazonaws.com;S3_AUTH_FILE=/var/lib/irods/news3resc.keypair;S3_REGIONNAME=us-east-1;S3_RETRY_COUNT=1;S3_WAIT_TIME_SEC=3;S3_PROTO=HTTP;ARCHIVE_NAMING_POLICY=consistent;HOST_MODE=cacheless_attached"

Demonstration of the S3 Plugin

Create a simple test file and put it to the S3 resource:

$ echo 'this is a test file' > test.txt
$ iput -R news3resc test.txt

Using the S3 commands, check that the file exists:

$ aws s3 ls s3://justinkylejames-irods1/irods/Vault/home/rods/
2019-02-18 14:55:44         20 test.txt

Get the file using iRODS:

$ iget test.txt -
this is a test file

Demonstration of the S3 Plugin

Rename the data object and check that it has been renamed in S3:

$ imv test.txt newname.txt
$ ils -L
/tempZone/home/rods:
  rods              0 news3resc           20 2019-02-18.14:55 & newname.txt
        generic    /justinkylejames-irods1/irods/Vault/home/rods/newname.txt
$ aws s3 ls s3://justinkylejames-irods1/irods/Vault/home/rods/
2019-02-18 15:23:24         20 newname.txt

Remove the file:

$ irm -f newname.txt
$ ils
/tempZone/home/rods:
$ aws s3 ls s3://justinkylejames-irods1/irods/Vault/home/rods/

Demonstration of the S3 Plugin

For a more comprehensive test, I've created a file that is large enough to use the iRODS parallel file transfer mechanism.  I have already created a 64M file that contains data that I will put into iRODS.

$ iput -R news3resc 64Mfile

Get the file:

$ iget -f 64Mfile 64Mfile2

Compare the original file (64Mfile) to the file we retrieved (64Mfile2):

$ diff 64Mfile 64Mfile2

$ cksum 64Mfile 64Mfile2
1941261876 67108864 64Mfile
1941261876 67108864 64Mfile2

Next Steps

The cacheless S3 plugin is a work-in-progress.

 

  • As you may have noticed with the iget example, file downloads are currently slow.  The iRODS parallel get is conflicting with the S3 multipart download.

 

  • Some existing S3 features are still being tested and/or have not yet been ported to the new modes:
    • Comma-separated list for S3_DEFAULT_HOST
    • S3_ENABLE_MD5 flag
    • S3_ENABLE_MPU flag
    • ARCHIVE_NAMING_POLICY flag

 

  • Enhance the S3 authentication options so that the credentials may be stored in the catalog or some other service (e.g. Vault).

 

  • Upon further testing and refinement, we plan to use the same approach to provide cacheless modes for other existing iRODS archive resources (WOS, etc.)