Policy Training

Publishing

Jason M. Coposky

@jason_coposky

Executive Director, iRODS Consortium

Policy Training

Publishing

June 25-28, 2019

iRODS User Group Meeting 2019

Utrecht, Netherlands

• Packaged and supported solutions
• Require configuration not code
• Derived from the majority of use cases observed in the user community

iRODS Capabilities

Publishing

A policy framework that provides an asynchronous, scalable data publishing service driven by metadata

• Publishing technology of choice is reached by delegating policy implementation
• Persistent identifier generation is delegated to a policy invocation

Publishing Policy Components

• Persistent Identifier
• Publishing Policy Implementation
  • irods_policy_publishing_object_publish_<technology>
  • irods_policy_publishing_object_purge_<technology>
  • irods_policy_publishing_collection_publish_<technology>
  • irods_policy_publishing_collection_purge_<technology>

<technology> is directly derived from metadata and is used to delegate the policy invocation

Core Competencies

Policy

Capabilities

Publishing Overview

Getting Started

wget http://people.renci.org/~jasonc/irods/irods-rule-engine-plugin-dataworld.deb

wget http://people.renci.org/~jasonc/irods/irods-rule-engine-plugin-publishing.deb

wget http://people.renci.org/~jasonc/irods/irods-rule-engine-plugin-persistent-identifier.deb

As the ubuntu user

Download the publishing packages

sudo dpkg -i irods-rule-engine-plugin-dataworld.deb irods-rule-engine-plugin-publishing.deb irods-rule-engine-plugin-persistent-identifier.deb

Install the Publishing packages

        "rule_engines": [

            ...

            {
                "instance_name": "irods_rule_engine_plugin-dataworld-instance",
                "plugin_name": "irods_rule_engine_plugin-dataworld",
                "plugin_specific_configuration": {
                }
            },
            {
                "instance_name": "irods_rule_engine_plugin-publishing-instance",
                "plugin_name": "irods_rule_engine_plugin-publishing",
                "plugin_specific_configuration": {
                }
            },

            {
                "instance_name": "irods_rule_engine_plugin-persistent_identifier-instance",
                "plugin_name": "irods_rule_engine_plugin-persistent_identifier",
                "plugin_specific_configuration": {
                }
            },

As the irods user

Edit /etc/irods/server_config.json

imeta set -u <user name> irods::publishing::api_token <your token>

Go to https://data.world:

• Create an account
• Find your API Token:  User Icon > Settings > Advanced

Annotate the user with the token

iadmin mkuser <username> rodsuser

iadmin moduser <username> password <password>

Create an iRODS user that matches your data.world account

We will use the irodsconsortium user name and account

Collections and Data Objects are tagged with metadata to indicate they should be published

A new AVU applied to a populated collection will schedule all objects for publication

New objects cannot be placed into a collection with a publishing AVUs applied.  Nor can those object be modified with POSIX operations.

Publishing metadata takes the form:

A:  irods::publishing::publish
V:  <service>

The service name is directly applied the the policy name template, which dictates which policies are invoked.

Download some data

wget https://cdn.patricktriest.com/data/books.zip

unzip books.zip

imkdir published_collection

Create a collection to be published

iput -r ./books published_collection/books0

Put a collection of data to the collection to be indexed

Initialize the ubuntu user as your iRODS dataworld username

imeta set -C published_collection irods::publishing::publish dataworld

Set the metadata on published_collection

id     name
11269 {"collection-name":"/tempZone/home/rods/published_collection","publish-type":"collection","publisher":"dataworld","rule-engine-instance-name":"irods_rule_engine_plugin-publishing-instance","rule-engine-operation":"irods_policy_publishing_collection_publish","user-name":"rods"}

A delayed execution job is scheduled which will then scan and schedule publishing jobs

Once the queue is empty check data.world for the data set

imeta set -d published_file0 irods::publishing::publish dataworld

Set the metadata on published_object0

truncate -s 1M 1Mfile

Create a new data object

iput 1Mfile published_file0

imeta ls -d published_file0

Inspect the metadata for the object

AVUs defined for dataObj published_file0:
attribute: irods::access_time
value: 1561193732
units:
----
attribute: irods::publishing::persistent_identifier
value: OTcwNTczZGMtY2UwMC00MDU4LWJmNjEtMWRlNmMwZjAwNWEw
units:
----
attribute: irods::publishing::publish
value: dataworld
units:

remote addresses: 127.0.0.1 ERROR: rmUtil: rm error for /tempZone/home/irodsconsortium/published_file0, status = -35000 status = -35000 SYS_INVALID_OPR_TYPE
Level 0: object is published and now immutable [/tempZone/home/irodsconsortium/file3]

imeta rm -d file3 irods::publishing::publish dataworld

irm -f published_file0

rodsusers cannot modify or delete published content

Users cannot remove publication metadata

remote addresses: 127.0.0.1 ERROR: Level 0: publishing metadata tags are immutable [/tempZone/home/irodsconsortium/file3]
remote addresses: 127.0.0.1 ERROR: rcModAVUMetadata failed with error -35000 SYS_INVALID_OPR_TYPE
Level 0: publishing metadata tags are immutable [/tempZone/home/irodsconsortium/file3]

The data.world publication policy delegates the generation of persistent identifiers.

• By default it is a base64 encoded UUID

irods_policy_publishing_persistent_identifier(
    *object_path, *service_name, *pid) {
    writeLine("serverLog", "Persistent Identifier - [*object_path]")
    *pid = "ABC123"
}

As the irods user

edit /etc/irods/persistent_identifier.re

Configure the rule base

            {
                "instance_name": "irods_rule_engine_plugin-irods_rule_language-instance",
                "plugin_name": "irods_rule_engine_plugin-irods_rule_language",
                "plugin_specific_configuration": {
                    "re_data_variable_mapping_set": [
                        "core"
                    ],
                    "re_function_name_mapping_set": [
                        "core"
                    ],
                    "re_rulebase_set": [
                        "persistent_identifier",
                        "document_type",
                        "elasticsearch",
                        "core"
                    ],

Disable the Persistent Identifier plugin

        "rule_engines": [

            {
                "instance_name": "irods_rule_engine_plugin-dataworld-instance",
                "plugin_name": "irods_rule_engine_plugin-dataworld",
                "plugin_specific_configuration": {
                }
            },
            {
                "instance_name": "irods_rule_engine_plugin-publishing-instance",
                "plugin_name": "irods_rule_engine_plugin-publishing",
                "plugin_specific_configuration": {
                }
            },
            {
                "instance_name": "irods_rule_engine_plugin-persistent_identifier-instance",
                "plugin_name": "irods_rule_engine_plugin-persistent_identifier",
                "plugin_specific_configuration": {
            }

iput 1Mfile publishing_pid_test

imeta set -d publishing_pid_test irods::publishing::publish dataworld

As the ubuntu user test the new policy

AVUs defined for dataObj pid_test7:
attribute: irods::publishing::publish
value: dataworld
units:
----
attribute: irods::publishing::persistent_identifier
value: ABC123
units:

imeta ls -d publishing_pid_test

Check our results

Policy Signatures - Implement these four policies to provide integration to a new publishing service

irods_policy_publishing_object_publish_<service>(

    *object_path, *user_name, *service_name)
irods_policy_publishing_object_purge_<service>(

    *object_path, *user_name, *service_name)

irods_policy_publishing_collection_index_<service>(

    *collection_name, *user_name, *service_name)
irods_policy_indexing_collection_purge_<service>(

    *collection_name, *user_name, *service_name

The Publishing Policy provides a reactive framework to metadata attributes.  Once the publishing service policy is invoked, it may provide any implementation desired.

For instance, some services may simply need a URI to the data set whereas others may require the data be uploaded, such as data.world.

The publishing service may require a specific submission package format, additional metadata or other requirements which would require the publishing job to wait until these needs are met.

Questions?