Policy Training

Indexing

Jason M. Coposky

@jason_coposky

Executive Director, iRODS Consortium

Policy Training

Indexing

June 25-28, 2019

iRODS User Group Meeting

University of Utrecht, NL

• Packaged and supported solutions
• Require configuration not code
• Derived from the majority of use cases observed in the user community

iRODS Capabilities

Indexing

A policy framework that provides an asynchronous, scalable full text and metadata indexing service driven by collection assigned metadata

• Indexing technology of choice is reached by delegating policy implementation

 

• Document Type identification is delegated to a policy invocation

Indexing Policy Components

• Document Type

• Indexing Policy Implementation

  • irods_policy_indexing_object_index_<technology>

  • irods_policy_indexing_object_purge_<technology>

  • irods_policy_indexing_metadata_index_<technology>

  • irods_policy_indexing_metadata_purge_<technology>

<technology> is directly derived from metadata and is used to delegate the policy invocation

Core Competencies

Policy

Capabilities

Indexing Overview

Getting Started

wget http://people.renci.org/~jasonc/irods/irods-externals-cpr1.3.0-0_1.0~xenial_amd64.deb
wget http://people.renci.org/~jasonc/irods/irods-externals-elasticlient0.1.0-0_1.0~xenial_amd64.deb
wget http://people.renci.org/~jasonc/irods/irods-rule-engine-plugin-document-type.deb
wget http://people.renci.org/~jasonc/irods/irods-rule-engine-plugin-elasticsearch.deb
wget http://people.renci.org/~jasonc/irods/irods-rule-engine-plugin-indexing.deb

As the ubuntu user

Download the Indexing packages

sudo dpkg -i irods-externals-cpr1.3.0-0_1.0~xenial_amd64.deb irods-externals-elasticlient0.1.0-0_1.0~xenial_amd64.deb irods-rule-engine-plugin-document-type.deb irods-rule-engine-plugin-elasticsearch.deb irods-rule-engine-plugin-indexing.deb

Install the Indexing packages

        "rule_engines": [

            ...

            {
                "instance_name": "irods_rule_engine_plugin-indexing-instance",
                "plugin_name": "irods_rule_engine_plugin-indexing",
                "plugin_specific_configuration": {
                }
            },
            {
                "instance_name": "irods_rule_engine_plugin-elasticsearch-instance",
                "plugin_name": "irods_rule_engine_plugin-elasticsearch",
                "plugin_specific_configuration": {
                    "hosts" : ["http://localhost:9100/"],
                    "bulk_count" : 100,
                    "read_size" : 4194304
                }
            },
            {
                "instance_name": "irods_rule_engine_plugin-document_type-instance",
                "plugin_name": "irods_rule_engine_plugin-document_type",
                "plugin_specific_configuration": {
                }
            },

As the irods user

Edit /etc/irods/server_config.json

Exit the shell and log back in to evaluate the new group

sudo apt-get -y install docker.io

sudo usermod -aG docker $USER

As the ubuntu user

Install and configure docker

run docker ps to ensure you can do so without sudo

docker pull docker.elastic.co/elasticsearch/elasticsearch:6.7.0
docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.7.0

As the ubuntu user

curl -X PUT -H'Content-Type: application/json' http://localhost:9100/full_text_index
curl -X PUT -H'Content-Type: application/json' http://localhost:9100/full_text_index/_mapping/text -d '{ "properties" : { "object_path" : { "type" : "text" }, "data" : { "type" : "text" } } }'

Create both full_text and metadata indicies

curl -X PUT -H'Content-Type: application/json' http://localhost:9100/metadata_index
curl -X PUT -H'Content-Type: application/json' http://localhost:9100/metadata_index/_mapping/text -d '{ "properties" : { "object_path" : { "type" : "text" }, "attribute" : { "type" : "text" }, "value" : { "type" : "text" }, "unit" : { "type" : "text" } } }'

Wait for the container to start ...

Collections are tagged with metadata to indicate they should be indexed

A new AVU applied to a populated collection will schedule all objects for indexing

New objects placed into a collection with one or more indexing AVUs applied will also be indexed

Objects that are modified or moved into a collection with one or more indexing AVUs applied will also be indexed

Indexing metadata takes the form:

A:  irods::indexing::index
V:  <index name>::<index type>
U:  <technology>

• index name is specific to your index configuration
• index type is either: full_text or metadata
• technology specifies which policy will be invoked to perform the indexing - currently elasticsearch

Download some data

wget https://cdn.patricktriest.com/data/books.zip
unzip books.zip

imkdir indexed_collection

Create a collection to be indexed

iput -r ./books indexed_collection/books0

Put a directory of files into the collection to be indexed

As the irods user:

imeta set -C indexed_collection irods::indexing::index full_text_index::full_text elasticsearch

Set the metadata on indexed_collection for full_text

A delayed execution job is scheduled which will then scan and schedule indexing jobs

id     name
10232 {"attribute":"","index-name":"full_text_index","index-type":"full_text","indexer":"elastic","object-path":"/tempZone/home/rods/indexed_collection/books0/120.txt","rule-engine-instance-name":"irods_rule_engine_plugin-indexing-instance","rule-engine-operation":"irods_policy_indexing_object_index","source-resource":"EMPTY_RESOURCE_NAME","units":"","user-name":"rods","value":""}

...

Each indexing job will batch upload data for the full_text indexing type

Count the indexing jobs

curl -X GET -H'Content-Type: application/json' HTTP://localhost:9100/full_text_index/text/_search?pretty=true -d '

{

    "from": 0, "size" : 500,

    "_source" : ["object_path"],

    "query" : {

        "term" : { "object_path" : "books0"}

    }

}'

Search the index for all object_paths which contain "books0"

curl -X GET -H'Content-Type: application/json' HTTP://localhost:9100/full_text_index/text/_search?pretty=true -d '

{

    "from": 0, "size" : 500,

    "_source" : ["object_path"],

    "query" : {

        "term" : { "data" : "the"}

    }

}'

Search the index for all contents which contain "the"

iput -r ./books indexed_collection/books1

iqstat | wc -l

Put a collection of data to the collection to be indexed

An indexing event was asynchronously scheduled given the existing metadata tag on indexed_collection

Create a subcollection which indexes metadata

Any data put to this new collection will also be scheduled for full text index as its parent holds that metadata tag

imkdir indexed_collection/metadata_indexing

imeta set -C indexed_collection/metadata_indexing irods::indexing::index metadata_index::metadata elasticsearch

Add some metadata to an object in the collection

iput -r books indexed_collection/metadata_indexing/books2

imeta add -d indexed_collection/metadata_indexing/books2/33.txt attr0 val0 units0

Check the delayed execution queue

~$ iqstat | wc -l
100

Search for a match to the attribute field

curl -X GET -H'Content-Type: application/json' HTTP://localhost:9100/metadata_index/text/_search?pretty=true -d '
{
    "from": 0, "size" : 500,
    "_source" : ["object_path", "attribute", "value", "units"],
    "query" : {
        "term" : {"attribute" : "attr0"}
    }
}'

Add some metadata to an object in the collection

imeta add -d indexed_collection/metadata_indexing/books2/33.txt attr1 val1 units1

imeta add -d indexed_collection/metadata_indexing/books2/33.txt attr2 val2 units2

imeta add -d indexed_collection/metadata_indexing/books2/33.txt attr3 val3 units3

Wildcard query to match all the attribute fields

curl -X GET -H'Content-Type: application/json' HTTP://localhost:9100/metadata_index/text/_search?pretty=true -d '

{

    "from": 0, "size" : 500,

    "_source" : ["object_path", "attribute", "value", "units"],

    "query" : {

        "wildcard": {

            "attribute": {

                "value": "attr*",

                "boost": 1.0,

                "rewrite": "constant_score"

            }

        }

    }

}'

An administrator may wish to restrict indexing activities to particular resources, for example when automatically ingesting data.

In order to indicate a resource is available for indexing it may be annotated with metadata:

imeta add -R <resource name> irods::indexing::index true

If no resource be tagged it is assumed that all resources are available for indexing.

Should the tag exist on any resource in the system, it is assumed that all available resources for indexing are tagged.

Edit /etc/irods/document_type.re

irods_policy_indexing_document_type_elastic(
    *object_path, *source_resource, *document_type) {
    # do something terribly interesting with external services

    writeLine("serverLog", "Document Type [*object_path]")
    *document_type = "text"
}

The Document Type is used as part of the index, which is referenced in the url for the search.  Currently we have only indexed data in the default 'text' document type.

Edit /etc/irods/server_config.json

            {
                "instance_name": "irods_rule_engine_plugin-irods_rule_language-instance",
                "plugin_name": "irods_rule_engine_plugin-irods_rule_language",
                "plugin_specific_configuration": {
                    "re_data_variable_mapping_set": [
                        "core"
                    ],
                    "re_function_name_mapping_set": [
                        "core"
                    ],
                    "re_rulebase_set": [
                        "document_type",
                        "core"
                    ],

Edit /etc/irods/server_config.json

        "rule_engines": [
            {
                "instance_name": "irods_rule_engine_plugin-indexing-instance",
                "plugin_name": "irods_rule_engine_plugin-indexing",
                "plugin_specific_configuration": {
                }
            },
            ...
            {
                "instance_name": "irods_rule_engine_plugin-document_type-instance",
                "plugin_name": "irods_rule_engine_plugin-document_type",
                "plugin_specific_configuration": {
                }
            },
            ...

Trigger a new indexing event

grep "Document Type" log/rodsLog*

<SNIP> inString = Document Type [/tempZone/home/rods/indexed_collection/file3]

Check for debug message

iput VERSION.json indexed_collection/file0

iqstat

Policy Signatures - Implement these four policies to provide service to a new technology

irods_policy_indexing_object_index_<technology>(

    *object_path, *source_resource, *index_name, *index_type)
irods_policy_indexing_object_purge_<technology>(

    *object_path, *source_resource, *index_name, *index_type)

irods_policy_indexing_metadata_index_<technology>(

    *object_path, *attribute, *value, *unit, *index_name)
irods_policy_indexing_metadata_purge_<technology>(

    *object_path, *attribute, *value, *unit, *index_name)

The Indexing Policy provides a reactive framework to metadata attributes.  Once the indexing technology policy is invoked, it may provide any implementation desired.

For instance, given a document type, a Solr implementation can implement geographic indexing rather than full text for the "full_text" type and ignore the "metadata" type.

An implementation for Jena would ignore the "full_text" type and only implement the metadata policies.

Questions?