Rule Engine Plugins

June 14, 2016

Public Health England

London, England

Jason M. Coposky

@jason_coposky

Interim Executive Director

Anatomy of a Rule Engine Plugin

Represents the last of the core plugin interfaces

 

Each plugin must define six operations:

  • start
  • stop
  • rule_exists
  • exec_rule
  • exec_rule_text
  • exec_rule_expression

 

Plugins soon to be released:

  • audit-amqp
  • python

Configuration of Rule Engine Plugins

"rule_engines": [
    {
        ....
    }
]

Within /etc/irods/server_config.json -

    New top-level JSON array holding rule engine configuration

Order Is Important

  • multiple rule engines can be run concurrently, but this array defines which rule engine takes priority

Configuration of Rule Engine Plugins

Anatomy of a Rule Engine Plugin JSON object

{
    "instance_name": "<UNIQUE NAME>",
    "plugin_name": "<DERIVED FROM SHARED OBJECT>",
    "plugin_specific_configuration": {
        <ANYTHING GOES HERE>
    }
    "shared_memory_instance": "<UNIQUE SHM NAME>"
}

Dynamic Policy Enforcement and Rule Engine Plugins

dynPEPs are now enforced by a rule engine plugin

  • plugin_name will be 'dispatcher' before release
  • must be listed first for dynPEPs to work

Namespaces are enforced by the individual plugins:

  • represents a prefix for a given rule name and
  • enforced by the 'rule_exists' plugin operation
{
    "instance_name": "re-instance",
    "plugin_name": "re",
    "plugin_specific_configuration": {
        "namespaces": [
            {"namespace": ""},
            {"namespace": "audit_"},
            {"namespace": "indexing_"}]
    }
},

iRODS Rule Language Configuration

Similar to 4.1

  • dvm, fnm, and rulebase were moved into the plugin_specific_configuration
{
    "instance_name": "re-irods-instance",
    "plugin_name": "re-irods",
    "plugin_specific_configuration": {
        "re_data_variable_mapping_set": [
            { "filename": "core" }
        ],
        "re_function_name_mapping_set": [
            { "filename": "core" }
        ],
        "re_rulebase_set": [
            { "filename": "training" },
            { "filename": "core"}
        ]
    },
    "shared_memory_instance": "legacy_re"
}

Basic iRODS Rule Language Example

acPostProcForPut() {
    if("ufs_cache" == $rescName) {
        delay("<PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>") {
            *CacheRescName = "comp_resc;ufs_cache";
            msisync_to_archive("*CacheRescName", $filePath, $objPath );
        }
    }
}
pep_resource_resolve_hierarchy_pre(
  *INST_NAME,*CTX,*OUT,*OP_TYPE,*HOST,*RESC_HIER,*VOTE){
    if( "CREATE" == *OP_TYPE ) {
        if( "pt1" == *INST_NAME) {
            *OUT = "read=1.0;write=0.5"
        }
        else if ( "pt2" == *INST_NAME ) {
            *OUT = "read=1.0;write=1.0"
        }
    }
}

Static (Legacy) Policy Enforcement Points

Dynamic Policy Enforcement Points

Questions?

Installing the Python Rule Engine Plugin

sudo apt-get -y install irods-rule-engine-plugin-python

ls /var/lib/irods/plugins/rule_engines/
libre-irods.so  libre.so  librule-engine-plugin-python.so

Install the plugin

See the new shared object

Python Rule Engine Configuration

Edit /etc/irods/server_config.json

  • insert the new python plugin configuration stanza after the dispatcher and before the iRODS Rule Language plugin
  • this allows it to service requests first
{
    "instance_name": "re-instance",
    "plugin_name": "re",
    "plugin_specific_configuration": {
        "namespaces": [
            {"namespace": ""},
            {"namespace": "audit_"},
            {"namespace": "indexing_"}]
    }
},
{
    "instance_name" : "rule-engine-plugin-python-instance",
    "plugin_name" : "rule-engine-plugin-python",
    "plugin_specific_configuration" : {}
},

A Combination Use of both Python and iRODS Rules

add_metadata_to_objpath(*str, *objpath, *objtype) {
    msiString2KeyValPair(*str, *kvp);
    msiAssociateKeyValuePairsToObj(*kvp, *objpath, *objtype);
}

getSessionVar(*name,*output) {
    *output = eval("str($"++*name++")");
}

Update the iRODS Rule Language /etc/irods/training.re rulebase

A Python Static Policy Enforcement Point

# core.py
import os
import sys
import EXIF
def acPostProcForPut(rule_args, callback):
    phypath = callback.getSessionVar('filePath', 'dummy')[PYTHON_RE_RET_OUTPUT][1]
    objpath = callback.getSessionVar('objPath', 'dummy')[PYTHON_RE_RET_OUTPUT][1]
    exiflist = []
    with open(phypath, 'rb') as f:
        tags = EXIF.process_file(f, details=False)
        for (k, v) in tags.iteritems():
            if k not in ('JPEGThumbnail', 'TIFFThumbnail', 'Filename', 'EXIF MakerNote'):
                exifpair = '{0}={1}'.format(k, v)
                exiflist.append(exifpair)
    exifstring = '%'.join(exiflist)
    callback.add_metadata_to_objpath(exifstring, objpath, '-d')
    callback.writeLine('serverLog', 'PYTHON - acPostProcForPut() complete')

Define a python instantiation of the static PEP in /etc/irods/core.py

Test our combination of rules

wget ftp://ftp.renci.org/pub/irods/training/stickers.jpg

iput stickers.jpg

Get some test data into iRODS

imeta ls -d stickers.jpg

 

AVUs defined for dataObj stickers.jpg:
attribute: Image Orientation
value: Horizontal (normal)
units:
----
attribute: EXIF ColorSpace
value: sRGB
units:
----

Confirm the EXIF metadata was harvested

Implementing the Storage Balanced Plugin

findRescType(*INST_NAME, *OUT) {
    foreach ( *ROW in SELECT RESC_TYPE_NAME WHERE RESC_NAME = '*INST_NAME' ) {
        *OUT = *ROW.RESC_TYPE_NAME;
    }
}
findInstId(*INST_NAME, *OUT) {
    foreach ( *ROW in SELECT RESC_ID WHERE RESC_NAME = '*INST_NAME' ) {
        *OUT = *ROW.RESC_ID;
    }
}
findBytesUsed(*INST_ID, *OUT) {
    foreach ( *ROW1 in SELECT RESC_NAME WHERE RESC_PARENT = '*INST_ID' ) {
        *STORAGE_RESC = *ROW1.RESC_NAME;
        *TEMP = 0
        foreach ( *ROW2 in SELECT sum(DATA_SIZE) WHERE RESC_NAME = '*STORAGE_RESC' ) {
            *TEMP = *TEMP + int(*ROW2.DATA_SIZE)
        }
        *OUT = "*TEMP"
    }
}
findContextString(*INST_NAME, *OUT) {
    foreach ( *ROW in SELECT RESC_CONTEXT WHERE RESC_NAME = '*INST_NAME' ) {
        *OUT = *ROW.RESC_CONTEXT;
    }
}

A few helper functions to add to /etc/irods/training.re

Code can be found at ~/irods_training/advanced/python_storage_balancing.re

The Python Storage Balancing Rule - 1/2

import re
def pep_resource_resolve_hierarchy_pre(rule_args, callback):
    if rule_args[3] == 'CREATE':
        resc_type = ''
        ret = callback.findRescType(rule_args[0], resc_type)
        resc_type = str(ret[PYTHON_RE_RET_OUTPUT][1])
        if (resc_type == 'passthru'):
            inst_id = ''
            ret = callback.findInstId(rule_args[0], inst_id)
            inst_id = str(ret[PYTHON_RE_RET_OUTPUT][1])

            bytes_used = ''
            ret = callback.findBytesUsed(inst_id, bytes_used)
            bytes_used = str(ret[PYTHON_RE_RET_OUTPUT][1])

            max_bytes = -1.0
            context_string = ''
            ret = callback.findContextString(rule_args[0], context_string)
            context_string = str(ret[PYTHON_RE_RET_OUTPUT][1])

Add to /etc/irods/core.py

Code can be found at ~/irods_training/advanced/python_storage_balancing.py

The Python Storage Balancing Rule - 2/2

            max_bytes_index = context_string.find('max_bytes')
            if max_bytes_index != -1:
                max_bytes_re = 'max_bytes=(\d+)'
                max_bytes_search = re.search(max_bytes_re, context_string)
                max_bytes_str = max_bytes_search.group(1)
                max_bytes = float(max_bytes_str)

            percent_full = 0.0
            if max_bytes == -1:
                percent_full = 0.0
            elif max_bytes == 0:
                percent_full = 1.0
            else:
                percent_full = float(bytes_used)/max_bytes

            write_weight = 1.0 - percent_full
            rule_args[2] = 'read=1.0;write=' + str(write_weight)

Testing the Storage Balancing Rule

iput -R def_resc VERSION.json f1
iput -R def_resc VERSION.json f2
iput -R def_resc VERSION.json f3
iput -R def_resc VERSION.json f4

ils -l
/tempZone/home/rods:
  rods              0 def_resc;pt2;ufs2          645 2016-06-03.16:47 & f1
  rods              0 def_resc;pt1;ufs1          645 2016-06-03.16:47 & f2
  rods              0 def_resc;pt2;ufs2          645 2016-06-03.16:47 & f3
  rods              0 def_resc;pt1;ufs1          645 2016-06-03.16:47 & f4

Remove all data from def_resc

  • irmtrash - any data in the trashcan will count towards the total being used to determine the distribution of incoming files

Questions?

A Storage Balancing C++ Rule Engine Plugin

Code can be found at

~/irods_training/advanced/irods_rule_engine_plugin_storage_balancing/src/libre-storage-balancing.cpp

 

 

Start with the Factory

extern "C"
irods::pluggable_rule_engine<irods::default_re_ctx>* plugin_factory(const std::string& _instance_name,
                                 const std::string& _context) {
    auto re{new irods::pluggable_rule_engine<irods::default_re_ctx>(_instance_name , _context)};
    re->add_operation<irods::default_re_ctx&, const std::string&>(
            "start",
            std::function<irods::error(irods::default_re_ctx&, const std::string&)>(start));
    re->add_operation<irods::default_re_ctx&, const std::string&>(
            "stop",
            std::function<irods::error(irods::default_re_ctx&, const std::string&)>(stop));
    re->add_operation<irods::default_re_ctx&, std::string, bool&>(
            "rule_exists",
            std::function<irods::error(irods::default_re_ctx&, std::string, bool&)>(rule_exists));
    re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>(
            "exec_rule",
            std::function<irods::error(irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>(exec_rule));
    re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>(
            "exec_rule_text",
            std::function<irods::error(irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>(exec_rule_text));
    re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>(
            "exec_rule_expression",
            std::function<irods::error(irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>(exec_rule_expression));
    return re;
}

Anatomy of the Plugin Factory

extern "C"
irods::pluggable_rule_engine<irods::default_re_ctx>*
    plugin_factory(
        const std::string& _instance_name,
        const std::string& _context) {
....
}
  • Must have C linkage
  • Returns an irods::pluggable_rule_engine<>*
  • Accepts two const std::string& as instance name and context

Instantiate a new rule engine plugin

extern "C"
irods::pluggable_rule_engine<irods::default_re_ctx>*
    plugin_factory(
        const std::string& _instance_name,
        const std::string& _context) {
            auto re{new irods::pluggable_rule_engine<irods::default_re_ctx>( 
                        _instance_name,
                        _context)};
}
  • Allocate a raw pointer to an irods::pluggable_rule_engine
  • Pass the instance name and context to the ctor
    • attributes of the irods::plugin_base class

Wire the plugin operations

...

re->add_operation<irods::default_re_ctx&, const std::string&>(
        "start",
        std::function<
            irods::error(
                irods::default_re_ctx&,
                const std::string&)>(start));

...
  • Template parameters are the parameters of the function operation
  • First parameter is the calling name of the operation - "start"
  • Second parameter is a std::function wrapping the local function definition
    • Takes the full signature of the function as a template parameter
    • Takes the function pointer as an argument

Similar treatment for the other operations

  • stop
  • rule_exists
  • exec_rule
  • exec_rule_text
  • exec_rule_expression
re->add_operation<irods::default_re_ctx&, const std::string&>(
            "stop",
            std::function<irods::error(irods::default_re_ctx&, const std::string&)>(stop));
re->add_operation<irods::default_re_ctx&, std::string, bool&>(
            "rule_exists",
            std::function<
                irods::error(irods::default_re_ctx&, std::string, bool&)>(rule_exists));
re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>(
            "exec_rule",
            std::function<irods::error(
                irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>(
                    exec_rule));
re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>(
            "exec_rule_text",
            std::function<irods::error(
                irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>(
                    exec_rule_text));
re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>(
            "exec_rule_expression",
            std::function<irods::error(
                irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>(
                    exec_rule_expression));

start, stop, and rule_exists

irods::error start(irods::default_re_ctx&, const std::string&) {
    return SUCCESS();
}

irods::error stop(irods::default_re_ctx&, const std::string&) {
    return SUCCESS();
}

irods::error rule_exists(irods::default_re_ctx&, std::string _rule_name, bool& _ret) {
    _ret = _rule_name == "pep_resource_resolve_hierarchy_pre";
    return SUCCESS();
}
  • start and stop are no-ops
  • rule_exists is simply matching on pep_resource_resolve_hierarchy_pre as that is the only operation supported by the rule engine plugin

exec_rule

irods::error exec_rule(
    irods::default_re_ctx&,
    std::string            _rule_name,
    std::list<boost::any>& _rule_arguments,
    irods::callback        _effect_handler) {
    try {
        auto it_args{std::begin(_rule_arguments)};
        const auto& arg_resource_name{boost::any_cast<std::string&>(*it_args)};
        auto& arg_plugin_context{boost::any_cast<irods::plugin_context&>(*++it_args)};
        auto& arg_out{*boost::any_cast<std::string*>(*++it_args)};
        const auto& arg_operation_type{*boost::any_cast<const std::string*>(*++it_args)};
        const auto& arg_host{*boost::any_cast<const std::string*>(*++it_args)};
        auto& arg_hierarchy_parser{*boost::any_cast<irods::hierarchy_parser*>(*++it_args)};
        auto& arg_vote{*boost::any_cast<float*>(*++it_args)};
  • Begin by extracting all of the arguments from the list
    • Rule arguments are packed into a std::list for every operation ( recall the api plugin call wrappers )

Target specific operations

        if (arg_operation_type != "CREATE") {
            return SUCCESS();
        }

        ruleExecInfo_t& rei{get_rei(_effect_handler)};

        const std::string resource_type{
                              get_resource_type(
                                  arg_resource_name, *rei.rsComm)};
        if (resource_type != "passthru") {
            return SUCCESS();
        }

        const boost::optional<uint64_t> max_bytes{
            get_max_bytes(arg_resource_name, *rei.rsComm)};

 

  • Skip non CREATE operations
  • Fetch the ruleExecInfo_t from the framework
  • Fetch other values via helper functions

Handle max_bytes edge cases

        if (!max_bytes) {
            arg_out = "read=1.0;write=1.0";
            return SUCCESS();
        } else if (*max_bytes == 0) {
            arg_out = "read=1.0;write=0.0";
            return SUCCESS();
        }
  • The context string may not be defined
  • Short circuit if max_bytes is set to 0 explicitly

Compute the write_weight

        const uint64_t bytes_used_by_children{
                           get_bytes_used_by_all_children(
                               arg_resource_name,
                               *rei.rsComm)};
        const uint64_t bytes_required_for_new_data_object{
                           get_bytes_of_incoming_data_object(
                               arg_plugin_context)};
        const uint64_t hypothetical_bytes_used{
                           bytes_used_by_children +
                           bytes_required_for_new_data_object};
        const double percent_used{
             std::max(0.0, std::min(1.0, static_cast<double>
                  (hypothetical_bytes_used) / *max_bytes))};
        const double write_weight{1.0 - percent_used};
  • Fetch bytes used by resource and new data object
  • Compute new total bytes used
  • Compute write_weight given max_bytes

Wrapping up exec_rule

        std::stringstream out_stream;
        out_stream << "read=1.0;write=" << write_weight_string;
        arg_out = out_stream.str();
        return SUCCESS();
    } catch (const irods::exception& e) {
        rodsLog(LOG_ERROR, e.what());
        return ERROR(e.code(), "irods exception in exec_rule");
    }
    return SUCCESS();
}
  • Build the weight string given the computed write weight
  • Set the out variable - arg_out
  • Return SUCCESS() if all goes well
  • Catch the exception, log and return an error otherwise

exec_rule_text and exec_rule_expression

irods::error exec_rule_text(
    irods::default_re_ctx&,
    std::string _rule_text,
    std::list<boost::any>& _rule_arguments,
    irods::callback _effect_handler) {
    return ERROR(SYS_NOT_SUPPORTED, "not supported");
}

irods::error exec_rule_expression(
    irods::default_re_ctx&,
    std::string _rule_text,
    std::list<boost::any>& _rule_arguments,
    irods::callback _effect_handler) {
    return ERROR(SYS_NOT_SUPPORTED, "not supported");
}
  • Both return SYS_NOT_SUPPORTED
  • Another Rule Engine Plugin could pick them up

Building and Installing the Package

sudo apt-get -y install irods-externals-* irods-dev

 

export PATH=/opt/irods-externals/cmake3.5.2-0/bin:/opt/irods-externals/clang3.8-0/bin:$PATH

 

 which clang++

 which cmake

 

mkdir ~/build_storage_cpp

cd ~/build_storage_cpp

cmake ../irods_training/advanced/irods_rule_engine_plugin_storage_balancing/

make package

 

 sudo dpkg -i ./irods-re-storage-balancing-cpp_4.2.0~trusty_amd64.deb

Configuring the Rule Engine Plugin

Edit /etc/irods/server_config.json

        {
            "instance_name" : "re-storage-balancing-instance",
            "plugin_name" : "re-storage-balancing",
            "plugin_specific_configuration" : {}
        },
        {
            "instance_name" : "rule-engine-plugin-python-instance",
            "plugin_name" : "rule-engine-plugin-python",
            "plugin_specific_configuration" : {}
        },

Run ils to determine if syntax is correct

Testing the Rule Engine Plugin

iput -R def_resc VERSION.json f1
iput -R def_resc VERSION.json f2
iput -R def_resc VERSION.json f3
iput -R def_resc VERSION.json f4


ils -l
/tempZone/home/rods:
  rods              0 def_resc;pt2;ufs2          645 2016-06-03.20:10 & f1
  rods              0 def_resc;pt1;ufs1          645 2016-06-03.20:10 & f2
  rods              0 def_resc;pt2;ufs2          645 2016-06-03.20:10 & f3
  rods              0 def_resc;pt1;ufs1          645 2016-06-03.20:10 & f4

 

Remove all data from def_resc

  • irmtrash - any data in the trashcan will count towards the total being used to determine the distribution of incoming files

Questions?

Public Health England - Rule Engine Plugins

By jason coposky

Public Health England - Rule Engine Plugins

  • 1,827