Rule Engine Plugins
November 3, 2016
Bibliothèque et Archives nationales du Québec
Québec, Montréal, Canada
Terrell Russell, Ph.D.
@terrellrussell
Chief Technologist, iRODS Consortium
Anatomy of a Rule Engine Plugin
Represents the last of the core plugin interfaces
Each plugin must define six operations:
Plugins soon to be released:
Configuration of Rule Engine Plugins
"plugin_configuration": { "rule_engines": [ { ... },
... ] }
Within /etc/irods/server_config.json -
New JSON array holding rule engine configuration
Order Is Important
Configuration of Rule Engine Plugins
Anatomy of a Rule Engine Plugin JSON object
{ "instance_name": "<UNIQUE NAME>", "plugin_name": "<DERIVED FROM SHARED OBJECT>", "plugin_specific_configuration": { <ANYTHING GOES HERE> } "shared_memory_instance": "<UNIQUE SHM NAME>" }
iRODS Rule Language Configuration
Similar to 4.1
{ "instance_name": "irods_rule_engine_plugin-irods_rule_language-instance", "plugin_name": "irods_rule_engine_plugin-irods_rule_language", "plugin_specific_configuration": { "re_data_variable_mapping_set": [ "core" ], "re_function_name_mapping_set": [ "core" ], "re_rulebase_set": [ "training", "core" ], "regexes_for_supported_peps": [ "ac[^ ]*", "msi[^ ]*", "[^ ]*pep_[^ ]*_(pre|post)" ] }, "shared_memory_instance": "upgraded_legacy_re" }
Basic iRODS Rule Language Example
acPostProcForPut() { if("ufs_cache" == $KVPairs.rescName) { delay("<PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>") { *CacheRescName = "comp_resc;ufs_cache"; msisync_to_archive("*CacheRescName", $filePath, $objPath ); } } }
pep_resource_resolve_hierarchy_pre( *INST_NAME,*CTX,*OUT,*OP_TYPE,*HOST,*RESC_HIER,*VOTE){ if( "CREATE" == *OP_TYPE ) { if( "pt1" == *INST_NAME) { *OUT = "read=1.0;write=0.5" } else if ( "pt2" == *INST_NAME ) { *OUT = "read=1.0;write=1.0" } } }
Static (Legacy) Policy Enforcement Points
Dynamic Policy Enforcement Points
Questions?
Installing the Python Rule Engine Plugin
sudo apt-get -y install irods-rule-engine-plugin-python
ls /var/lib/irods/plugins/rule_engines/
libirods_rule_engine_plugin-cpp_default_policy.so
libirods_rule_engine_plugin-irods_rule_language.so
libirods_rule_engine_plugin-python.so
Install the plugin
See the new shared objects
Python Rule Engine Configuration
Edit /etc/irods/server_config.json
"rule_engines": [
{
"instance_name" : "irods_rule_engine_plugin-python-instance",
"plugin_name" : "irods_rule_engine_plugin-python",
"plugin_specific_configuration" : {}
},
{
"instance_name": "irods_rule_engine_plugin-irods_rule_language-instance",
A Combination Use of both Python and iRODS Rules
add_metadata_to_objpath(*str, *objpath, *objtype) { msiString2KeyValPair(*str, *kvp); msiAssociateKeyValuePairsToObj(*kvp, *objpath, *objtype); } getSessionVar(*name,*output) { *output = eval("str($"++*name++")"); }
Update the iRODS Rule Language /etc/irods/training.re rulebase
A Python Static Policy Enforcement Point
# core.py
import os
import sys
import EXIF
def acPostProcForPut(rule_args, callback):
phypath = callback.getSessionVar('filePath', 'dummy')[PYTHON_RE_RET_OUTPUT][1]
objpath = callback.getSessionVar('objPath', 'dummy')[PYTHON_RE_RET_OUTPUT][1]
exiflist = []
with open(phypath, 'rb') as f:
tags = EXIF.process_file(f, details=False)
for (k, v) in tags.iteritems():
if k not in ('JPEGThumbnail', 'TIFFThumbnail', 'Filename', 'EXIF MakerNote'):
exifpair = '{0}={1}'.format(k, v)
exiflist.append(exifpair)
exifstring = '%'.join(exiflist)
callback.add_metadata_to_objpath(exifstring, objpath, '-d')
callback.writeLine('serverLog', 'PYTHON - acPostProcForPut() complete')
Define a python instantiation of the static PEP in /etc/irods/core.py
Test our combination of rules
wget ftp://ftp.renci.org/pub/irods/training/stickers.jpg
iput stickers.jpg
Get some test data into iRODS
imeta ls -d stickers.jpg
AVUs defined for dataObj stickers.jpg:
attribute: Image Orientation
value: Horizontal (normal)
units:
----
attribute: EXIF ColorSpace
value: sRGB
units:
----
Confirm the EXIF metadata was harvested
irm -f stickers.jpg
Make sure earlier stickers example is removed
Implementing the Storage Balanced Plugin
findRescType(*INST_NAME, *OUT) { foreach ( *ROW in SELECT RESC_TYPE_NAME WHERE RESC_NAME = '*INST_NAME' ) { *OUT = *ROW.RESC_TYPE_NAME; } } findInstId(*INST_NAME, *OUT) { foreach ( *ROW in SELECT RESC_ID WHERE RESC_NAME = '*INST_NAME' ) { *OUT = *ROW.RESC_ID; } } findBytesUsed(*INST_ID, *OUT) { foreach ( *ROW1 in SELECT RESC_NAME WHERE RESC_PARENT = '*INST_ID' ) { *STORAGE_RESC = *ROW1.RESC_NAME; *TEMP = 0 foreach ( *ROW2 in SELECT sum(DATA_SIZE) WHERE RESC_NAME = '*STORAGE_RESC' ) { *TEMP = *TEMP + int(*ROW2.DATA_SIZE) } *OUT = "*TEMP" } } findContextString(*INST_NAME, *OUT) { foreach ( *ROW in SELECT RESC_CONTEXT WHERE RESC_NAME = '*INST_NAME' ) { *OUT = *ROW.RESC_CONTEXT; } }
A few helper functions to add to /etc/irods/training.re
Code can be found at ~/irods_training/advanced/python_storage_balancing.re
The Python Storage Balancing Rule - 1/2
import re def pep_resource_resolve_hierarchy_pre(rule_args, callback): if rule_args[3] == 'CREATE': resc_type = '' ret = callback.findRescType(rule_args[0], resc_type) resc_type = str(ret[PYTHON_RE_RET_OUTPUT][1]) if (resc_type == 'passthru'): inst_id = '' ret = callback.findInstId(rule_args[0], inst_id) inst_id = str(ret[PYTHON_RE_RET_OUTPUT][1]) bytes_used = '' ret = callback.findBytesUsed(inst_id, bytes_used) bytes_used = str(ret[PYTHON_RE_RET_OUTPUT][1]) max_bytes = -1.0 context_string = '' ret = callback.findContextString(rule_args[0], context_string) context_string = str(ret[PYTHON_RE_RET_OUTPUT][1])
Add to /etc/irods/core.py
Code can be found at ~/irods_training/advanced/python_storage_balancing.py
The Python Storage Balancing Rule - 2/2
max_bytes_index = context_string.find('max_bytes') if max_bytes_index != -1: max_bytes_re = 'max_bytes=(\d+)' max_bytes_search = re.search(max_bytes_re, context_string) max_bytes_str = max_bytes_search.group(1) max_bytes = float(max_bytes_str) percent_full = 0.0 if max_bytes == -1: percent_full = 0.0 elif max_bytes == 0: percent_full = 1.0 else: percent_full = float(bytes_used)/max_bytes write_weight = 1.0 - percent_full rule_args[2] = 'read=1.0;write=' + str(write_weight)
Testing the Storage Balancing Rule
iput -R def_resc VERSION.json f1 iput -R def_resc VERSION.json f2 iput -R def_resc VERSION.json f3 iput -R def_resc VERSION.json f4 ils -l /tempZone/home/rods: rods 0 def_resc;pt2;ufs2 645 2016-06-03.16:47 & f1 rods 0 def_resc;pt1;ufs1 645 2016-06-03.16:47 & f2 rods 0 def_resc;pt2;ufs2 645 2016-06-03.16:47 & f3 rods 0 def_resc;pt1;ufs1 645 2016-06-03.16:47 & f4
Remove all data from def_resc
Questions?
A Storage Balancing C++ Rule Engine Plugin
Code can be found at
~/irods_training/advanced/irods_rule_engine_plugin_storage_balancing/src/irods_rule_engine_plugin-cpp-storage_balancing.cpp
Start with the Factory
extern "C" irods::pluggable_rule_engine<irods::default_re_ctx>* plugin_factory(const std::string& _instance_name, const std::string& _context) { auto re{new irods::pluggable_rule_engine<irods::default_re_ctx>(_instance_name , _context)}; re->add_operation<irods::default_re_ctx&, const std::string&>( "start", std::function<irods::error(irods::default_re_ctx&, const std::string&)>(start)); re->add_operation<irods::default_re_ctx&, const std::string&>( "stop", std::function<irods::error(irods::default_re_ctx&, const std::string&)>(stop)); re->add_operation<irods::default_re_ctx&, std::string, bool&>( "rule_exists", std::function<irods::error(irods::default_re_ctx&, std::string, bool&)>(rule_exists)); re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>( "exec_rule", std::function<irods::error(irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>(exec_rule)); re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>( "exec_rule_text", std::function<irods::error(irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>(exec_rule_text)); re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>( "exec_rule_expression", std::function<irods::error(irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>(exec_rule_expression)); return re; }
Anatomy of the Plugin Factory
extern "C" irods::pluggable_rule_engine<irods::default_re_ctx>* plugin_factory( const std::string& _instance_name, const std::string& _context) { .... }
Instantiate a new rule engine plugin
extern "C"
irods::pluggable_rule_engine<irods::default_re_ctx>*
plugin_factory(
const std::string& _instance_name,
const std::string& _context) {
auto re{new irods::pluggable_rule_engine<irods::default_re_ctx>(
_instance_name,
_context)};
}
Wire the plugin operations
... re->add_operation<irods::default_re_ctx&, const std::string&>( "start", std::function< irods::error( irods::default_re_ctx&, const std::string&)>(start));
...
Similar treatment for the other operations
re->add_operation<irods::default_re_ctx&, const std::string&>( "stop", std::function<irods::error(irods::default_re_ctx&, const std::string&)>(stop)); re->add_operation<irods::default_re_ctx&, std::string, bool&>( "rule_exists", std::function< irods::error(irods::default_re_ctx&, std::string, bool&)>(rule_exists)); re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>( "exec_rule", std::function<irods::error( irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>( exec_rule)); re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>( "exec_rule_text", std::function<irods::error( irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>( exec_rule_text)); re->add_operation<irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback>( "exec_rule_expression", std::function<irods::error( irods::default_re_ctx&, std::string, std::list<boost::any>&, irods::callback)>( exec_rule_expression));
start, stop, and rule_exists
irods::error start(irods::default_re_ctx&, const std::string&) { return SUCCESS(); } irods::error stop(irods::default_re_ctx&, const std::string&) { return SUCCESS(); } irods::error rule_exists(irods::default_re_ctx&, std::string _rule_name, bool& _ret) { _ret = _rule_name == "pep_resource_resolve_hierarchy_pre"; return SUCCESS(); }
exec_rule
irods::error exec_rule(
irods::default_re_ctx&,
std::string _rule_name,
std::list<boost::any>& _rule_arguments,
irods::callback _effect_handler) {
try {
auto it_args{std::begin(_rule_arguments)};
const auto& arg_resource_name{boost::any_cast<std::string&>(*it_args)};
auto& arg_plugin_context{boost::any_cast<irods::plugin_context&>(*++it_args)};
auto& arg_out{*boost::any_cast<std::string*>(*++it_args)};
const auto& arg_operation_type{*boost::any_cast<const std::string*>(*++it_args)};
const auto& arg_host{*boost::any_cast<const std::string*>(*++it_args)};
auto& arg_hierarchy_parser{*boost::any_cast<irods::hierarchy_parser*>(*++it_args)};
auto& arg_vote{*boost::any_cast<float*>(*++it_args)};
Target specific operations
if (arg_operation_type != "CREATE") { return SUCCESS(); } ruleExecInfo_t& rei{get_rei(_effect_handler)}; const std::string resource_type{ get_resource_type( arg_resource_name, *rei.rsComm)}; if (resource_type != "passthru") { return SUCCESS(); } const boost::optional<uint64_t> max_bytes{ get_max_bytes(arg_resource_name, *rei.rsComm)};
Handle max_bytes edge cases
if (!max_bytes) { arg_out = "read=1.0;write=1.0"; return SUCCESS(); } else if (*max_bytes == 0) { arg_out = "read=1.0;write=0.0"; return SUCCESS(); }
Compute the write_weight
const uint64_t bytes_used_by_children{ get_bytes_used_by_all_children( arg_resource_name, *rei.rsComm)}; const uint64_t bytes_required_for_new_data_object{ get_bytes_of_incoming_data_object( arg_plugin_context)}; const uint64_t hypothetical_bytes_used{ bytes_used_by_children + bytes_required_for_new_data_object}; const double percent_used{ std::max(0.0, std::min(1.0, static_cast<double> (hypothetical_bytes_used) / *max_bytes))}; const double write_weight{1.0 - percent_used};
Wrapping up exec_rule
std::stringstream out_stream; out_stream << "read=1.0;write=" << write_weight_string; arg_out = out_stream.str(); return SUCCESS(); } catch (const irods::exception& e) { rodsLog(LOG_ERROR, e.what()); return ERROR(e.code(), "irods exception in exec_rule"); } return SUCCESS(); }
exec_rule_text and exec_rule_expression
irods::error exec_rule_text( irods::default_re_ctx&, std::string _rule_text, std::list<boost::any>& _rule_arguments, irods::callback _effect_handler) { return ERROR(SYS_NOT_SUPPORTED, "not supported"); } irods::error exec_rule_expression( irods::default_re_ctx&, std::string _rule_text, std::list<boost::any>& _rule_arguments, irods::callback _effect_handler) { return ERROR(SYS_NOT_SUPPORTED, "not supported"); }
Building and Installing the Package
sudo apt-get -y install irods-externals-* irods-dev
export PATH=/opt/irods-externals/cmake3.5.2-0/bin:/opt/irods-externals/clang3.8-0/bin:$PATH
which clang++
which cmake
mkdir ~/build_storage_cpp
cd ~/build_storage_cpp
cmake ../irods_training/advanced/irods_rule_engine_plugin_storage_balancing/
make package
sudo dpkg -i ./irods_rule_engine_plugin-cpp-storage-balancing_4.2.0~trusty_amd64.deb
Configuring the Rule Engine Plugin
Edit /etc/irods/server_config.json
{ "instance_name" : "irods_rule_engine_plugin-cpp-storage-balancing-instance", "plugin_name" : "irods_rule_engine_plugin-cpp-storage-balancing", "plugin_specific_configuration" : {} }, { "instance_name" : "irods_rule_engine_plugin-python-instance", "plugin_name" : "irods_rule_engine_plugin-python", "plugin_specific_configuration" : {} },
Run ils to determine if syntax is correct
Testing the Rule Engine Plugin
iput -R def_resc VERSION.json f1 iput -R def_resc VERSION.json f2 iput -R def_resc VERSION.json f3 iput -R def_resc VERSION.json f4 ils -l /tempZone/home/rods: rods 0 def_resc;pt2;ufs2 645 2016-06-03.20:10 & f1 rods 0 def_resc;pt1;ufs1 645 2016-06-03.20:10 & f2 rods 0 def_resc;pt2;ufs2 645 2016-06-03.20:10 & f3 rods 0 def_resc;pt1;ufs1 645 2016-06-03.20:10 & f4
Remove all data from def_resc
Questions?