August 3-6, 2020
KU Leuven Training
Webinar Presentation
Daniel Moore
Applications Engineer, iRODS Consortium
The Python Rule Engine
The Python Rule Engine
Writing iRODS Rules in Python
With the Python Rule Engine Plugin loaded and configured:
- Python functions within the /etc/irods/core.py module's namespace can be called as iRODS rules.
- If function names correspond to the conventional names of static (old-style) or dynamic (new-style) Policy Enforcement Points, we can program iRODS system policy directly in Python (see the default core.py.template file for examples)
Translating basic iRODS Rule Language Examples
A Static (Legacy) Policy Enforcement Point:
The Python version:
acPostProcForPut() {
if("ufs_cache" == $KVPairs.rescName) {
delay("<PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>") {
*CacheRescName = "comp_resc;ufs_cache";
msisync_to_archive("*CacheRescName", $filePath, $objPath );
}
}
}import session_var
def acPostProcForPut(rule_args, callback, rei):
Map = session_vars.get_map(rei)
Kvp = { str(a):str(b) for a,b in Map['key_value_pairs'].items() }
if 'ufs_cache' == Kvp['rescName']:
callback.delayExec(
("<PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>" +
"<INST_NAME>irods_rule_engine_plugin-python-instance</INST_NAME>" ),
"callback.msisync_to_archive('{cacheResc}','{file_path}','{object_path}')".format(
cacheResc='comp_resc;ufs_cache',**Map['data_object'] ),
"")Translating basic iRODS Rule Language Examples
Dynamic Policy Enforcement Point:
The Python Version:
pep_resource_resolve_hierarchy_pre(
*INST_NAME,*CTX,*OUT,*OP_TYPE,*HOST,*RESC_HIER,*VOTE){
if( "CREATE" == *OP_TYPE ) {
if( "pt1" == *INST_NAME) {
*OUT = "read=1.0;write=0.5"
}
else if ( "pt2" == *INST_NAME ) {
*OUT = "read=1.0;write=1.0"
}
}
}def pep_resource_resolve_hierarchy_pre (rule_args, callback, rei):
(INST_NAME, CTX, OUT, OP_TYPE, HOST, RESC_HIER, VOTE) = rule_args
if "CREATE" == OP_TYPE :
if "pt1" == INST_NAME:
rule_args[2] = "read=1.0;write=0.5"
elif "pt2" == INST_NAME:
rule_args[2] = "read=1.0;write=1.0"Composing a Rule-base in Python
Creating an iRODS rule in Python
- Rules are defined in top-level module /etc/irods/core.py or imported there from other modules
- By convention they are declared with three input parameters: rule_args, callback, and rei.
import helpers # --> utility module, also in /etc/irods
def top_level_rule(rule_args, callback, rei ):
input_ = rule_args[0]
rv = callback.get_complex_result( input_ ) # Execute another rule.
rule_output = rv['arguments'][0] # --> get that rule's output ...
end_result = helpers.to_fortran(complex(rule_output)) # --> Format complex value for use in FORTRAN
rule_args[0] = end_result # --> and return the resulting string.
#import another set of rules
from rulebase1 import *
Below is an example rule in Python. In the course of its execution, it calls a different rule called "get_complex_result"
Composing a Rule-base in Python
The parameters of the rule signature are as follows:
rule_args: Conveys rule arguments into, and results out of, the rule.
callback: Allows calling other rules and microservices via the framework.
rei : The rule execution instance; contains session variables. (As seen in the static PEP).
__all__ = ['get_complex_result'] #minimize core.py namespace pollution
import helpers
def get_complex_result (rule_args, callback, rei):
radian_angle = float(rule_args[0])
result = str(helpers.rotate_unit_phasor(radian_angle))
callback.writeLine ("serverLog", "Result was: " + result)
rule_args[0] = result
Suppose that we make the auxiliary rule get_complex_result a Python rule as well, and we place it in /etc/irods/rulebase1.py:
import cmath # -- any standard Python library module can be used
def to_fortran (cplx): return "({0.real},{0.imag})".format(cplx)
def rotate_unit_phasor(radn): return cmath.exp(1j * radn)And then our utility module /etc/irods/helpers.py which contains only traditional, "direct-call" python functions can be very simple:
Composing a Rule-base in Python
Finally we can test our rulebase with the following launch script call_top_level.r
def main(rule_args, callback, rei):
x = global_vars ['*x'][1:-1]
returned = callback.top_level_rule(x)
callback.writeLine( "stdout", "call to top_level_rule gives:" +
str( returned['arguments'][0] )
)
input *x="3.1416"
output ruleExecOutUsing this irule command:
irule -r irods_rule_engine_plugin-python-instance -F top_level_rule.r
Calling from Python rules into Native rules
As with Native rule language, we can call iRODS microservices from a Python rule. If we have a Python rule:
def myrule (rule_args, callback, rei ):
arg0 = str( rule_args [0] )
return_val = callback.error_msg_from_index (arg0)
msg = return_val ['arguments'] [0]
rule_args [0] = ("System Code {} - {}".format(arg0,msg))
error_types() { list("Unsupported operation", "Syntax", "Runtime") }
error_msg_from_index (*io) { *io = elem(error_types(), int(*io)) }
And the following is defined in the native rulebase /etc/irods/core.re:
$ irule -r irods_rule_engine_plugin-irods_rule_language-instance \
"*x='1'; myrule(*x) writeLine('stderr',*x)" null ruleExecOut
System Code 1 - Syntax
We can run the Python rule as follows: (Note that, even targeting the iRODS rule language, the call to myrule can still resolve properly via the rule framework):
Calling from Native rules into Python rules
As with Native rule language, we can call iRODS microservices from a Python rule. If we have a Python rule:
def math_function ( rule_args, callback, rei):
import math
funcname = str( rule_args [0] ) #e.g. "sin", atan","sqrt"
argument = rule_args [1]
fHandle = getattr(math, funcname )
result = fHandle (float(argument))
rule_args [1] = str(result)do_math(*fn,*x) {
*intm = str(eval(*x))
*y= math_function(*fn, *intm)
double(*intm)
}
$ irule -r irods_rule_engine_plugin-irods_rule_language-instance \
" writeLine('stdout',do_math('cos','355.0/452.0')" null ruleExecOut
0.707107
And a native rule:
We see that Python rules can add utility (even system calls) to the Native rule language , much like microservices:
Calling from Python rules into iRODS microservices
As with Native rule language, we can call iRODS microservices from a Python rule. Call this rule script test_msvc.r
from irods_types import (objType, char_array, RodsObjStat)
def main(rule_args,callback,rei): # <-- rule under test
path = rule_args[0] if rule_args else global_vars['*x'][1:-1]
description = ""
st = RodsObjStat()
try:
rv = callback.msiObjStat(path,st)
st = rv['arguments'][1]
except RuntimeError: #if the microservice fails
callback.writeLine("stderr","[msiObjStat failed on path '{}'".format(path))
else:
if st.objType == objType.DATA_OBJ:
description = 'data object; size = ' + str(st.objSize)
elif st.objType == objType.COLL_OBJ:
description = 'collection'
callback.writeLine("stderr","object type ["+description+"]")
if len(rule_args) > 0: rule_args[0] = description
INPUT *x=$"/tempZone/home/rods"
OUTPUT ruleExecOut
$ irule -r irods_rule_engine_plugin-irods_rule_language-instance -F test_msvc.r null ruleExecOutAgain using irule (you'll have the option of changing *x if inaccurate):
Calling from Python rules into iRODS microservices
Things to note with regard to the test script:
- While under test, our Python rule must be named 'main'.
- Rule and microservice calls can always fail, so we anticipate failure by wrapping such calls in a try/except clause.
- Input variables ( *x in this example ) are accessible through the global_vars dictionary object. This is only true in a Python rule script, not within the rule-base proper.
- Rule scripts may only be executed by a rodsadmin user. This is because the default python interpreter gives us access to the filesystem and potentially other system resources required by iRODS.
After the main rule is tested to our satisfaction , we can rename and relocate it to our permanent rulebase.
Failing in Python rules
As in Native rules, in Python we have the option of issuing a fail condition. This is done by raising RuntimeError.
Place the following rule code into /etc/irods/core.py :
from irods_types import (objType, char_array, RodsObjStat)
def testy(rule_args,callback,rei): # <-- rule under test
if int( rule_args[0]) % 2 : raise RuntimeError( 'Not an even number') # fail
# exit naturally, ie success$ irule -r irods_rule_engine_plugin-irods_rule_language-instance \
"writeLine('stdout', errorcode( testy('*x') ))" "x=5" ruleExecOut And try it out using irule :
Python REP's Built-in Utility Modules
session_vars
- exports a get_map function for access to session variables stored in the third rule parameter, rei
- These correspond to the session variables in the legacy rule language, ie the "dollar-variables" listed in /etc/irods/core.dvm or (as in the earlier acPostProcForPut example) the fields of the $KVPairs structure .
import session_vars
def pep_resource_create_post (rule_args, callback, rei):
# get info from Rule Execution Instance
svars = session_vars.get_map( rei )
proxy_user = svars['proxy_user']['user_name']
callback.writeLine("serverLog","proxy_user "+ proxy_user)
# get info from PluginContext object
d=(rule_args[1].map())
callback.writeLine("serverLog","physical_path "+ d['physical_path'])
An example usage:
Python REP's Built-in Utility Modules
genquery
- exports a Query class with similar functionality to the Language Integrated Query in the native rule language.
- By default result rows are packed as tuples.
from genquery import Query
def data_ids_from_file_extension (rule_args,callback,rei):
ext = rule_args[0]
mylist = []
for data_id in Query(callback,
"DATA_ID", "DATA_NAME like '%.{ext}'".format(**locals)):
mylist.append(data_id)
rule_args[0] = ",".join(mylist) from genquery import Query, AS_DICT
# ...
# Essential arguments are: (1) callback, (2) selected columns, (3) condition, (4) row_type
for row in Query(callback, #(1)
[ "DATA_NAME", "COLL_NAME", "META_DATA_ATTR_VALUE"], #(2)
("META_DATA_ATTR_NAME = "
" 'irods::netcdf::group_{}_variable_{}".format(group,var)), #(3)
AS_DICT #(4)
):
key = "{COLL_NAME}/{DATA_NAME}".format.(**row)
subhashw = netcdf_vars.setdefault( key, {} )
subhash[group,var] = row["META_DATA_ATTR_VALUE"]
Example with dictionary result rows and "auto-joined" tables:
Sample equivalent rules in Native Rule Lang vs Python
Sample rule in the Legacy iRODS Rule Language:
verify_replica_placement(*violations) {
*attr = verify_replicas_attribute
# get a list of all matching collections given the metadata attribute
foreach(*row0 in SELECT COLL_NAME, META_COLL_ATTR_VALUE WHERE META_COLL_ATTR_NAME = "*attr") {
*resource_list = *row0.META_COLL_ATTR_VALUE
*number_of_resources = size(split(*resource_list, ","))
*coll_name = *row0.COLL_NAME
foreach(*row1 in SELECT COLL_NAME, DATA_NAME WHERE COLL_NAME like "*coll_name%") {
*matched = 0
*coll_name = *row1.COLL_NAME
*data_name = *row1.DATA_NAME
foreach(*row2 in SELECT RESC_NAME WHERE COLL_NAME = "*coll_name" AND DATA_NAME = "*data_name") {
*resource_name = *row2.RESC_NAME
*split_list = split(*resource_list, ",")
while(size(*split_list) > 0) {
*name = str(hd(*split_list))
*split_list = tl(*split_list)
*name = triml(trimr(*name,' '),' ')
# ( set write permission for collaborator - ?? )
if(*name == *resource_name) { *matched = *matched + 1 }
}
} # for resources
if(*matched < *number_of_resources) {
*violations = cons("*coll_name/*data_name violates the placement policy "++*resource_list,*violations)
}
} # for objects
} # for collections
} # verify_replica_placementImplementing the replica placement policy
The rule translated into Python :
__all__ = ['verify_replica_placement']
from genquery import Query, AS_DICT
from irods_capability_integrity_utils import *
def verify_replica_placement (rule_args, callback, rei):
violations = split_text_lines( rule_args[0] )
attr = verify_replicas_attribute()
for row0 in Query (callback, ['COLL_NAME','META_COLL_ATTR_VALUE'],
"META_COLL_ATTR_NAME = '{}'".format(attr), AS_DICT):
resource_list = row0 ['META_COLL_ATTR_VALUE']
split_resource_list = set(r.strip() for r in resource_list.split(","))
number_of_resources = len( split_resource_list )
coll_name = row0 ['COLL_NAME']
for row1 in Query (callback, ['COLL_NAME','DATA_NAME'],
"COLL_NAME like '{}%'".format( coll_name ), AS_DICT):
matched = 0
coll_name = row1['COLL_NAME']
data_name = row1['DATA_NAME']
for row2 in Query (callback, ['RESC_NAME'],
"COLL_NAME = '{0}' AND DATA_NAME = '{1}'".format(coll_name,data_name), AS_DICT):
resource_name = row2['RESC_NAME']
# set modify for all collaborators - ?
for r in split_resource_list:
# set write permission for collaborator - ?
if r == resource_name:
matched += 1
# -- end -- for resources
if matched < number_of_resources:
violations.append("Object {coll_name}/{data_name} violates the replica replacement policy".format(**locals()))
# -- end -- for objects
# -- end -- for collections
rule_args [0] = join_text_lines( violations )
Common utility functions for the Python rule base:
/etc/irods/irods_capability_integrity_utils.py
def get_error_value () : return "ERROR_VALUE"
def RULE_ENGINE_CONTINUE () : return 5000000
def SYS_INVALID_INPUT_PARAM () : return -130000
def verify_replicas_attribute() : return "irods::verification::replica_placement"
def verify_checksum_attribute() : return "irods::verification::replica_checksum"
def verify_replica_number_attribute() : return "irods::verification::replica_number"
def TRUE(): return "true"
def FALSE(): return ""
def split_text_lines( string_in ) :
return filter (None, string_in.split("\n"))
def join_text_lines( list_in ) :
return "\n".join(filter(None, list_in)) + "\n"
Setting up to test the Python rulebase
Place imports for the rule modules containing our python rules, near the near of the core (/etc/irods/core.py) module:
Stage the Python modules from the repo directory (and make sure we've removed the native rulesets from /etc/irods/server_config.json):
From the repo directory:
sudo cp *.py /etc/irods/.
from verify_replica_number import *
from verify_replica_placement import *
from verify_checksum import *Check out the code from:
Testing the replica placement policy
The Python version of the rule launch script is as follows:
def main (rule_args, callback, rei):
from irods_capability_integrity_utils import (split_text_lines, )
violations = ""
retval = callback.verify_replica_placement(violations)
violations = retval ['arguments'][0]
for v in split_text_lines(violations):
callback.writeLine("stdout", v)
INPUT null
OUTPUT ruleExecOutirule -r irods_rule_engine_plugin-python-instance \
-F execute_replica_placement_policy_Py.r
Test the Python replica placement policy as shown:
Questions?
Python Rule Engine
By Daniel Moore
Python Rule Engine
Python Rule Language Training Module
