8. Workflow Automation
Rules
Microservices
Examination of an example
Rule Language Overview
Rules
Rules - C-like scripts executed by the iRODS Rule Engine
Rule Engine - integrated language interpreter
Simple Example:
HelloWorld {
writeLine("stdout", "Hello World");
}
Rule Execution
Workflow may be automated in several ways
- irule - initiated by a user
- Policy Enforcement Points - initiated automatically
- Delayed Execution Queue - initiated asynchronously
Microservices
C++ plugins bound into the Rule Engine as functions
Useful for
- leveraging specialized libraries - curl, gdal, etc
- tasks which are computationally intensive
Microservices
Invoked like a function call
HelloWorld2 {
msihello_world( *msg );
}
INPUT *msg = "my message"
OUTPUT ruleExecOut
Installation
Packaged Microservices - installed via native packaging system ( rpm, dpkg )
- Installed into /var/lib/irods/plugins/microservices
Hand-Built Microservices - copied by hand into the proper plugin directory
- /var/lib/irods/plugins/microservices
- $IRODS_HOME/plugins/microservices
Note - for Run-In-Place installations $IRODS_HOME will be wherever the system was built and deployed
Deploying Rules for Policy Enforcement Points
Directly edit and add rule code to the PEP defined in the default rule base:
/etc/irods/core.re
Override the Policy Enforcement Point -
include a new file within the rule base containing the desired rule code
Note - The rule base is configured in server_config.json
The Example Rule - command line demo
Goals
- given - training_acPostProcForPut.re and msiget_image_meta are installed
- add training_acPostProcForPut to /etc/irods/server_config.json
- iput the training_jpg directory into iRODS
- examine the harvested metadata with imeta
Examining the rule file - defining the PEP
Found at - /etc/irods/training_acPostProcForPut.re
First we will define the PEP for the rule definition:
acPostProcForPut {
This PEP takes no parameters, so does not need parenthesis
Examining the rule file - filter file types
We only want to harvest metadata from image files - filter using an 'if' statement
Session Variable - global variables holding values about the data object in flight
$filePath - session variable holding the physical path
if( $filePath like "*.jpg" || $filePath like "*.jpeg" || $filePath like "*.bmp" || $filePath like "*.tif" || $filePath like "*.tiff" || $filePath like "*.rif" || $filePath like "*.gif" || $filePath like "*.png" || $filePath like "*.svg" || $filePath like "*.xpm") {
Examining the rule file - extract the metadata
Once we have filtered the file type
- invoke the microservice to harvest the metadata
- metadata is encoded as a string in the 'out variable' *meta
msiget_image_meta($filePath, *meta);
Examining the rule - applying the metadata
The metadata encoded string is converted to an internal iRODS key-value data structure in the 'out variable' *meta_kvp
msiString2KeyValPair(*meta, *meta_kvp);
Once we have the key-value pairs we apply them to our data object
msiAssociateKeyValuePairsToObj(*meta_kvp, $objPath, "d");
The Data object is referenced by the session variable $objPath which is the logical iRODS path
The "d" signifies to the microservice that we are referencing a Data Object, not a Collection or Resource
Examining the rule - closing the scope
Now that metadata is applied we need to close our two scope blocks - one for the 'if' and the one for the PEP itself -
} # if } # acPostProcForPut
Examining the rule - the final product
acPostProcForPut { if( $filePath like "*.jpg" || $filePath like "*.jpeg" || $filePath like "*.bmp" || $filePath like "*.tif" || $filePath like "*.tiff" || $filePath like "*.rif" || $filePath like "*.gif" || $filePath like "*.png" || $filePath like "*.svg" || $filePath like "*.xpm") { msiget_image_meta($filePath, *meta); msiString2KeyValPair(*meta, *meta_kvp); msiAssociateKeyValuePairsToObj(*meta_kvp, $objPath, "-d"); } # if } # acPostProcForPut
Rule Language Overview
A short description can be found at the end of this section in the training manual
The full description can be found at -
https://docs.irods.org/4.1.2/manual/rule_language/
UGM2015-Training - Workflow Automation
By jason coposky
UGM2015-Training - Workflow Automation
Introduction to Workflow Automation with example rule and metadata extraction microservice
- 1,571