Microservice Plugins

June 14, 2016

Public Health England

London, England

Jason M. Coposky

@jason_coposky

Interim Executive Director

The iRODS Plugin Architecture

iRODS 4.2 provides 7 plugin interfaces

  • microservices
  • resources
  • authentication
  • network
  • database
  • RPC API
  • rule engine

Anatomy of a Plugin

  • Built as a dynamic shared object
  • provides a plugin_factory method which instantiates the plugin object
  • contains a class derived from a plugin interface base class:
    • ​api_entry
    • auth
    • network
    • resource
    • database
    • ms_table_entry
    • pluggable_rule_engine<>


NOTE: microservices do not require a derived class, the factory can return an instance of ms_table_entry

Follow along in the irods_training repository

The code can be found at:

~/irods_training/advanced/irods_microservice_plugin/src/lib-microservice-example.cpp

Start with the Factory

 extern "C"
 irods::ms_table_entry* plugin_factory() {
     irods::ms_table_entry* msvc = new irods::ms_table_entry(3);
     msvc->add_operation<
     msParam_t*,
     msParam_t*,
     msParam_t*,
     ruleExecInfo_t*>(
         "msiexample_microservice",
         std::function<int(
         msParam_t*,
         msParam_t*,
         msParam_t*,
         ruleExecInfo_t*)>(msiexample_microservice));
     return msvc;
 }

Anatomy of the Plugin Factory

 extern "C"
 irods::ms_table_entry* plugin_factory() {
 
     ...

     return msvc;
 }
  • Must have C linkage
  • Must return an irods::plugin_base derived type
  • Must be named "plugin_factory"

Instantiating a microservice plugin

 extern "C"
 irods::ms_table_entry* plugin_factory() {
     irods::ms_table_entry* msvc = new irods::ms_table_entry(3);

     ...

     return msvc;
 }
  • allocate a raw pointer to irods::ms_table_entry
  • pass its constructor the number of parameters not including the ruleExecInfo_t*
  • Note: microservices do not use a derived class definition, unlike other plugin types

Wiring a plugin operation

...
     msvc->add_operation<
         msParam_t*,
         msParam_t*,
         msParam_t*,
         ruleExecInfo_t*>(
             "msiexample_microservice",
             std::function<int(
                 msParam_t*,
                 msParam_t*,
                 msParam_t*,
                 ruleExecInfo_t*)>(msiexample_microservice));
...
  • Template parameters are the parameters of the function operation
  • First parameter is the calling name of the operation - "msiexample_microservice"
  • Second parameter is a std::function wrapping the local function definition
    • Takes the full signature of the function as a template parameter
    • Takes the function pointer as an argument

The microservice definition

 int msiexample_microservice(
     msParam_t*      _string_param,
     msParam_t*      _int_param,
     msParam_t*      _double_param,
     ruleExecInfo_t* _rei ) {
 
     char *string_param = parseMspForStr( _string_param );
     if( !string_param ) {
         std::cout << "null _string_param" << std::endl;
         return SYS_INVALID_INPUT_PARAM;
     }

     int int_param = parseMspForPosInt( _int_param );
     if( int_param < 0 ) {
         std::cout << "invalid _int_param" << std::endl;
         return SYS_INVALID_INPUT_PARAM;
     }

     double double_param = 0.0;
     int ret = parseMspForDouble( _double_param, &double_param );
     if( ret < 0 ) {
         std::cout << "invalid _double_param" << std::endl;
         return SYS_INVALID_INPUT_PARAM;
     }

     std::cout << __FUNCTION__ << " string [" << string_param << "] int [" << int_param << "] double [" << double_param << "]" << std::endl;

     return 0;
}

The signature

 int msiexample_microservice(
     msParam_t*      _string_param,
     msParam_t*      _int_param,
     msParam_t*      _double_param,
     ruleExecInfo_t* _rei ) {

     ...

     return 0;
}
  • Returns an int
  • Takes N (per the ctor) number of msParam_t*
  • Must end in a ruleExecInfo_t*

Parameter parsing and Error Checking

...
     char *string_param = parseMspForStr( _string_param );
     if( !string_param ) {
         std::cout << "null _string_param" << std::endl;
         return SYS_INVALID_INPUT_PARAM;
     }

     int int_param = parseMspForPosInt( _int_param );
     if( int_param < 0 ) {
         std::cout << "invalid _int_param" << std::endl;
         return SYS_INVALID_INPUT_PARAM;
     }

     double double_param = 0.0;
     int ret = parseMspForDouble( _double_param, &double_param );
     if( ret < 0 ) {
         std::cout << "invalid _double_param" << std::endl;
         return SYS_INVALID_INPUT_PARAM;
     }
....
  • msParam_t* must be parsed into native types
  • Individual parseMspFor* exist for most iRODS types
  • Either the type is returned, if possible - or an error code

Highly complex application code

 int msiexample_microservice(
     msParam_t*      _string_param,
     msParam_t*      _int_param,
     msParam_t*      _double_param,
     ruleExecInfo_t* _rei ) {
 
     ...

     std::cout << __FUNCTION__ 
               << " string [" << string_param 
               << "] int [" << int_param 
               << "] double [" << double_param 
               << "]" << std::endl;

     return 0;
}

Many things are possible in a microservice

  • Calling server-side API endpoints - rsGenQuery
  • Manipulating data at rest
  • Return 0 for success

Building and Installing the Example Code

sudo apt-get -y install irods-externals-*

 

export PATH=/opt/irods-externals/cmake3.5.2-0/bin:/opt/irods-externals/clang3.8-0/bin:$PATH

 

 which clang++
/opt/irods-externals/clang3.8-0/bin/clang++

 which cmake
/opt/irods-externals/cmake3.5.2-0/bin/cmake

 

sudo apt-get install irods-dev

mkdir ~/build_msvc

cd ~/build_msvc

cmake ~/irods_training/advanced/irods_microservice_plugin

make package

sudo dpkg -i irods-microservice-example-4.2.0-ubuntu14-x86_64.deb

A rule to test the microservice

acPostProcForPut() {

    microservice_example("XXXX - test string", 314, 123.4);

    if("ufs_cache" == $rescName ) {
        writeLine( "serverLog", "XXXX - calling delayed replication" );
        delay("<PLUSET>1s</PLUSET><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF>") {
            *CacheRescName = "comp_resc;ufs_cache";
            msisync_to_archive("*CacheRescName", $filePath, $objPath );
        }
    }
}

Edit the /etc/irods/training.re rulebase

Ensure the iRODS Rule Language plugin is first

Edit /etc/irods/server_config.json

  • Delete any other rule engine plugins
        {   
            "instance_name" : "re-storage-balancing-instance",
            "plugin_name" : "re-storage-balancing",
            "plugin_specific_configuration" : {}
        },
        {   
            "instance_name" : "rule-engine-plugin-python-instance",
            "plugin_name" : "rule-engine-plugin-python",
            "plugin_specific_configuration" : {}
        },
        {   
            "instance_name": "re-irods-instance",
            "plugin_name": "re-irods",

 

Test the microservice

iput VERSION.json

 

  grep -dskip "XXXX -" ./log/*

./log/rodsLog.2016.06.XX:msiexample_microservice string [XXXX - test string] int [314] double [123.4]

Questions?

Public Health England - Microservice Plugins

By jason coposky

Public Health England - Microservice Plugins

  • 1,744