Taking Data To Compute

June , 2018

iRODS User Group Meeting 2018

Durham, NC

Daniel Moore

dmoore@renci.org

Application Engineer, iRODS Consortium

Taking Data To Compute

Integrating iRODS with a compute environment

In order of increasing complexity...

iRODS as a compute orchestrator

• Launch a job via irule, or as part of a PEP
• Implement a Landing Zone for product capture

iRODS as part of a compute job script

• Stage the source data via replication for the application
• Capture the products and ingest them into iRODS

iRODS as part of the compute application

• Compute application directly leverages the iRODS API to open, read, and write data

Goals - Develop generic interface concept for compute

• Develop a metadata-driven interface to drive path for input data and compute results. Utilize it to
  • push data to the proper storage resource
  • get a name for the host on which to launch compute job(s)

• Separate configuration from implementation
  • Keep deployment specifics in configuration files
  • Keep rule-base, scripts, and modules free of hard-wired values

Components of the System

System Component

Job Scheduler

Job Launching Script

Tools to Execute

Job Endpoint

Implementation

SLURM

bash

Image Magick convert

iRODS Rule Base

(user extension of the iRODS API)

and SLURM prolog / epilog

Getting Started

Get the irods_training repository and irods development,

and set up paths (run what's in red only if necessary)

cd "$HOME";git clone https://github.com/irods/irods_training
sudo apt-get -y install irods-externals-* irods-dev
export PATH=/opt/irods-externals/cmake3.5.2-0/bin:$PATH
echo 'export PATH=/opt/irods-externals/cmake3.5.2-0/bin:$PATH' >> ~/.bashrc

cd; mkdir build_data_to_compute
cd build_data_to_compute
cmake ../irods_training/advanced/hpc_data_to_compute/
make package
sudo dpkg -i ./irods-hpc-data-to-compute-example_4.2.3~xenial_amd64.deb

Build and Install MUNGE and SLURM (job scheduler)

Build and Install the Data to Compute package

cd ~/irods_training/advanced/hpc_data_to_compute/ubuntu_16 ;
./install_munge_and_slurm.sh

Package Contents

$ dpkg -c ./irods-hpc-data-to-compute-example_4.2.2~xenial_amd64.deb
drwxrwxr-x root/root         0 2018-05-31 09:38 ./etc/
drwxrwxr-x root/root         0 2018-05-31 09:38 ./etc/irods/
-r--r--r-- root/root       234 2018-05-31 06:58 ./etc/irods/data_to_compute.re 
                  [  ...  ]
-r--r--r-- root/root     11250 2018-05-31 02:12 ./var/lib/irods/compute/common.py
-r--r--r-- root/root       565 2018-05-31 06:40 ./var/lib/irods/compute/job_params.json
-r--r--r-- root/root      1441 2018-05-31 07:23 ./var/lib/irods/compute/util.py
-r--r--r-- root/root       523 2018-05-31 07:07 ./var/lib/irods/create_thumbnails.r
-r--r--r-- root/root       360 2018-05-31 02:12 ./var/lib/irods/find_thumbnails.r
drwxrwxr-x root/root         0 2018-05-31 09:38 ./var/lib/irods/msiExecCmd_bin/
-r-xr-xr-x root/root       571 2018-05-31 02:12 ./var/lib/irods/msiExecCmd_bin/convert.SLURM
-r-xr-xr-x root/root       343 2018-05-31 02:12 ./var/lib/irods/msiExecCmd_bin/submit_thumbnail_job.sh
-r--r--r-- root/root        74 2018-05-31 02:12 ./var/lib/irods/msiExecCmd_bin/thumbnail.submit
-r--r--r-- root/root       169 2018-05-31 07:18 ./var/lib/irods/put_data_to_compute.json
-r-xr--r-- root/root       511 2018-05-31 07:20 ./var/lib/irods/put_data_to_compute.py

Python Rule Engine Configuration (re-ordering)

Edit rule engine order in 
    /etc/irods/server_config.json :

• insert the python plugin configuration stanza after  the iRODS Rule Language plugin
• if it already exists elsewhere in the config, move (cut/paste) it from that location, but it must occur only once.
• for this exercise, native rule code must supercede python!

"rule_engines": [

    {
         "instance_name": "irods_rule_engine_plugin-irods_rule_language-instance",

                   ...

    },
    {
         "instance_name" : "irods_rule_engine_plugin-python-instance",
         "plugin_name" : "irods_rule_engine_plugin-python",
         "plugin_specific_configuration" : {}
    }, 
​​

Configure the LTS and Image Processing Resources

As the irods user:

Make two unix file system resources

iadmin mkresc lts_resc unixfilesystem `hostname`:/tmp/irods/lts_resc
iadmin mkresc img_resc unixfilesystem `hostname`:/tmp/irods/img_resc

Annotate them with appropriate metadata given their roles

  - defined in the configuration as part of the contract

imeta add -R lts_resc COMPUTE_RESOURCE_ROLE LONG_TERM_STORAGE
imeta add -R img_resc COMPUTE_RESOURCE_ROLE IMAGE_PROCESSING

cp ~/irods_training/stickers.jpg /tmp
sudo mkdir -p /tmp/irods/thumbnails /tmp/irods/{src,dst}
sudo chown -R irods:irods /tmp/irods

As the ubuntu user:

Stage data and destination directory for thumbnail creation

The configuration interface

Define interfaces for any necessary conventions

• Metadata attributes and values

• Naming conventions for logical and physical paths

• Metadata values for implemented roles

• Interface to job scheduler for launching compute

Single Point of Truth - allows for the use of the same 'end-points' for various metadata standards and naming conventions

Users may utilize metadata conventions to provide inputs to a given compute job

The configuration interface

For the thumbnail service we will need to

• Get the metadata attribute string that holds the role

• Get the tag for an Image Compute resource

• Get the tag for a Long Term Storage resource

• Get the logical collection name for thumbnails

• Get the physical path for a thumbnail

• Get the name of a thumbnail

• Get a list of desired thumbnail sizes

Python rule engine allows a cleaner system design

Writing rules in Python means easy access to  functionality and configuration data, both from the iRODS rule base:

import sys
sys.path.insert(0, "/var/lib/irods")
from compute.common import jobParams
def run_compute_jobs ( rule_args , callback , rei ):
  # ...
  dest_dir = jobParams() ['phys_dir_for_output'] 
  # ...   

and from python iRODS client scripts/modules:

def register_replicate_and_trim_thumbnail ( size_string ):

  # ...

  c = get_collection( jobParams()['output_collection'] )

The configuration interface

Provide an interface for job submission

submit_thumbnail_job(*server_host, *size_str, *src_phy_path, *dst_phy_path ) {
    remote(*server_host, "") {
        *cmd_opt = '/var/lib/irods/msiExecCmd_bin/convert.SLURM -thumbnail *size_str *src_phy_path *dst_phy_path'
        *err = errormsg(msiExecCmd(
                           "submit_thumbnail_job.sh",
                           *cmd_opt, "null", "null", "null", *std_out_err), *msg);
        msiGetStdoutInExecCmdOut(*std_out_err,*std_out);
        
        msiGetStderrInExecCmdOut(*std_out_err,*std_err);
       
        if(*err != 0) {
            writeLine( "serverLog", "FAILED: [*cmd_opt] [*err] [*msg]" );
            failmsg(*err,*cmd_opt)
        }
    } # remote
}

The configuration interface

Abstraction of job submission via shell script

#!/bin/bash

# $1 - executable
# $2 - thumbnail option
# $3 - sizing string
# $4 - source physical path
# $5 - destination physical path

SBATCH_OPTIONS="-o /tmp/slurm-%j.out"

SCRIPT="$1" # assume full path to executable

/usr/local/bin/sbatch $SBATCH_OPTIONS "$SCRIPT" \
    ${2+"$2"} \
    ${3+"$3"} \
    ${4+"$4"} \
    ${5+"$5"} \
    >/dev/null 2>&1

Thumbnail Service - testing

irods@icat:~$ iput /tmp/stickers.jpg
irods@icat:~$ ils -l
/tempZone/home/rods:
  rods              0 demoResc      2157087 2018-03-22.18:42 & stickers.jpg

irods@icat:~$ irule -F create_thumbnails.r
irods@icat:~$ ils -l
/tempZone/home/rods:
  rods              0 demoResc      2157087 2018-03-22.18:42 & stickers.jpg
  rods              1 img_resc      2157087 2018-03-22.18:43 & stickers.jpg
  C- /tempZone/home/rods/stickers_thumbnails

irods@icat:~$ iqstat

...

irods@icat:~$ ils -l /tempZone/home/rods/stickers_thumbnails
/tempZone/home/rods/stickers_thumbnails:
  rods              1 lts_resc       229954 2018-03-22.18:43 & stickers_thumbnail_1024x1024.jpg
  rods              1 lts_resc         6456 2018-03-22.18:43 & stickers_thumbnail_128x128.jpg
  rods              1 lts_resc        19355 2018-03-22.18:43 & stickers_thumbnail_256x256.jpg
  rods              1 lts_resc        63036 2018-03-22.18:43 & stickers_thumbnail_512x512.jpg

irods@icat:~$ irule -F find_thumbnails.r
thumbnail [/tempZone/home/rods/stickers_thumbnails/stickers_thumbnail_1024x1024.jpg]
thumbnail [/tempZone/home/rods/stickers_thumbnails/stickers_thumbnail_512x512.jpg]
thumbnail [/tempZone/home/rods/stickers_thumbnails/stickers_thumbnail_256x256.jpg]
thumbnail [/tempZone/home/rods/stickers_thumbnails/stickers_thumbnail_128x128.jpg]

Extending iRODS with the Rule Engine

• All rules should be created and tested in user space before being installed as a rule base

• Rules may be refactored into a microservice plugin

• Rules may be refactored into a C++ rule engine plugin

• Rules may be refactored into an API plugin