Advanced Training:
Compute to Data
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826819/hex_title_page_bg_2020_widescreen.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/10541392/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/10541393/pasted-from-clipboard.png)
May 28-31, 2024
iRODS User Group Meeting 2024
Amsterdam, Netherlands
Alan King, Senior Software Developer
Martin Flores, Software Developer
iRODS Consortium
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/6311866/renci_logo_white.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/11314881/pasted-from-clipboard.png)
The Compute to Data Use Case
![](https://s3.amazonaws.com/media-p.slid.es/uploads/320802/images/3854018/simple_compute_to_data_diagram.jpg)
Data is assumed to already be routed to an appropriate storage resource
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Goals - Develop generic interface concept for compute
- Develop a metadata driven interface for labeling resources which provide computational capabilities
- ultimately relies upon convention
- Separate configuration from implementation
- isolate deployment specific concepts
- Consider a rule base as an extension of iRODS
- rules are not just data management policy
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
"Compute To Data" Pattern - Salient Features
Implemented as an iRODS rulebase -
following the Template Method pattern
- If necessary, replicate input data to an appropriate resource
- Check permissions
- Launch compute container (Docker) :
- Process input data via Jupyter notebook
- Save results
- Register the resultant directory into iRODS
- Apply metadata to newly registered results
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Components of the System
System Component
Job Initialization
Container Technology
User Provided Compute
Implementation
iRODS Rule Base
Docker
Jupyter Notebook
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Getting Started
git clone https://github.com/irods/irods_training
sudo apt-get -y install \
irods-externals-cmake3.21.4-0 \
irods-externals-clang13.0.1-0 \
irods-externals-qpid-proton0.36.0-2 \
irods-externals-fmt8.1.1-1 \
irods-dev
export PATH=/opt/irods-externals/cmake3.21.4-0/bin:$PATH
Clone irods_training repository and configure build tools
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" sudo apt-get update sudo apt-get install -y docker-ce sudo usermod -aG docker ${USER}
Install Docker
Getting Started
cd mkdir build_compute_to_data cd build_compute_to_data cmake ../irods_training/advanced/hpc_compute_to_data make package sudo dpkg -i irods-hpc-compute-to-data-example.deb cd mkdir build_register_microservice cd build_register_microservice cmake ../irods_training/advanced/hpc_compute_to_data/msvc__msiregister_as_admin/ make package sudo dpkg -i irods-microservice-register_as_admin-4.3.2-ubuntu22-x86_64.deb
Install packages for the compute-to-data example
docker pull irods/irods-training-jupyter-digital-filter
Pull Docker image for processing
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Getting Started - Python extensions to iRODS
sudo apt-get -y install python3-pip
Also, install python's pip package:
Make sure the Python rule engine plugin is installed.
sudo apt-get -y install irods-rule-engine-plugin-python
python3 -m pip install docker==7.0.0 --user python3 -m pip install requests==2.31.0 --user
As service account user irods, install the Python Docker API. Manually downgrade requests due to bug in a recent release.
Add irods user to the docker group:
Restart the iRODS server:
sudo su irods -c '~/irodsctl restart'
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
sudo usermod -aG docker irods
Further Setup and Configuration
Place Python Rule Engine stanza after irods_rule_engine_plugin stanza:
Create /etc/irods/core.py with the following import:
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
from compute_to_data import *
"rule_engines": [
{
"instance_name": "irods_rule_engine_plugin-irods_rule_language-instance",
"plugin_name": "irods_rule_engine_plugin-irods_rule_language",
...
"shared_memory_instance": "irods_rule_language_rule_engine"
},
{
"instance_name": "irods_rule_engine_plugin-python-instance",
"plugin_name": "irods_rule_engine_plugin-python",
"plugin_specific_configuration": {}
},
...
Continued... Data-to-Compute Set-up / Configuration
iadmin mkuser alice rodsuser iadmin moduser alice password apass
This demonstration will be run as rodsuser 'alice'
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Configure the Tagged Resources - if necessary
Make a unixfilesystem resource for the SIGNAL_PROCESSING role.
iadmin mkresc dsp_resc unixfilesystem $(hostname):/tmp/irods/dsp_resc
Annotate it with appropriate metadata given its role
- defined in the configuration as part of the contract
As the irods service account
imeta add -R dsp_resc COMPUTE_RESOURCE_ROLE SIGNAL_PROCESSING
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Finally ...
$ iinit ERROR: environment_properties::capture: missing environment file. should be at [/home/ubuntu/.irods/irods_environment.json] Enter the host name (DNS) of the server to connect to: localhost Enter the port number [1247]: 1247 Enter your irods user name: alice Enter your irods zone: tempZone Enter your current iRODS password:
$ ils
/tempZone/home/alice:
Remember to log in as 'alice' in the ubuntu training account:
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
The configuration interface
Define interfaces for any necessary conventions
- Metadata attributes and values
- Metadata values for implemented roles
Single Point of Truth - Template Method Pattern
- execute defined preconditions
- run user's requested container
Users may utilize metadata conventions within a rule to provide inputs to the generalized container service.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Reminder ...
Implemented as an iRODS rulebase -
following the Template Method pattern
- If necessary, replicate input data to an appropriate resource
- Check permissions
- Launch compute container (Docker) :
- Process input data via Jupyter notebook
- Save results
- Register the resultant directory into iRODS
- Apply metadata to newly registered results (but not today...)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Separation of Concerns
iRODS Rule -> Python Rule -> Docker Container -> Jupyter Notebook
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
The iRODS Rule Language Rule File
main { container_dispatch("containers.run","/tempZone/home/alice/task_config.json","dsp_resc","","") } INPUT null OUTPUT ruleExecOut
Note - add a delay() directive for asynchronous behavior.
Contents of /home/ubuntu/spawn_remote_containers.r
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
The Python Rulebase
Located at:
/home/ubuntu/irods_training/advanced/hpc_compute_to_data/compute_to_data.py
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
The rulebase:
- performs pre-flight checks on the input data
- launches the Docker container (Jupyter notebook)
- registers the results in iRODS
The Digital Signal Processing container
FROM jupyter/base-notebook USER root COPY lpfilter.ipynb /home/jovyan/work/. COPY mymodule/ /home/jovyan/work/mymodule/ USER jovyan RUN conda init RUN conda install -y -c conda-forge matplotlib numpy RUN jupyter trust /home/jovyan/work/lpfilter.ipynb RUN mkfifo /tmp/fifo CMD cat /tmp/fifo
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Dockerfile can be found at:
~/irods_training/advanced/hpc_compute_to_data/jupyter_notebook/Dockerfile
The Jupyter Notebook
Located at:
/home/ubuntu/irods_training/advanced/hpc_compute_to_data/jupyter_notebook/lpfilter.ipynb
The notebook:
- loads an input waveform
- applies a digital lowpass filter
- plots 3 graphs of the results
- saves graphs and filtered data
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Compute to Data - Digital Filter Testing
ubuntu $ icd ; imkdir notebook_input notebook_output ubuntu $ cd ; iput task_config.json ubuntu $ for x in {1..512}; do echo $((x%24)) ; done >input.dat ubuntu $ iput input.dat notebook_input ubuntu $ ils -lr /tempZone/home/alice: alice 0 demoResc 853 2024-06-08.20:27 & task_config.json C- /tempZone/home/alice/notebook_input /tempZone/home/alice/notebook_input: alice 0 demoResc 1318 2024-06-08.20:27 & input.dat C- /tempZone/home/alice/notebook_output /tempZone/home/alice/notebook_output: ubuntu $ irule -r irods_rule_engine_plugin-irods_rule_language-instance -F spawn_remote_containers.r ubuntu $ ils -lr /tempZone/home/alice: alice 0 demoResc 853 2024-06-08.20:27 & task_config.json C- /tempZone/home/alice/notebook_input /tempZone/home/alice/notebook_input: alice 0 demoResc 1318 2024-06-08.20:27 & input.dat alice 1 dsp_resc 1318 2024-06-08.20:28 & input.dat C- /tempZone/home/alice/notebook_output /tempZone/home/alice/notebook_output: alice 0 dsp_resc 0 2024-06-08.20:28 & .fe6cc526-063a-11ee-912b-377adb02dba0 C- /tempZone/home/alice/notebook_output/fe6cc526-063a-11ee-912b-377adb02dba0 /tempZone/home/alice/notebook_output/fe6cc526-063a-11ee-912b-377adb02dba0: alice 0 dsp_resc 0 2024-06-08.20:28 & .fe6cc526-063a-11ee-912b-377adb02dba0 alice 0 dsp_resc 746805 2024-06-08.20:28 & lowpass_filter_processing.html alice 0 dsp_resc 3200 2024-06-08.20:28 & lowpass_filtered_input.dat
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Compute to Data - Digital Filter Results
sudo su - irods cd /tmp/irods/dsp_resc/home/alice/notebook_output python3 -m http.server 8888
![](https://s3.amazonaws.com/media-p.slid.es/uploads/842539/images/6283541/2019-06-21.png)
Open in web browser
View the html file
Start a simple http server to view the output
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
Questions?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/336929/images/7826818/hex_inner_page_bg_2020_widescreen.jpg)
UGM 2024 - Compute to Data
By iRODS Consortium
UGM 2024 - Compute to Data
iRODS User Group Meeting 2024 - Advanced Training Module
- 192