Getting Started
Getting Started
Jason M. Coposky
@jason_coposky
Executive Director, iRODS Consortium
January 14-16, 2020
CINES
Montpellier, France
Little Slips of Paper & Dependencies
curl -LO https://github.com/irods/irods_training/raw/ugm2019/training.pem
chmod 600 training.pem
ssh -i training.pem ubuntu@#.#.#.#
Logging in to your VM:
Install iRODS Build Requirements:
sudo apt-get update
sudo apt-get -y install git g++ make python-dev help2man unixodbc libfuse-dev libcurl4-gnutls-dev libbz2-dev zlib1g-dev libpam0g-dev libssl-dev libxml2-dev libkrb5-dev unixodbc-dev libjson-perl python-psutil python-jsonschema super python-exif odbc-postgresql unzip
Little Slips of Paper & Dependencies
Windows (using Putty):
Little Slips of Paper & Dependencies
Open Putty
Little Slips of Paper & Dependencies
Enter Username
Little Slips of Paper & Dependencies
Select Private Key
C:\Users\Student\Desktop\training20
Little Slips of Paper & Dependencies
Enter Host Name (or IP address)
Name Session
Save Session
Open Connection
Little Slips of Paper & Dependencies
First-Run
Security Alert
Little Slips of Paper & Dependencies
Google Chrome can also connect via "Secure Shell" extension:
https://chrome.google.com/webstore/detail/secure-shell/pnhechapfaindjhompbnflcldabbghjo
Download public and private keys:
https://github.com/irods/irods_training/raw/ugm2018/irodstraining
https://github.com/irods/irods_training/raw/ugm2018/irodstraining.pub
Then:
Acquire the Prerequisites
Clone the training repository:
git clone https://github.com/irods/irods_training
Install and configure Postgres
sudo apt-get -y install postgresql
sudo su - postgres
As the ubuntu user
psql
CREATE DATABASE "ICAT";
CREATE USER irods WITH PASSWORD 'testpassword';
GRANT ALL PRIVILEGES ON DATABASE "ICAT" to irods;
\q
exit
As the postgres user
Configure the Repository, Install, Run setup
wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add - echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" | \ sudo tee /etc/apt/sources.list.d/renci-irods.list sudo apt-get update
sudo apt-get -y install irods-server irods-database-plugin-postgres
Install public key and add repository:
Install from repository:
Read more: https://packages.irods.org
sudo python /var/lib/irods/scripts/setup_irods.py < /var/lib/irods/packaging/localhost_setup_postgres.input
Run setup with provided input file:
Setup iRODS for Auditing
Before we continue with the training, we are going to set up some auditing in iRODS so that we can report on all of the activities in our iRODS instance.
At the end of today's training, we will revisit this and visualize what has happened in iRODS throughout the day.
The first action is to install the auditing plugin...
sudo apt-get -y install irods-rule-engine-plugin-audit-amqp
Setup the iRODS Audit Plugin
Edit /etc/irods/server_config.json
add a new stanza to the rule_engines array after the irods_rule_engine_plugin-cpp_default_policy-instance definition
add the audit namespace
"rule_engines": [
{
"instance_name": "irods_rule_engine_plugin-cpp_default_policy-instance",
"plugin_name": "irods_rule_engine_plugin-cpp_default_policy",
"plugin_specific_configuration": {}
},
{
"instance_name": "irods_rule_engine_plugin-audit_amqp-instance",
"plugin_name": "irods_rule_engine_plugin-audit_amqp",
"plugin_specific_configuration" : {
"amqp_location" : "ANONYMOUS@localhost:5672",
"amqp_options" : "",
"amqp_topic" : "audit_messages",
"pep_regex_to_match" : "audit_.*"
}
}
...
...
"rule_engine_namespaces": [
"",
"audit_"
],
Setup Monitoring
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y docker-ce
sudo usermod -aG docker ${USER}
The iRODS plugin produces AMQP messages for each dynamic policy enforcement point.
We will now setup a docker container which will accept these messages, store them in an Elastic database, and provide a visualization web tool.
First we need to install docker.
After doing this log out of your virtual machine and log back in to make sure your group list is updated.
Setup Monitoring
docker pull irods/irods_audit_elk_stack docker run -p 8080:15672 -p 5672:5672 -p 80:5601 -p 9200:9200 -dit irods/irods_audit_elk_stack
From within your virtual machine,
download the docker image and run it.
You now have a docker container instance running within your virtual machine.
This instance is running the following:
RabbitMQ - Message broker that stores the AMQP messages
Elasticsearch - Database that stores the AMQP messages.
Logstash - Reads message from RabbitMQ and writes them to Elasticsearch
Kibana - A data visualization plugin for ElasticSearch
Configure the visualization
Port 80 on your VM is now mapped to the Kibana web tool.
Let's configure Kibana.
In a web browser, navigate to http://#.#.#.#/ to open the Kibana dashboard.
Click on Management (left pane) -> Index Patterns.
Type "irods_audit" in the index pattern field and click next step.
Select "@timestamp" in the time filter field name and click create index pattern.
Configure the visualization
Now we want to create visualizations and a dashboard to view iRODS activity. We have already created a dashboard for you to load.
Save the irods_training/advanced/example_kibana_dashboard.json to your local computer (not the VM).
You can copy/paste this from your VM into a local file or clone the irods_training repository on your local machine.
Click on Management (left pane) -> Saved Objects and click Import.
Select the file you saved in #1. Confirm the changes.
Click on the Dashboard (left pane) -> and then iRODS Dashboard.
Go ahead and click on the Auto Refresh button at the top and change the refresh period to 30 seconds.
Click on the clock icon (Last 15 minutes) at the top of the screen and select "today".
Configure the visualization
You should see a dashboard that looks similar to the screenshot below.
If you have not executed a put or get to iRODS the Bytes Read Per Minute and Bytes Written Per Minute will report no data.
What to Consider in an iRODS Deployment
Things to consider
Number of users and expected simultaneous connections
Expected ingest rate
Sizes of files
many small files (more overhead per connection)
Partial read / write vs whole file usage
Replication for durability
Replication for locality of reference
Load balancing vs High Availability
iRODS will run on a laptop or a rack of servers
Questions?
Anatomy of an iRODS installation
/etc/irods/core.* - iRODS Rule Language
/etc/irods/host_access_control_config.json - hostname filtering
/etc/irods/hosts_config.json - local /etc/hosts style configuration
/etc/irods/server_config.json - primary server configuration
/etc/irods/service_account.config - service account information
/usr/bin/* - iCommands
/usr/sbin/irodsAgent
/usr/sbin/irodsPamAuthCheck
/usr/sbin/irodsReServer
/usr/sbin/irodsServer
/var/lib/irods - service account home directory
/usr/lib/irods/plugins - plugins location
Introduction to iCommands
iRODS command line equivalent to standard Unix operations
ils
icd
ipwd
iput
iget
irepl
use -h to get help with any particular iCommand
ihelp will show all available iCommands
Questions?