December 2, 2020

TRiRODS

Virtual Presentation

Jason Coposky and Bo Zhou

iRODS Consortium

C++ REST API and Zone Management Tool (ZMT)

C++ REST API and Zone Management Tool (ZMT)

Our Membership

Consortium

Member

Consortium

Member

Consortium

Member

Consortium

Member

Motivation

Create a simple to use, easy to deploy, fast and light-weight REST API

  • Based on the C++ API
  • Acts as a proxied mid-tier application layer
  • Utilizes JSON Web Tokens for AAI

Provides a single executable per endpoint, suitable for containerized deployment

https://github.com/irods/irods_client_rest_cpp

Configuration

The REST API provides an executable for each individual API endpoint.

An nginx template is provided for reference.

  • /etc/irods/irods_client_rest_cpp.json.template
  • /etc/irods/irods-client-rest-cpp-reverse-proxy.conf.template

Copy the irods_client_rest_cpp.json.template to /etc/irods and edit accordingly

Copy irods_client_rest_cpp.json to /etc/nginx/sites-available and link to /etc/nginx/sites-enabled

Template configuration files are installed by default:

Configuration - irods_client_rest_cpp.json

{
    "irods_rest_cpp_access_server" : {
        "port" : 8080,
        "threads" : 4,
        "maximum_idle_timeout_in_seconds" : 10
    },
    "irods_rest_cpp_auth_server" : {
        "port" : 8081,
        "threads" : 4,
        "maximum_idle_timeout_in_seconds" : 10
    },
    "irods_rest_cpp_list_server" : {
        "port" : 8082,
        "threads" : 4,
        "maximum_idle_timeout_in_seconds" : 10
    },
    "irods_rest_cpp_query_server" : {
        "port" : 8083,
        "threads" : 4,
        "maximum_idle_timeout_in_seconds" : 10
    },
    "irods_rest_cpp_stream_get_server" : {
        "port" : 8084,
        "threads" : 4,
        "maximum_idle_timeout_in_seconds" : 10
    },
    "irods_rest_cpp_stream_put_server" : {
        "port" : 8085,
        "threads" : 4,
        "maximum_idle_timeout_in_seconds" : 10
    },
    "irods_rest_cpp_zone_report_server" : {
        "port" : 8086,
        "threads" : 4,
        "maximum_idle_timeout_in_seconds" : 10
    },
    "irods_rest_cpp_admin_server" : {
        "port" : 8087,
        "threads" : 4,
        "maximum_idle_timeout_in_seconds" : 10
    }

}

Configuration - irods-client-rest-cpp-reverse-proxy.conf

server {
    listen 80;

    add_header 'Access-Control-Allow-Origin' '*' always;
    add_header 'Access-Control-Allow-Headers' '*' always;
    add_header 'Access-Control-Allow-Methods' 'AUTHORIZATION,ACCEPT,GET,POST,OPTIONS,PUT,DELETE' always;

    location /irods-rest/1.0.0/access {
        if ($request_method = 'OPTIONS') {
            return 204;
        }
        proxy_pass http://localhost:8080;
    }

    location /irods-rest/1.0.0/auth {
        if ($request_method = 'OPTIONS') {
            return 204;
        }
        proxy_pass http://localhost:8081;
    }

    location /irods-rest/1.0.0/list {
        if ($request_method = 'OPTIONS') {
            return 204;
        }

        proxy_pass http://localhost:8082;
    }

    location /irods-rest/1.0.0/query {
        if ($request_method = 'OPTIONS') {
            return 204;
        }

        proxy_pass http://localhost:8083;
    }

    location /irods-rest/1.0.0/stream {
        if ($request_method = 'OPTIONS') {
            return 204;
        }

        if ($request_method = GET ) {
            proxy_pass http://localhost:8084;
        }

        if ($request_method = PUT ) {
            proxy_pass http://localhost:8085;
        }
    }

    location /irods-rest/1.0.0/zone_report {
        if ($request_method = 'OPTIONS') {
            return 204;
        }

        proxy_pass http://localhost:8086;
    }

    location /irods-rest/1.0.0/admin {
        if ($request_method = 'OPTIONS') {
            return 204;
        }

        proxy_pass http://localhost:8087;
    }

}

Authentication

This REST API relies on the use of JSON Web Tokens to pass:

  • Identity
  • Authentication information
  • Authorization information
  • Future role based information

The /auth endpoint must be invoked first, generating a JWT

Send the JWT via the Authorization header for subsequent endpoints

curl -X GET -H "Authorization: ${TOKEN}" ...

For example:

/access

This endpoint provides a service for the generation of an iRODS ticket to a given logical path, be that a collection or a data object.

Method : POST

Parameters:

  • path: The url encoded logical path to a collection or data object for which access is desired

Example Curl Command:

Returns:

An iRODS ticket token within the X-API-KEY header, and a URL for streaming the object.

{
  "headers": [
    "X-API-KEY: CS11B8C4KZX2BIl"
  ],
  "url": "/irods-rest/1.0.0/stream?path=%2FtempZone%2Fhome%2Frods%2Ffile0&offset=0&limit=33064"
}
curl -X POST -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/access?path=%2FtempZone%2Fhome%2Frods%2Ffile0"

/admin

The administration interface to the iRODS Catalog which allows the creation, removal and modification of users, groups, resources, and other entities within the zone.

Method : POST

Parameters:

  • action : dictates the action taken: add, modify, or remove
  • target : the subject of the action: user, zone, resource, childtoresc, childfromresc, token, group, rebalance, unusedAVUs, specificQuery
  • arg2 : generic argument, could be user name, resource name, depending on the value of action and target
  • arg3 : generic argument , see above
  • arg4 : generic argument , see above
  • arg5 : generic argument , see above
  • arg6 : generic argument , see above
  • arg7 : generic argument , see above

Example Curl Command:

Returns:

"Success" or an iRODS exception

curl -X POST -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/admin?action=add&target=resource&arg2=ufs0&arg3=unixfilesystem&arg4=/tmp/irods/ufs0&arg5=&arg6=tempZone"

/auth

This endpoint provides an authentication service for the iRODS zone, currently only native iRODS authentication is supported.

Method : POST

Parameters:

  • password : The user's secret, depending on the method of authentication
  • auth_type : The iRODS method of authentication - native, pam, krb, gsi, etc.

Example Curl Command:

Returns:

An encrypted JWT which contains everything necessary to interact with the other endpoints. This token is expected in the Authorization header for the other services.

export TOKEN=$(curl -X POST "http://localhost:80/irods-rest/1.0.0/auth?user_name=rods&password=rods&auth_type=native")

/list

This endpoint provides a recursive listing of a collection, or stat, metadata, and access control information for a given data object.

Method : GET

Parameters:

  • path : The url encoded logical path which is to be listed
  • stat : Boolean flag to indicate stat information is desired
  • permissions : Boolean flag to indicate access control information is desired
  • metadata : Boolean flag to indicate metadata is desired
  • offset : number of records to skip for pagination
  • limit : number of records desired per page

Example Curl Command:

Returns:

An encrypted JWT which contains everything necessary to interact with the other endpoints. This token is expected in the Authorization header for the other services.

curl -X GET -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=0&limit=100" | jq

/list (continued)

Returns:

A JSON structured response within the body containing the listing, or an iRODS exception

{
  "_embedded": [
    {
      "logical_path": "/tempZone/home/rods/subcoll",
      "type": "collection"
    },
    {
      "logical_path": "/tempZone/home/rods/subcoll/file0",
      "type": "data_object"
    },
    {
      "logical_path": "/tempZone/home/rods/subcoll/file1",
      "type": "data_object"
    },
    {
      "logical_path": "/tempZone/home/rods/subcoll/file2",
      "type": "data_object"
    },
    {
      "logical_path": "/tempZone/home/rods/file0",
      "type": "data_object"
    }
  ],
  "_links": {
    "first": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=0&limit=100",
    "last": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=UNSUPPORTED&limit=100",
    "next": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=100&limit=100",
    "prev": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=0&limit=100",
    "self": "/irods-rest/1.0.0/list?path=%2FtempZone%2Fhome%2Frods&stat=0&permissions=0&metadata=0&offset=0&limit=100"
  }
}

/stream

Stream data into and out of an iRODS data object

Methods : PUT and GET

Parameters:

  • path : The url encoded logical path to a data object
  • offset : The offset in bytes into the data object
  • limit : The maximum number of bytes to read

Example Curl Command:

Returns:

PUT : Nothing, or iRODS Exception

GET : The data requested in the body of the response

curl -X PUT -H "Authorization: ${TOKEN}" -d"This is some data" "http://localhost/irods-rest/1.0.0/stream?path=%2FtempZone%2Fhome%2Frods%2FfileX&offset=0&limit=1000"
curl -X GET -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/stream?path=%2FtempZone%2Fhome%2Frods%2FfileX&offset=0&limit=1000"

or

/zone_report

Requests a JSON formatted iRODS Zone report, containing all configuration information for every server in the grid.

Method : POST

Parameters:

  • none

Example Curl Command:

Returns:

JSON formatted Zone Report

curl -X POST -H "Authorization: ${TOKEN}" "http://localhost/irods-rest/1.0.0/zone_report" | jq
{
  "schema_version": "file:///var/lib/irods/configuration_schemas/v3/zone_bundle.json",
  "zones": [
    {
    <snip>
    }]
}

Zone Management Tool

A single page web administration tool based on the C++ REST API

  • Based on Vue.JS and the Material UI Framework
  • Exposes reporting and configuration of all iRODS servers
  • Will provide a single dashboard view into an IRODS Zone
  • Will provide configuration of both logical and physical aspects of iRODS
  • Will assume the administration duties of Metalnx

Keep user facing and administrator facing concerns properly separated

Zone Management Tool

Future Work

  • Policy manipulation and deployment
  • Metadata manipulation
  • Delayed execution administration
  • Ticket administration
  • Monitoring and metrics

Community feed back is critical for future direction

https://github.com/irods/irods_client_zone_management_tool

Zone Management Tool

Demo

Made with Slides.com