June 25-28, 2019

iRODS User Group Meeting 2019

Utrecht, Netherlands

Justin James

Applications Engineer

iRODS Consortium

Administration

Auditing

Administration

Auditing

Tracking What We've Done

Remember at the end of the "Getting Started" section:​​

 

• We installed an audit plugin that generated events for every dynamic policy enforcement point executed
   
• We started a docker container with the following software:
  • RabbitMQ - message broker to catch events
  • Logstash - to read the events and write to DB
  • Elasticsearch - database to store the events
  • Kibana - visualization tool

Tracking What We've Done

• Visit http://<ip> where ip is the public IP for your VM
• Click on Dashboard
• Select "Today" for the time period at the top of the screen

Tracking What We've Done

You have a visualization of what has happened in your iRODS zone for the day.

 

You can see the bytes written and received, connections, top users, etc.

 

These are just a sample of what can be visualized.

 

All of the data is in the Elastic database and can be queried for additional interesting patterns or characteristics.

Tracking Origin of a File

Now let's say we want to track the origin (provenance) of some files in our system.

 

We have PEPs stored in our Elastic database that provide an audit trail for us.

 

Before we get started, let's install jq so that we can parse the JSON output of an elasticsearch query.

sudo apt-get -y install jq

Tracking Who Wrote to the File

curl -XGET 'localhost:9200/irods_audit/_search?pretty' -H 'Content-Type: application/json' -d'
{
     "_source": [ "@timestamp", "user_user_name", "obj_path" ],
     "sort" : [
         {"@timestamp":{"order": "asc"}}
     ],
     "size" :10000,
     "query": {
         "bool":  {
             "must": [
                 { "match": { "rule_name": "audit_pep_api_data_obj_put_pre" } },
                 { "match_phrase": { "obj_path": "tempZone/home/rods/stickers.jpg" } }
             ]
        }
     }
 }' | jq ".hits.hits[] | ._source"

Tracking Who Wrote to the File

This query returns the following five records showing the user rods put stickers.jpg five times:

{
  "@timestamp": "2018-05-30T20:13:01.331Z",
  "obj_path": "/tempZone/home/rods/stickers.jpg",
  "user_user_name": "rods"
}
{
  "@timestamp": "2018-05-30T21:02:59.350Z",
  "obj_path": "/tempZone/home/rods/stickers.jpg",
  "user_user_name": "rods"
}
{
  "@timestamp": "2018-05-30T21:03:16.370Z",
  "obj_path": "/tempZone/home/rods/stickers.jpg",
  "user_user_name": "rods"
}
{
  "@timestamp": "2018-05-30T21:03:31.671Z",
  "obj_path": "/tempZone/home/rods/stickers.jpg",
  "user_user_name": "rods"
}
{
  "@timestamp": "2018-05-30T21:12:01.143Z",
  "obj_path": "/tempZone/home/rods/stickers.jpg",
  "user_user_name": "rods"
}

Tracking Read Access to the File

curl -XGET 'localhost:9200/irods_audit/_search?pretty' -H 'Content-Type: application/json' -d'
{
     "_source": [ "@timestamp", "user_user_name", "obj_path" ],
     "sort" : [
         {"@timestamp":{"order": "asc"}}
     ],
     "size" :10000,
     "query": {
         "bool":  {
             "must": [
                 { "match": { "rule_name": "audit_pep_api_data_obj_get_pre" } },
                 { "match_phrase": { "obj_path": "tempZone/home/rods/stickers.jpg" } }
             ]
        }
     }
 }' | jq ".hits.hits[] | ._source"

There are two reads from user rods:

{
  "@timestamp": "2018-05-30T21:05:54.588Z",
  "obj_path": "/tempZone/home/rods/stickers.jpg",
  "user_user_name": "rods"
}
{
  "@timestamp": "2018-05-30T21:07:08.202Z",
  "obj_path": "/tempZone/home/rods/stickers.jpg",
  "user_user_name": "rods"
}

Look for all the "pre" PEPs

Search for all the "pre" PEPs that have been executed today, but exclude any authentication PEPs

curl -XGET 'localhost:9200/irods_audit/_search?pretty' -H 'Content-Type: application/json' -d'
{
     "_source": [ "@timestamp", "rule_name" ],
     "sort" : [
         {"@timestamp":{"order": "asc"}}
     ],
     "size" :10000,
     "query": {
         "bool": {
             "must" : {
                  "regexp": {"rule_name": "audit_pep_api_.*_pre"}
              },
              "must_not" : {
                    "regexp": {"rule_name": "audit_pep_api_auth_.*_pre"}
               }
          }
      }
 }' | jq ".hits.hits[] | ._source"