Alan King

Software Developer

iRODS Consortium

December 1, 2021

TRiRODS

Chapel Hill, NC

iRODS Testing Environment

Overview

Motivation
Features
Demo
Future Work

Motivation: Testing iRODS is Hard

Current testing in CI is consistent and fast - it works.

However, documentation of test runs, usability, and extensibility are difficult.

Running the tests requires a lot of setup and careful alignment of volume mounts and image names.

Motivation: iRODS Testing Environment

Consistent environment for testing iRODS server in a number of configurations which can easily be run at the bench

Based on Docker/Compose python API
Uses minimal base images for maximal flexibility
Composable services for different OS/database combinations

Provides a small, specialized python library and suite of scripts for installing, configuring, and running iRODS in containers

https://github.com/irods/irods_testing_environment

Motivation: iRODS Development Environment

Set of containers which build and debug iRODS server using directories on the host machine.

Useful for developing a workflow for building and testing iRODS in conjunction with the iRODS testing environment.

https://github.com/irods/irods_development_environment

Motivation: Workflow Development

source code

↓

packages

↓

irods_testing_environment

irods_development_environment

Features: Stand It Up

Inspired by UGM 2021 Lightning talk by Kory Draughn,

Just Stand It Up: https://youtu.be/Be1R-WPNrc4

  --verbose, -v         Increase the level of output to stdout. CRITICAL and
                        ERROR messages will always be printed. Add more to see
                        more log messages (e.g. -vvv displays DEBUG).
  --project-directory PATH_TO_PROJECT_DIRECTORY
                        Path to Compose project on which packages will be
                        installed.
  --project-name PROJECT_NAME
                        Name of Compose project on which to install packages.
  --odbc-driver-path PATH_TO_ODBC_DRIVER_ARCHIVE
                        Path to the ODBC driver archive file on the local
                        machine. If not provided, the driver will be
                        downloaded.
  --irods-package-directory PATH_TO_DIRECTORY_WITH_PACKAGES
                        Path to local directory which contains iRODS packages.
  --irods-package-version PACKAGE_VERSION_TO_DOWNLOAD
                        Version of official iRODS packages to download and
                        install. If neither this or --package-directory are
                        specified, the latest available version will be
                        installed.
  --irods-externals-package-directory PATH_TO_DIRECTORY_WITH_IRODS_EXTERNALS_PACKAGES
                        Path to local directory which contains iRODS externals
                        packages.
  --use-ssl             Indicates that SSL should be configured and enabled in
                        the Zone.

Features: Federate

Stand up and federate multiple iRODS zones, including pre-existing zones.

  --consumers-per-zone IRODS_CATALOG_CONSUMER_INSTANCES_PER_ZONE
                        Number of iRODS Catalog Service Consumer service
                        instances per Zone.
  --federate-consumers  If indicated, the iRODS Catalog Service Consumers for
                        each Zone will be federated with each of the other
                        Zones in addition to the iRODS Catalog Service
                        Providers (which are required to be federated).
  --zone-names IRODS_ZONE_NAME [IRODS_ZONE_NAME ...]
                        Space-delimited list of zone names to set up.
  --skip-setup          If indicated, the Zones will not be set up, only
                        federated.

Features: Run Tests

A top-to-bottom solution for standing up an iRODS zone, running python test suite, and collecting the results in a log. Topology and federation also supported.

  --tests TESTS [TESTS ...]
                        Space-delimited list of tests to be run. If not
                        provided, ALL tests will be run (--run_python-suite).
  --output-directory FULLPATH_TO_DIRECTORY_FOR_OUTPUT, -o FULLPATH_TO_DIRECTORY_FOR_OUTPUT
                        Full path to local directory for output from
                        execution. Individual job runs will appear as
                        subdirectories in this directory. Defaults to
                        temporary directory.
  --job-name JOB_NAME, -j JOB_NAME
                        Name of the directory where output from a specific job
                        will appear within the output directory. Defaults to a
                        UUID.
  --fail-fast           If indicated, exits on the first test that returns a
                        non-zero exit code.

Features: Libraries

Scripts are built on the library but each set of functionality has been captured in a script

install
setup
configuration
execution

One can execute each step of the process oneself.

DEMO

Adding a new platform

Add a Compose project for the OS for each database plugin
Add local package file extension, and package installation and upgrade commands to install module
Add ODBC setup methods for each database plugin

Engineering efforts are underway to make this easier for the community to add support for additional platforms.

Future Work and Dreams

Plugins and clients
Parallel execution
Horizontal scaling (DOCKER_HOST)
Automation/CI
Environment reproduction (zone report)
Pre-baked image options
Client-server compatibility matrix
Container technology agnosticism

December 2021 TriRODS - iRODS Testing Environment

December 2021 TriRODS - iRODS Testing Environment

Alan King

December 2021 TriRODS - iRODS Testing Environment

More from Alan King