Alan King

Senior Software Developer

iRODS Consortium

iRODS Development and Testing Environments (v8)

iRODS Development and Testing Environments (v8)

November 13-18, 2022

Supercomputing 2022

Dallas, TX

Overview

  • Motivation
  • History
  • Limitations of the past
  • Development Environment
  • Testing Environment
  • Future Work

 

Motivation

Passing tests build confidence in the changes developers make to software, asserting the correctness of the changes and of the entire system.

 

Therefore, building and testing iRODS should be easy and consistent for everybody.

History of Build-and-Test Systems at iRODS

  • v1 - July 2011: Python → Node.js → RabbitMQ → Celery → Eucalyptus
  • v2 - October 2012: Python → Node.js → ssh → OpenStack
  • v3 - January 2013: Hudson → Python → OpenStack
  • v4 - October 2013: Hudson → Python → vSphere long-running VMs
  • v5 - Spring 2015: Jenkins → Python → Ansible → zone_bundles → vSphere dynamic VMs
  • v6 - Spring 2017: Jenkins → Python → vSphere dynamic VMs → build/test hooks
  • v7 - Summer 2019: Docker → Jenkins → Python → Docker → build/test hooks

Limitations of the past

  • Tied to Jenkins
    • Everybody is required to be an administrator
    • Relatively inflexible
    • Difficult to maintain
  • Difficult to manage test results and built package output
  • Docker image explosion (one tag per test run(!))

 

Bottom line: Our needs as developers and project maintainers were not being met satisfactorily.

Basic Build-and-Test Workflow

source code

↓

 

↓

packages

↓

 

↓

test results

run tests

build packages

v8 - 2021/2022

source code

↓

 

↓

packages

↓

 

↓

test results

irods_testing_environment

irods_development_environment

iRODS Development Environment

https://github.com/irods/irods_development_environment

Containerized package builders for iRODS, plugins, and externals for all supported OS's, plus debugging tools.

  • Builds local code and produces local packages
  • Runs the standard build process(es) in containers

Advantages:

  • No repeated cloning or copying repositories required
    • Developer makes edits with preferred local tools
  • Build cache allows for faster iteration
  • Consistent process for development and release

iRODS Development Environment: Example Usage

$ docker run --rm \
           -v ${irods_sourcedir}:/irods_source:ro \
           -v ${irods_builddir}:/irods_build \
           -v ${icommands_sourcedir}:/icommands_source:ro \
           -v ${icommands_builddir}:/icommands_build \
           -v ${irods_packagedir}:/irods_packages \
           -v ${externals_packagedir}:/irods_externals_packages \
           irods-core-builder-m:${PLATFORM}-${VERSION}

Run a Docker container that builds packages

  • Volume mounts for source code (input)
  • Volume mount for build cache and built packages (output)
  • Use separate Docker image tags for each OS/version
  • Optionally use custom iRODS externals packages

iRODS Development Environment: Example Usage

$ docker run --rm \
           -v ${plugin_sourcedir}:/irods_plugin_source:ro \
           -v ${plugin_builddir}:/irods_plugin_build \
           -v ${plugin_packagedir}:/irods_plugin_packages \
           -v ${irods_packagedir}:/irods_packages:ro \
           -v ${externals_packagedir}:/externals:ro \
           irods-plugin-builder-m:${PLATFORM}-${VERSION}

Run a Docker container that runs plugin "build hook"

  • Volume mounts for source code (input)
  • Volume mount for build cache and built packages (output)
  • Use separate Docker image tags for each OS/version
  • Optionally use custom iRODS externals packages

v8 - 2021/2022

source code

↓

 

↓

packages

↓

 

↓

test results

irods_testing_environment

irods_development_environment

iRODS Testing Environment

https://github.com/irods/irods_testing_environment

Scripts built on python library which uses Docker Compose to stand up iRODS zones in various configurations and run tests.

  • Local scripts execute commands in long-running containers
  • Uses local or released packages and create local test results
  • Scripts can run tests or just stand up iRODS zone(s)

Advantages:

  • Precision controls for running tests in parallel
  • Convenient way to reproduce issues
  • Consistent process for bench and automated testing

iRODS Testing Environment: Capabilities

  • Stand up long-running, topological iRODS zone(s)
    • stand_it_up.py - stand up a zone
    • federate.py - stand up and federate multiple zones
  • Run iRODS tests in various configurations
    • run_core_tests.py - run iRODS server tests
    • run_unit_tests.py - run unit tests for iRODS libraries
    • run_topology_tests.py - run tests on multi-server zone
    • run_federation_tests.py - run tests in federated zones
    • run_plugin_tests.py - run tests for iRODS plugins

iRODS Testing Environment: Running Tests

$ python run_core_tests.py \
  --project-directory projects/ubuntu-20.04/ubuntu-20.04-postgres-10.12 \
  --irods-package-directory ~/hdd/builds/irods_packages/4-3-stable/ubuntu-20.04 \
  --concurrent-test-executor-count 4
  • Stand up 4 iRODS zones using locally built packages
  • Distribute the test suite between the zones and run each set in parallel
  • Copy test results and log files to host machine
  • Tear down the 4 zones and remove containers

iRODS Testing Environment: Running Tests

-----
results for [ubuntu-2004-postgres-1012_irods-catalog-provider_2]
        passed tests:
                [[   3.1020]s]  [test_auth]
                [[ 386.7651]s]  [test_delay_queue]
                [[  13.6752]s]  [test_icd]
                [[ 176.7343]s]  [test_icp]
                [[   4.6180]s]  [test_ihelp]
                [[  27.8215]s]  [test_imeta_admin_mode]
                [[ 269.8607]s]  [test_imeta_set]
                [[  20.6078]s]  [test_itrim]
                [[  12.2143]s]  [test_load_balanced_suite]
                [[   2.7831]s]  [test_prep_genquery_iterator]
                [[1109.4858]s]  [test_resource_types.Test_Resource_Compound]
                [[1018.1528]s]  [test_resource_types.Test_Resource_MultiLayered]
                [[1009.2253]s]  [test_resource_types.Test_Resource_RandomWithinRandom]
                [[1542.5102]s]  [test_resource_types.Test_Resource_ReplicationToTwoCompoundResourcesWithPreferArchive]
                [[  69.4207]s]  [test_rule_engine_plugin_framework]
                [[  35.5109]s]  [test_ssl]
        skipped tests:
        failed tests:
        return code:[0]
        time elapsed: [6.723e+03]seconds ([   1]hours [    52.04]minutes)
-----

iRODS Testing Environment: Running Tests

-----
results for [ubuntu-2004-postgres-1012_irods-catalog-provider_4]
        passed tests:
                [[  30.0808]s]  [test_collection_mtime]
                [[ 837.2263]s]  [test_iadmin]
                [[  59.8457]s]  [test_ichmod]
                [[  13.3225]s]  [test_ifsck]
                [[  58.9188]s]  [test_ils]
                [[   8.9275]s]  [test_imeta_help]
                [[  34.9601]s]  [test_imv]
<snip>
                [[  28.2190]s]  [test_quotas]
                [[1081.7692]s]  [test_resource_types.Test_Resource_CompoundWithUnivmss]
                [[ 808.2622]s]  [test_resource_types.Test_Resource_Passthru]
                [[2091.9287]s]  [test_resource_types.Test_Resource_Replication]
                [[ 857.6486]s]  [test_resource_types.Test_Resource_Unixfilesystem]
                [[ 917.5663]s]  [test_rulebase]
                [[  78.1721]s]  [test_symlink_operations]
        skipped tests:
        failed tests:
        return code:[0]
        time elapsed: [7.345e+03]seconds ([   2]hours [    2.424]minutes)
-----

iRODS Testing Environment: Running Tests

All tests passed! :)
time elapsed: [10955.3559]seconds ([   3]hours [ 2.5893]minutes)
==== end of test run results ====

2022-07-04 20:57:00,726 ERROR - collecting logs [/tmp/ubuntu-2004-postgres-10123x3tjb2r/ubuntu-2004-postgres-1012_be703715-7901-4a34-affa-10ebea651ff4]

iRODS Testing Environment: Standard options

  --verbose, -v         Increase the level of output to stdout. CRITICAL and
                        ERROR messages will always be printed. Add more to see
                        more log messages (e.g. -vvv displays DEBUG).
  --project-directory PATH_TO_PROJECT_DIRECTORY
                        Path to Compose project on which packages will be
                        installed.
  --project-name PROJECT_NAME
                        Name of Compose project on which to install packages.
  --odbc-driver-path PATH_TO_ODBC_DRIVER_ARCHIVE
                        Path to the ODBC driver archive file on the local
                        machine. If not provided, the driver will be
                        downloaded.
  --irods-package-directory PATH_TO_DIRECTORY_WITH_PACKAGES
                        Path to local directory which contains iRODS packages.
  --irods-package-version PACKAGE_VERSION_TO_DOWNLOAD
                        Version of official iRODS packages to download and
                        install. If neither this or --package-directory are
                        specified, the latest available version will be
                        installed.
  --irods-externals-package-directory PATH_TO_DIRECTORY_WITH_IRODS_EXTERNALS_PACKAGES
                        Path to local directory which contains iRODS externals
                        packages.

iRODS Testing Environment: stand_it_up.py

  --consumer-instance-count IRODS_CATALOG_SERVICE_CONSUMER_INSTANCE_COUNT
                        Number of iRODS Catalog Service Consumer service
                        instances.
  --use-ssl             Indicates that SSL should be configured and enabled in
                        the Zone.

iRODS Testing Environment: run_core_tests.py

  --tests TESTS [TESTS ...]
                        Space-delimited list of tests to be run. If not
                        provided, ALL tests will be run (--run_python-suite).
  --output-directory FULLPATH_TO_DIRECTORY_FOR_OUTPUT, -o FULLPATH_TO_DIRECTORY_FOR_OUTPUT
                        Full path to local directory for output from
                        execution. Individual job runs will appear as
                        subdirectories in this directory. Defaults to
                        temporary directory.
  --job-name JOB_NAME, -j JOB_NAME
                        Name of the directory where output from a specific job
                        will appear within the output directory. Defaults to a
                        UUID.
  --fail-fast           If indicated, exits on the first test that returns a
                        non-zero exit code.
  --concurrent-test-executor-count EXECUTOR_COUNT
                        Number of concurrent executors to run tests at the
                        same time.
  --discard-logs        Indicates that the logs should not be collected from
                        the containers.
  --leak-containers     If indicated, the containers will not be torn down.
  --skip-setup          If indicated, the iRODS servers will not be set up.

Basic Build-and-Test Workflow

source code

↓

 

↓

packages

↓

 

↓

test results

irods_testing_environment

irods_development_environment

Future Work and Dreams

  • Web application
    • Automation/CI
  • Client build/test
  • Environment reproduction (zone report)
  • Execution environment agnosticism (Docker, podman, VMs)
  • Orchestration

Bonus: irods_demo

https://github.com/irods/irods_demo

Docker Compose project which runs the following services:

  • iRODS catalog provider (plus database)
  • iRODS C++ REST Client (plus nginx reverse-proxy)
  • iRODS Zone Management Tool
  • Metalnx
  • NFSRODS

While less flexible than the testing environment, irods_demo has proven useful in several POCs (and, obviously, demonstrations).

We are seeking community participation!

 

Please give it a whirl and let us know how it goes.

Thanks for listening

SC22 - iRODS Development and Testing Environments (v8)

By iRODS Consortium

Source: Alan King

SC22 - iRODS Development and Testing Environments (v8)

  • 497

More from iRODS Consortium