Alan King

Senior Software Developer

iRODS Consortium

July 6, 2022

iRODS User Group Meeting 2022

KU Leuven, Belgium

iRODS Development and Testing Environment

Overview

Motivation
History
Limitations of the past
Development Environment
Testing Environment
Demo
Future Work

Motivation

Tests should be meaningful and consistent.

Running tests should be reliable and accessible.

Passing tests build confidence in the changes developers make to software, asserting the correctness of the changes and of the entire system.

History of Build-and-Test Systems at iRODS

v1 - July 2011: Python → Node.js → RabbitMQ → Celery → Eucalyptus
v2 - October 2012: Python → Node.js → ssh → OpenStack
v3 - January 2013: Hudson → Python → OpenStack
v4 - October 2013: Hudson → Python → vSphere long-running VMs
v5 - Spring 2015: Jenkins → Python → Ansible → zone_bundles → vSphere dynamic VMs
v6 - Spring 2017: Jenkins → Python → vSphere dynamic VMs → build/test hooks
v7 - Summer 2019: Docker → Jenkins → Python → Docker → build/test hooks

Limitations of the past

Vendor lock-in (Jenkins)
Difficult to manage test results and built package output
Docker image explosion
Opaque business logic

v8 - 2021/2022

iRODS Development Environment

https://github.com/irods/irods_development_environment

Set of Dockerized applications used to build, run, and debug packages for the iRODS server, iCommands, and plugins.

iRODS Testing Environment

https://github.com/irods/irods_testing_environment

Scripts built on python library which uses Docker Compose to stand up iRODS zones in various configurations and run tests.

Basic Build-and-Test Workflow

source code

↓

packages

↓

test results

irods_testing_environment

irods_development_environment

iRODS Development Environment

Features:

Package builders for iRODS, plugins, and externals
Debuggers

Advantages:

No cloning or copying repositories
Build cache allows for faster iteration
Consistent process for development and release

Basic Build-and-Test Workflow

source code

↓

packages

↓

test results

irods_testing_environment

irods_development_environment

iRODS Testing Environment

Features:

Parallel/remote test execution for iRODS and plugins
Multi-server and federated iRODS zones
Extensible python library for custom scripts
Stand up iRODS zone for any (recent-ish) released version

Advantages:

Precision controls for running tests in parallel
Convenient way to reproduce issues
Consistent process for bench and automated testing

iRODS Testing Environment

$ python run_core_tests.py --project-directory projects/ubuntu-20.04/ubuntu-20.04-postgres-10.12 \
                           --irods-package-directory ~/hdd/builds/irods_packages/4-3-stable/ubuntu-20.04 \
                           --concurrent-test-executor-count 4 -v
<snip>
2022-06-23 15:38:30,381 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_irm', '--xml_output']]
2022-06-23 15:39:12,397 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_4]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_ireg', '--xml_output']]
2022-06-23 15:39:17,437 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_4]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_irmtrash', '--xml_output']]
2022-06-23 15:40:43,075 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_4]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_iscan', '--xml_output']]
2022-06-23 15:41:27,853 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_4]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_itouch', '--xml_output']]
2022-06-23 15:42:17,991 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_4]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_iuserinfo', '--xml_output']]
2022-06-23 15:42:21,349 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_4]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_negotiation', '--xml_output']]
2022-06-23 15:42:49,652 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_4]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_quotas', '--xml_output']]
2022-06-23 15:44:15,902 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_3]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_resource_types.Test_Resource_CompoundWithMockarchive', '--xml_output']]
2022-06-23 15:45:35,288 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_all_rules', '--xml_output']]
2022-06-23 15:49:06,316 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_irsync', '--xml_output']]
2022-06-23 15:49:19,821 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_isysmeta', '--xml_output']]
2022-06-23 15:49:39,938 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_itrim', '--xml_output']]
2022-06-23 15:49:46,930 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_control_plane', '--xml_output']]
2022-06-23 15:49:52,223 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_load_balanced_suite', '--xml_output']]
2022-06-23 15:49:55,635 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_prep_genquery_iterator', '--xml_output']]
<snip>
2022-06-23 18:19:10,572 ERROR - ==== begin test run results ====
-----
results for [ubuntu-2004-postgres-1012_irods-catalog-provider_1]
        passed tests:
                [[1697.3373]s]  [test_all_rules]
                [[ 251.6414]s]  [test_control_plane]
                [[  89.0497]s]  [test_ibun]
                [[3715.5025]s]  [test_icommands_file_operations]
                [[  14.8783]s]  [test_igroupadmin]
                [[  18.9958]s]  [test_ilsresc]
                [[  26.3037]s]  [test_imeta_interactive]
                [[  30.7978]s]  [test_ipasswd]
                [[   5.0171]s]  [test_ipwd]
                [[1463.7713]s]  [test_irepl]
                [[   9.3546]s]  [test_irodsctl]
                [[  58.6852]s]  [test_istream]
                [[  35.5464]s]  [test_itree]
                [[   7.3381]s]  [test_izonereport]
                [[ 261.9476]s]  [test_native_rule_engine_plugin]
                [[  14.3952]s]  [test_resource_configuration]
                [[ 837.4747]s]  [test_resource_types.Test_Resource_Deferred]
                [[ 960.3886]s]  [test_resource_types.Test_Resource_Random]
                [[1379.2288]s]  [test_resource_types.Test_Resource_ReplicationToTwoCompound]
                [[  27.3775]s]  [test_resource_types.Test_Resource_WeightedPassthru]
                [[   7.5812]s]  [test_special_collections]
        skipped tests:
        failed tests:
        return code:[0]
        time elapsed: [1.091e+04]seconds ([   3]hours [    1.877]minutes)
-----
-----
results for [ubuntu-2004-postgres-1012_irods-catalog-provider_2]
        passed tests:
                [[   3.2683]s]  [test_auth]
                [[ 385.5667]s]  [test_delay_queue]
                [[  13.5900]s]  [test_icd]
                [[ 178.9789]s]  [test_icp]
                [[   4.5107]s]  [test_ihelp]
                [[  27.5451]s]  [test_imeta_admin_mode]
                [[ 273.2416]s]  [test_imeta_set]
                [[ 362.5961]s]  [test_iphymv]
                [[   5.4457]s]  [test_iqmod]
                [[  17.6846]s]  [test_irm]
                [[ 635.9353]s]  [test_irsync]
                [[  13.5050]s]  [test_isysmeta]
                [[  20.1163]s]  [test_itrim]
                [[  12.2845]s]  [test_load_balanced_suite]
                [[   3.4121]s]  [test_prep_genquery_iterator]
                [[1037.5428]s]  [test_resource_types.Test_Resource_Compound]
                [[ 899.7080]s]  [test_resource_types.Test_Resource_MultiLayered]
                [[ 965.3594]s]  [test_resource_types.Test_Resource_RandomWithinRandom]
                [[1485.8112]s]  [test_resource_types.Test_Resource_ReplicationToTwoCompoundResourcesWithPreferArchive]
                [[  66.3305]s]  [test_rule_engine_plugin_framework]
                [[  23.2960]s]  [test_ssl]
        skipped tests:
        failed tests:
        return code:[0]
        time elapsed: [6.436e+03]seconds ([   1]hours [    47.26]minutes)
-----
-----
results for [ubuntu-2004-postgres-1012_irods-catalog-provider_3]
        passed tests:
                [[  22.8854]s]  [test_catalog]
                [[  18.0757]s]  [test_dynamic_peps]
                [[ 161.7720]s]  [test_ichksum]
				[[   3.4590]s]  [test_iexit]
                [[  13.3650]s]  [test_ilocate]
                [[  18.2060]s]  [test_imeta_error_handling]
                [[  18.8343]s]  [test_imkdir]
                [[  46.6802]s]  [test_iput]
                [[ 121.4816]s]  [test_iquest]
                [[  53.1015]s]  [test_irmdir]
                [[  14.9655]s]  [test_irule]
                [[ 136.8253]s]  [test_iticket]
                [[  30.0276]s]  [test_iunreg]
                [[  77.1380]s]  [test_misc]
                [[   3.0893]s]  [test_python_rule_engine_plugin]
                [[ 878.0375]s]  [test_resource_types.Test_Resource_CompoundWithMockarchive]
                [[ 723.4627]s]  [test_resource_types.Test_Resource_NonBlocking]
                [[1060.6564]s]  [test_resource_types.Test_Resource_RandomWithinReplication]
                [[1159.1791]s]  [test_resource_types.Test_Resource_ReplicationWithinReplication]
                [[   4.4386]s]  [test_rule_engine_plugin_passthrough]
                [[  14.9310]s]  [test_stacktrace]
        skipped tests:
        failed tests:
        return code:[0]
        time elapsed: [4.581e+03]seconds ([   1]hours [    16.34]minutes)
-----
-----
results for [ubuntu-2004-postgres-1012_irods-catalog-provider_4]
        passed tests:
                [[  30.1313]s]  [test_collection_mtime]
                [[ 860.5208]s]  [test_iadmin]
                [[  59.1282]s]  [test_ichmod]
                [[  13.1591]s]  [test_ifsck]
                [[  58.4915]s]  [test_ils]
                [[   9.2195]s]  [test_imeta_help]
                [[  34.9259]s]  [test_imv]
                [[  92.3246]s]  [test_iput_options]
                [[ 156.5395]s]  [test_ireg]
                [[   5.0398]s]  [test_irmtrash]
                [[  85.6373]s]  [test_iscan]
                [[  44.7783]s]  [test_itouch]
                [[  50.1372]s]  [test_iuserinfo]
                [[   3.3577]s]  [test_negotiation]
                [[  28.3033]s]  [test_quotas]
                [[1258.4760]s]  [test_resource_types.Test_Resource_CompoundWithUnivmss]
                [[ 777.6891]s]  [test_resource_types.Test_Resource_Passthru]
                [[2057.2540]s]  [test_resource_types.Test_Resource_Replication]
                [[ 874.8417]s]  [test_resource_types.Test_Resource_Unixfilesystem]
                [[ 924.4358]s]  [test_rulebase]
                [[  77.2080]s]  [test_symlink_operations]
        skipped tests:
        failed tests:
        return code:[0]
        time elapsed: [7.502e+03]seconds ([   2]hours [    5.027]minutes)
-----
All tests passed! :)
time elapsed: [10912.6214]seconds ([   3]hours [ 1.8770]minutes)
==== end of test run results ====

Future Work and Dreams

Web application
Automation/CI
Client build/test
Environment reproduction (zone report)
Execution environment agnosticism (Docker, podman, VMs)
Orchestration

The Build-and-Test Monitor

UGM 2022 - iRODS Development and Testing Environment

UGM 2022 - iRODS Development and Testing Environment

Alan King

UGM 2022 - iRODS Development and Testing Environment

More from Alan King