iRODS 7th Generation

Build and Test Infrastructure

May 20, 2020

Renaissance Computing Institute

UNC-Chapel Hill

Jaspreet Gill

Automation Engineer, iRODS Consortium

iRODS 7th Generation

Build and Test Infrastructure

iRODS Build and Test - History

July 2011

• Python → Node.js → RabbitMQ → Celery → Eucalyptus

October 2012

• Python → Node.js → ssh → OpenStack

January 2013

• Hudson → Python → OpenStack

October 2013

• Hudson → Python → vSphere long-running VMs

Spring 2015

• Jenkins → Python → Ansible → zone_bundles → vSphere dynamic VMs

Spring 2017

• Moved iRODS build/test logic from Ansible to python modules (per-repository)
• Consolidated to two parameterized Jenkins jobs

Fall 2019

• Jenkins  → Python → Docker

iRODS Build and Test - 2019 Architecture

• Dockerized Jenkins
• All configuration and setup in git
• Launches sibling Docker containers
  • Build OS Images
  • Build iRODS Packages
  • Deploy and Test
    • core, plugins, topology, federation
• Development is same as production

iRODS Build and Test - Progress

iRODS Build and Test - 4.2.8 release cycle

• iCAT database runs in its own container for every test
• Serialized Workflow
  • Docker by default creates max 31 networks
  • GitHub rate limit exceeded exception
  • We are still learning about Docker
• Operating Systems supported → Ubuntu 16, Ubuntu 18, and CentOS 7
• Databases supported → PostgreSQL, MySQL/MariaDB, and Oracle
• Number of Core Test Suites per OS per Database → 65
• Number of Plugins Tested per OS per Database → 12
• Topology → 4 combinations (Provider/Consumer and SSL)
• Upgrade Topology Test → 2 combinations (Provider/Consumer)
• Federation → 1 combination

iRODS Build and Test - Serialized Workflow

• Core Tests run in parallel
• Other tests are serialized

Operating Systems = 1

Databases = 1

Test Parallelism = 4

		Containers Launched
iRODS Core Tests	4	8
Plugins	1	2
Federation	1 (2 providers)	4
Topology	1 (1 provider + 3 consumers)	5

iRODS Build and Test - Future

• Make iRODS Jenkins publicly accessible
• Investigate scaling out (Kubernetes?)
• Increase coverage
• Conformance testing
• Approachable for community developers
  • Confidence
  • Acceptance Criteria