iRODS 7th Generation
Build and Test Infrastructure
May 20, 2020
Renaissance Computing Institute
UNC-Chapel Hill
Jaspreet Gill
Automation Engineer, iRODS Consortium
iRODS 7th Generation
Build and Test Infrastructure
iRODS Build and Test - History
July 2011
-
Python → Node.js → RabbitMQ → Celery → Eucalyptus
October 2012
- Python → Node.js → ssh → OpenStack
January 2013
- Hudson → Python → OpenStack
October 2013
- Hudson → Python → vSphere long-running VMs
Spring 2015
- Jenkins → Python → Ansible → zone_bundles → vSphere dynamic VMs
Spring 2017
- Moved iRODS build/test logic from Ansible to python modules (per-repository)
- Consolidated to two parameterized Jenkins jobs
Fall 2019
- Jenkins → Python → Docker
iRODS Build and Test - 2019 Architecture
- Dockerized Jenkins
- All configuration and setup in git
- Launches sibling Docker containers
- Build OS Images
- Build iRODS Packages
- Deploy and Test
- core, plugins, topology, federation
- Development is same as production
iRODS Build and Test - Progress
iRODS Build and Test - 4.2.8 release cycle
- iCAT database runs in its own container for every test
-
Serialized Workflow
- Docker by default creates max 31 networks
- GitHub rate limit exceeded exception
- We are still learning about Docker
- Operating Systems supported → Ubuntu 16, Ubuntu 18, and CentOS 7
- Databases supported → PostgreSQL, MySQL/MariaDB, and Oracle
- Number of Core Test Suites per OS per Database → 65
- Number of Plugins Tested per OS per Database → 12
- Topology → 4 combinations (Provider/Consumer and SSL)
- Upgrade Topology Test → 2 combinations (Provider/Consumer)
- Federation → 1 combination
iRODS Build and Test - Serialized Workflow
- Core Tests run in parallel
- Other tests are serialized
Operating Systems = 1
Databases = 1
Test Parallelism = 4
Containers Launched | ||
---|---|---|
iRODS Core Tests | 4 | 8 |
Plugins | 1 | 2 |
Federation | 1 (2 providers) | 4 |
Topology | 1 (1 provider + 3 consumers) | 5 |
iRODS Build and Test - Future
- Make iRODS Jenkins publicly accessible
- Investigate scaling out (Kubernetes?)
- Increase coverage
- Conformance testing
- Approachable for community developers
- Confidence
- Acceptance Criteria
TRiRODS May 2020 - iRODS 7th Generation Build and Test Infrastructure
By iRODS Consortium
TRiRODS May 2020 - iRODS 7th Generation Build and Test Infrastructure
Overview of the iRODS Build and Test Infrastructure
- 911