@vipulgupta2048

Testing 100's of OS  Images with Jenkins

A Journey from Pull Request to Release

I am Vipul!

@vipulgupta2048

We-pull

  • Product Owner & Documentation Lead at balena
  • Run docs initiative called Mixster
  • Contributions: Sugar Labs, ScrapingHub 
  • Volunteer: PyCon India & ALiAS
  • Pronouns: He/him/his

MY team behind this!

@vipulgupta2048

@vipulgupta2048

Meet the test subject, balenaOS

  • Open-source Yocto-based embedded OS
  • Meant to run containers on IoT devices
  • Fault-tolerant, field-tested, completely free
  • Supports 100+ Single Board Computers

Haven't heard of it?

@vipulgupta2048

balenaOS in action

BalenaOS runs containers on CO2 sensors, farms, smart dustbins, trucks...

First to run Docker containers in space.

balenaOS making waves

... underwater drones, museums, photo booths, parkings, forklifts, and the rare wireless speakers

@vipulgupta2048

@vipulgupta2048

The challenges

With smart cities & even your toaster getting firmware updates. The stakes have never been higher

While, testing software is hard.

@vipulgupta2048

Testing hardware is painful.

@vipulgupta2048

@vipulgupta2048

Challenges building
embedded operating systems

  • Absolute fault tolerance 
  • Deploying releases confidently
  • Maintaining backward compatibility
  • Adding support for new devices 
  • Patching issues in existing devices

Our solution, Autokit

Autokit

(controller)

Raspberry Pi 4
(Device Under Test)

With Autokit, we enabled a Hardware in loop testing pipeline for balenaOS

Autokit Features

  • Flashing storage mediums (like, SD cards, eMMC etc)
  • Controlling the power
  • Detecting if the DUT is on or off
  • Boot mode selection & customized boot process
  • Simulate USB connections

Even more Features

  • Support display capture & serial communication
  • Works & can test with modems, HDDs 
  • Provide wifi or ethernet connection to the DUT
  • Ability to interact with other interfaces on the DUT

Leaving the best for last!

  • STL files available for 3D printing cases
  • No custom hardware, all of-the-shelf components
  • Docs to build your own autokit
  • Free and open-source!

The software layer, Leviathan

  • Finds an available autokit
  • Configures balenaOS images
  • Run the tests by interfacing with the autokit
  • The same test suite supports over 50+ device types 
  • Tests written in node-tap, results in JSON
  • Free and open-source.

balenaOS Hardware in the loop testing pipeline

Let's
Start Our
Journey

1. The Pull Request

A contributor opens a pull request on the balenaOS meta-balena repository.

2. Build them all!

This triggers a Jenkins job that builds balenaOS releases for every device we support. 

3. Freshly baked balenaOS

Artifacts built by upstream jobs are copied to downstream testing jobs managed by our testing framework, Leviathan.

4. Meet Leviathan

Leviathan is the software layer that interacts with balenaOS images, does configuration and talks with the hardware, autokit.

Turning on the autopilot.

Leviathan abstracts away complexities by enabling developers to write one test suite to target all devices. No matter their power requirements, boot process, or flash procedure. Leviathan takes care of everything.

// Leviathan test to flash a device connected to the autokit (Sample)

await this.worker.off(); // Ensure test device is off before flashing

await this.worker.flash(this.os.image.path); // Flash the device with balenaOS image

await this.worker.on(); // Turn on the device

test.true(
  true,
  `${this.os.image.path} should be flashed properly`,
);

5. Hardware in loop testing (HiLT)

Autokit automatically flashes the device under test with the OS, provides power, network & executes our commands on the test.

[
  {
    "suite": "Managed BalenaOS release suite",
    "stats": {
      "tests": 12,
      "ran": 10,
      "skipped": 2,
      "passed": 10,
      "failed": 0
    },
    "tests": {
      "Image preload test": "passed",
      "Move device to hostapd test App": "skipped",
      "Move device back to original app": "skipped",
      "Provisioning without deltas": "passed",
      "Override lock test": "passed",
      "Update supervisor randomized timer": "passed",
      "Set device environment variables": "passed",
      "Set service environment variables": "passed",
      "SSH authentication in production mode": "passed",
      "SSH authentication in development mode": "passed",
      "os-config service on boot": "passed",
      "os-config service randomized timer": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:43:52 GMT+0000 (Coordinated Universal Time)"
  },
  {
    "suite": "Hostapp update suite",
    "stats": {
      "tests": 5,
      "ran": 5,
      "skipped": 0,
      "passed": 5,
      "failed": 0
    },
    "tests": {
      "Broken balena-engine": "passed",
      "Broken VPN": "passed",
      "Rollback altboot (broken init) test": "passed",
      "HUP from previous release": "passed",
      "HUP from this release": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:58:25 GMT+0000 (Coordinated Universal Time)"
  },
  {
    "suite": "Unmanaged BalenaOS release suite",
    "stats": {
      "tests": 45,
      "ran": 36,
      "skipped": 9,
      "passed": 36,
      "failed": 0
    },
    "tests": {
      "check secure boot": "passed",
      "BeagleBone Black u-boot overlay test: deactivate HDMI": "skipped",
      "243390-rpi3 - CUS/EUS chipsets test": "skipped",
      "fingerprint file test": "passed",
      "ext4 filesystems are checked on boot": "passed",
      "OS-release file check": "passed",
      "Installer used migrator module": "passed",
      "issue file check": "passed",
      "issue.net file check": "passed",
      "Chronyd service": "passed",
      "Sync test": "passed",
      "Source test": "passed",
      "Offline sources test": "passed",
      "System time skew test": "passed",
      "kernel-overlap test": "passed",
      "Bluetooth scanning test": "skipped",
      "Container healthcheck test": "passed",
      "Container exposed variables test": "passed",
      "Identification test": "skipped",
      "Cellular tests": "passed",
      "hostname configuration test": "passed",
      "ntpServer test": "passed",
      "dnsServers test": "passed",
      "os.network.connectivity test": "passed",
      "os.network.wifi.randomMacAddressScan test": "passed",
      "udevRules test": "passed",
      "persistentLogging configuration test": "passed",
      "Reboot test": "skipped",
      "Wired test": "skipped",
      "Wireless test": "skipped",
      "Socks5 test": "passed",
      "Http-connect test": "passed",
      "Engine socket is exposed in development images": "passed",
      "Engine socket is not exposed in production images": "passed",
      "Engine watchdog recovery": "passed",
      "Engine healthcheck performance": "passed",
      "Under-voltage test": "passed",
      "Ramdisks, zram and loop devices are not scanned for rootfs": "passed",
      "by-state links are created": "passed",
      "DToverlay & DTparam tests": "skipped",
      "state partition reset": "passed",
      "data partition reset": "passed",
      "RevPi Core 3 DIO module test": "skipped",
      "zram is enabled and configured as swap": "passed",
      "Internet sharing iptables rules test": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:24:52 GMT+0000 (Coordinated Universal Time)"
  }
]

62+ tests ran

[
  {
    "suite": "Managed BalenaOS release suite",
    "stats": {
      "tests": 12,
      "ran": 10,
      "skipped": 2,
      "passed": 10,
      "failed": 0
    },
    "tests": {
      "Image preload test": "passed",
      "Move device to hostapd test App": "skipped",
      "Move device back to original app": "skipped",
      "Provisioning without deltas": "passed",
      "Override lock test": "passed",
      "Update supervisor randomized timer": "passed",
      "Set device environment variables": "passed",
      "Set service environment variables": "passed",
      "SSH authentication in production mode": "passed",
      "SSH authentication in development mode": "passed",
      "os-config service on boot": "passed",
      "os-config service randomized timer": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:43:52 GMT+0000 (Coordinated Universal Time)"
  },
  {
    "suite": "Hostapp update suite",
    "stats": {
      "tests": 5,
      "ran": 5,
      "skipped": 0,
      "passed": 5,
      "failed": 0
    },
    "tests": {
      "Broken balena-engine": "passed",
      "Broken VPN": "passed",
      "Rollback altboot (broken init) test": "passed",
      "HUP from previous release": "passed",
      "HUP from this release": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:58:25 GMT+0000 (Coordinated Universal Time)"
  },
  {
    "suite": "Unmanaged BalenaOS release suite",
    "stats": {
      "tests": 45,
      "ran": 36,
      "skipped": 9,
      "passed": 36,
      "failed": 0
    },
    "tests": {
      "check secure boot": "passed",
      "BeagleBone Black u-boot overlay test: deactivate HDMI": "skipped",
      "243390-rpi3 - CUS/EUS chipsets test": "skipped",
      "fingerprint file test": "passed",
      "ext4 filesystems are checked on boot": "passed",
      "OS-release file check": "passed",
      "Installer used migrator module": "passed",
      "issue file check": "passed",
      "issue.net file check": "passed",
      "Chronyd service": "passed",
      "Sync test": "passed",
      "Source test": "passed",
      "Offline sources test": "passed",
      "System time skew test": "passed",
      "kernel-overlap test": "passed",
      "Bluetooth scanning test": "skipped",
      "Container healthcheck test": "passed",
      "Container exposed variables test": "passed",
      "Identification test": "skipped",
      "Cellular tests": "passed",
      "hostname configuration test": "passed",
      "ntpServer test": "passed",
      "dnsServers test": "passed",
      "os.network.connectivity test": "passed",
      "os.network.wifi.randomMacAddressScan test": "passed",
      "udevRules test": "passed",
      "persistentLogging configuration test": "passed",
      "Reboot test": "skipped",
      "Wired test": "skipped",
      "Wireless test": "skipped",
      "Socks5 test": "passed",
      "Http-connect test": "passed",
      "Engine socket is exposed in development images": "passed",
      "Engine socket is not exposed in production images": "passed",
      "Engine watchdog recovery": "passed",
      "Engine healthcheck performance": "passed",
      "Under-voltage test": "passed",
      "Ramdisks, zram and loop devices are not scanned for rootfs": "passed",
      "by-state links are created": "passed",
      "DToverlay & DTparam tests": "skipped",
      "state partition reset": "passed",
      "data partition reset": "passed",
      "RevPi Core 3 DIO module test": "skipped",
      "zram is enabled and configured as swap": "passed",
      "Internet sharing iptables rules test": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:24:52 GMT+0000 (Coordinated Universal Time)"
  }
]

62 tests, over 200+ assertions

Avg. runtime: 90-120 minutes

Breaking things intentionally to see if the OS can recover & update

6. Jenkins reports back results

With each downstream test job completed, Jenkins updates statuses of these jobs back on GitHub for the PR to be reviewed.

7. PR is merged 

With each downstream test job completed, Jenkins updates statuses of these jobs back on GitHub for the PR to be reviewed.

8. Merge === Deploy

Device repositories use this new meta-balena layer now available to deploy device-specific releases. These are built again, tested, and then merged to be deployed automatically. All using Jenkins.

The Journey

  • Propose changes
  • Build draft balenaOS releases from PR changes.
  • Run draft release on an actual device.
  • Test the release end-to-end by automating the device.
  • Gain feedback from the loop to make improvements
  • Make changes to the pull request.
  • Tests provide instant feedback on changes & reviews
  • PR gets merged & the release gets deployed. 

This is hardware in loop testing with Jenkins

Gettin' Exponential Gains

With Hardware in Loop Testing & Jenkins for CI/CD, we have managed to:

  • Reduce our OS release from weeks to hours 
  • Achieve exponential scale to support new devices
  • Save thousands of hours in troubleshooting
  • Follow TDD & new tests being added every day

Most importantly, we even added QEMU support for testing virtual devices. Thanks to the Chip Shortage 2022

But Vipul, how can I use this now at work?

Autokit

(controller)

Devices that need to be controlled, automated and maintained using scripts

Autokit

(controller)

Feedback on changes

Testing software directly on hardware
in a CI/CD pipeline

?

Autokit

(controller)

?

?

?

?

?

?

?

?

?

?

?

Quality Assurance, stress testing, random testing, environmental testing

And, you can test Operating Systems

That's the talk.

What we learned

  • Testing operating systems is important
  • And, incredibly painful to scale
  • But doesn't have to be.
  • When you got tools like Autokit & Leviathan.
  • To achieve exponential gains with HiTL pipelines

Resources

And, that's about it!

Questions? Collaborate? Work with us? Reach out!

Vipul Gupta

Reviews cheesecakes, solves GitHub issues & runs memeservice.

Feedback please + Link to the slides

[cdCon + GitOpsCon 2023] Testing balenaOS images with Jenkins

By Vipul Gupta

Made with Slides.com

[cdCon + GitOpsCon 2023] Testing balenaOS images with Jenkins

This presentation is about using Jenkins to test hundreds of OS images, specifically focusing on balenaOS and the challenges that come with testing embedded operating systems. It was used for a talk on balenaOS's testing pipeline using Jenkins that took place at cdCon + GitOpsCon 2023 in Vancouver, Canada.

  • 427

More from Vipul Gupta

  • [Constellation 2024] How Balena Releases 100's of OS releases with GitHub Actions in Hours, Not Weeks
    Vipul Gupta
    264
  • [FOSS India 2024] Pains, Processes & the Eventual Gains of Building Open-Source Hardtech projects
    Vipul Gupta
    221
  • [Open Source Summit Japan 2023] Deep dive into Hardware in the loop testing
    Vipul Gupta
    354
  • [OSSEU 23] Building a Virtual Raspberry Pi using QEMU
    Vipul Gupta
    371