@vipulgupta2048

[Arch] Testing 100's of OS images with Jenkins

Exploring the architecture from Pull Request to Release

I am Vipul!

@vipulgupta2048

We-pull

  • Product Owner & Documentation Lead at balena
  • Run docs initiative called Mixster
  • Volunteer: PyCon India & ALiAS
  • Pronouns: He/him/his

The Team behind this!

@vipulgupta2048

@vipulgupta2048

Meet the test subject, balenaOS

  • Open-source Yocto-based embedded OS
  • Meant to run containers on IoT devices
  • Fault-tolerant, field-tested, completely free
  • Supports 100+ Single Board Computers (SBCs)

Haven't heard of it?

@vipulgupta2048

balenaOS in action

BalenaOS runs containers on CO2 sensors, farms, smart dustbins, trucks...

First to run Docker containers in space.

balenaOS making waves

... underwater drones, museums, photo booths, parkings, forklifts, and your talking wireless speakers

@vipulgupta2048

@vipulgupta2048

The challenges

With smart cities & even your toaster getting firmware updates. The stakes have never been higher

While, testing software is hard.

@vipulgupta2048

Testing hardware is painful.

@vipulgupta2048

@vipulgupta2048

Challenges building
embedded operating systems

  • Absolute fault tolerance 
  • Deploying releases confidently
  • Maintaining backward compatibility
  • Adding support for new devices 
  • Patching issues in existing devices

Our solution, Autokit

Worker

(controller)

Raspberry Pi 4
(Device Under Test)

With Autokit, we enabled a Hardware in loop testing pipeline for balenaOS

Autokit Features

  • Flashing storage mediums (like, SD cards, eMMC etc)
  • Controlling the power
  • Detecting if the DUT is on or off
  • Boot mode selection & customized boot process
  • Simulate USB connections
  • Support display capture & serial communication
  • Works & can test with modems, HDDs 
  • Provide wifi or ethernet connection to the DUT
  • Ability to interact with other interfaces on the DUT

Leaving the best for last!

  • STL files available for 3D printing cases
  • No custom hardware, all of-the-shelf components
  • Docs to build your own autokit
  • Free and open-source!

Show
Me how it
Works

1. The Pull Request

A contributor opens a pull request on the balenaOS meta-balena repository.

2. Build them all!

This triggers a Jenkins job that builds balenaOS releases for every device we support. 

3. Freshly baked balenaOS

Artifacts built by upstream jobs are copied to downstream testing jobs managed by our testing framework, Leviathan.

4. Leviathan takes over

Leviathan is the software layer that interacts with balenaOS images, does configuration and talks with the hardware, autokit.

The software glue, Leviathan

  • Finds an available autokit
  • Configures balenaOS images
  • Run the tests by interfacing with the autokit
  • The same test suite supports over 50+ device types 
  • Tests written in node-tap, results in JSON
  • Free and open-source.

Turning on the autopilot.

Leviathan abstracts away complexities by enabling developers to write one test suite to target all devices. No matter their power requirements, boot process, or flash procedure. Leviathan takes care of everything.

// Leviathan test to flash a device connected to the autokit (Sample)

await this.worker.off(); // Ensure test device is off before flashing

await this.worker.flash(this.os.image.path); // Flash the device with balenaOS image

await this.worker.on(); // Turn on the device

test.true(
  true,
  `${this.os.image.path} should be flashed properly`,
);

5. Hardware in loop testing (HiLT)

Workers automatically provision the device under test with the OS, provides power, network & executes our commands

[
  {
    "suite": "Managed BalenaOS release suite",
    "stats": {
      "tests": 12,
      "ran": 10,
      "skipped": 2,
      "passed": 10,
      "failed": 0
    },
    "tests": {
      "Image preload test": "passed",
      "Move device to hostapd test App": "skipped",
      "Move device back to original app": "skipped",
      "Provisioning without deltas": "passed",
      "Override lock test": "passed",
      "Update supervisor randomized timer": "passed",
      "Set device environment variables": "passed",
      "Set service environment variables": "passed",
      "SSH authentication in production mode": "passed",
      "SSH authentication in development mode": "passed",
      "os-config service on boot": "passed",
      "os-config service randomized timer": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:43:52 GMT+0000 (Coordinated Universal Time)"
  },
  {
    "suite": "Hostapp update suite",
    "stats": {
      "tests": 5,
      "ran": 5,
      "skipped": 0,
      "passed": 5,
      "failed": 0
    },
    "tests": {
      "Broken balena-engine": "passed",
      "Broken VPN": "passed",
      "Rollback altboot (broken init) test": "passed",
      "HUP from previous release": "passed",
      "HUP from this release": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:58:25 GMT+0000 (Coordinated Universal Time)"
  },
  {
    "suite": "Unmanaged BalenaOS release suite",
    "stats": {
      "tests": 45,
      "ran": 36,
      "skipped": 9,
      "passed": 36,
      "failed": 0
    },
    "tests": {
      "check secure boot": "passed",
      "BeagleBone Black u-boot overlay test: deactivate HDMI": "skipped",
      "243390-rpi3 - CUS/EUS chipsets test": "skipped",
      "fingerprint file test": "passed",
      "ext4 filesystems are checked on boot": "passed",
      "OS-release file check": "passed",
      "Installer used migrator module": "passed",
      "issue file check": "passed",
      "issue.net file check": "passed",
      "Chronyd service": "passed",
      "Sync test": "passed",
      "Source test": "passed",
      "Offline sources test": "passed",
      "System time skew test": "passed",
      "kernel-overlap test": "passed",
      "Bluetooth scanning test": "skipped",
      "Container healthcheck test": "passed",
      "Container exposed variables test": "passed",
      "Identification test": "skipped",
      "Cellular tests": "passed",
      "hostname configuration test": "passed",
      "ntpServer test": "passed",
      "dnsServers test": "passed",
      "os.network.connectivity test": "passed",
      "os.network.wifi.randomMacAddressScan test": "passed",
      "udevRules test": "passed",
      "persistentLogging configuration test": "passed",
      "Reboot test": "skipped",
      "Wired test": "skipped",
      "Wireless test": "skipped",
      "Socks5 test": "passed",
      "Http-connect test": "passed",
      "Engine socket is exposed in development images": "passed",
      "Engine socket is not exposed in production images": "passed",
      "Engine watchdog recovery": "passed",
      "Engine healthcheck performance": "passed",
      "Under-voltage test": "passed",
      "Ramdisks, zram and loop devices are not scanned for rootfs": "passed",
      "by-state links are created": "passed",
      "DToverlay & DTparam tests": "skipped",
      "state partition reset": "passed",
      "data partition reset": "passed",
      "RevPi Core 3 DIO module test": "skipped",
      "zram is enabled and configured as swap": "passed",
      "Internet sharing iptables rules test": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:24:52 GMT+0000 (Coordinated Universal Time)"
  }
]

200+ assertions, 62 tests

Avg. runtime: 90-120 minutes

62+ tests ran

[
  {
    "suite": "Managed BalenaOS release suite",
    "stats": {
      "tests": 12,
      "ran": 10,
      "skipped": 2,
      "passed": 10,
      "failed": 0
    },
    "tests": {
      "Image preload test": "passed",
      "Move device to hostapd test App": "skipped",
      "Move device back to original app": "skipped",
      "Provisioning without deltas": "passed",
      "Override lock test": "passed",
      "Update supervisor randomized timer": "passed",
      "Set device environment variables": "passed",
      "Set service environment variables": "passed",
      "SSH authentication in production mode": "passed",
      "SSH authentication in development mode": "passed",
      "os-config service on boot": "passed",
      "os-config service randomized timer": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:43:52 GMT+0000 (Coordinated Universal Time)"
  },
  {
    "suite": "Hostapp update suite",
    "stats": {
      "tests": 5,
      "ran": 5,
      "skipped": 0,
      "passed": 5,
      "failed": 0
    },
    "tests": {
      "Broken balena-engine": "passed",
      "Broken VPN": "passed",
      "Rollback altboot (broken init) test": "passed",
      "HUP from previous release": "passed",
      "HUP from this release": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:58:25 GMT+0000 (Coordinated Universal Time)"
  },
  {
    "suite": "Unmanaged BalenaOS release suite",
    "stats": {
      "tests": 45,
      "ran": 36,
      "skipped": 9,
      "passed": 36,
      "failed": 0
    },
    "tests": {
      "check secure boot": "passed",
      "BeagleBone Black u-boot overlay test: deactivate HDMI": "skipped",
      "243390-rpi3 - CUS/EUS chipsets test": "skipped",
      "fingerprint file test": "passed",
      "ext4 filesystems are checked on boot": "passed",
      "OS-release file check": "passed",
      "Installer used migrator module": "passed",
      "issue file check": "passed",
      "issue.net file check": "passed",
      "Chronyd service": "passed",
      "Sync test": "passed",
      "Source test": "passed",
      "Offline sources test": "passed",
      "System time skew test": "passed",
      "kernel-overlap test": "passed",
      "Bluetooth scanning test": "skipped",
      "Container healthcheck test": "passed",
      "Container exposed variables test": "passed",
      "Identification test": "skipped",
      "Cellular tests": "passed",
      "hostname configuration test": "passed",
      "ntpServer test": "passed",
      "dnsServers test": "passed",
      "os.network.connectivity test": "passed",
      "os.network.wifi.randomMacAddressScan test": "passed",
      "udevRules test": "passed",
      "persistentLogging configuration test": "passed",
      "Reboot test": "skipped",
      "Wired test": "skipped",
      "Wireless test": "skipped",
      "Socks5 test": "passed",
      "Http-connect test": "passed",
      "Engine socket is exposed in development images": "passed",
      "Engine socket is not exposed in production images": "passed",
      "Engine watchdog recovery": "passed",
      "Engine healthcheck performance": "passed",
      "Under-voltage test": "passed",
      "Ramdisks, zram and loop devices are not scanned for rootfs": "passed",
      "by-state links are created": "passed",
      "DToverlay & DTparam tests": "skipped",
      "state partition reset": "passed",
      "data partition reset": "passed",
      "RevPi Core 3 DIO module test": "skipped",
      "zram is enabled and configured as swap": "passed",
      "Internet sharing iptables rules test": "passed"
    },
    "dateTime": "Thu Apr 27 2023 11:24:52 GMT+0000 (Coordinated Universal Time)"
  }
]

200+ assertions, 62 tests

Avg. runtime: 90-120 minutes

Breaking things intentionally to see if the OS can recover & update

6. Jenkins reports back results

With each downstream test job completed, Jenkins updates statuses of these jobs back on GitHub for the PR to be reviewed.

7. PR is merged 

With each downstream test job completed, Jenkins updates statuses of these jobs back on GitHub for the PR to be reviewed.

8. Merge === Deploy

Device repositories use this new meta-balena layer now available to deploy device-specific releases. These are built again, tested, and then merged to be deployed automatically. All using Jenkins.

balenaOS Hardware in the loop testing pipeline

The Journey

  • Propose changes
  • Build draft balenaOS releases from PR changes.
  • Run draft release on an actual device.
  • Test the release end-to-end by automating the device.
  • Gain feedback from the loop to make improvements
  • Make changes to the pull request.
  • Tests provide instant feedback on changes & reviews
  • PR gets merged & the release gets deployed. 

This is hardware in loop testing with Jenkins

Gettin' Exponential Gains

With Hardware in Loop Testing & Jenkins for CI/CD, we have managed to:

  • Reduce our OS release cycle from weeks to hours 
  • Achieve exponential scale to support new devices
  • Save thousands of hours in troubleshooting
  • Follow TDD & new tests being added every day

Most importantly, we even added QEMU support for testing virtual devices. Thanks to the Chip Shortage 2022

Autokit is testrunner/worker

running test on your device under test.

What if the hardware is not in stock?

???

Autokit is testrunner/worker

running test on your device under test.

What if the hardware is still under development?

???

Autokit is testrunner/worker

running test on your device under test.

What if someone parked their ship wrong?

Meet QEMU

Virtual worker/Testrunner

Worker

(controller)

Virtual Raspberry Pi 4-like device

(Our ARMv8 VM)

Source: https://wiki.qemu.org/Documentation/Platforms

It's also well supported

Raspberry Pi 3, our actual device.

  • Cortex-A53 (ARMv8) 64-bit SoC @ 1.4GHz
  • 1GB LPDDR2 SDRAM
  • 2.4GHz and 5GHz IEEE 802.11.b/g/n/ac wireless LAN
  • Gigabit Ethernet over USB 2.0
  • Extended 40-pin GPIO header
  • Full-size HDMI
  • 4 USB 2.0 ports
  • CSI camera port // DSI display port
  • 4-pole stereo output and composite video port

Meet the virtual Raspberry Pi 3.

  • 4x 64-bit cores
  • 1GB RAM 
  • Internet access from the host system
  • Stdin access is available with serial or even SSH
  • USB ports with pass-through
  • Video output feature to SDL or VGA 
  • Audio pass-through
  • Lot of storage 

QEMU emulates specifications, not machines.

But Vipul, how can I use this now at work?

Worker

(controller)

Remote devices must be directly controlled, automated, and maintained using scripts.

Worker

(controller)

Feedback on changes

Testing software directly on hardware
in a CI/CD pipeline

?

Worker

(controller)

?

?

?

?

?

?

?

?

?

?

?

Quality Assurance, stress testing, random testing, environmental testing

And, you can test Operating Systems

That's the talk.

What we learned

  • Testing operating systems is important
  • And, incredibly painful to scale
  • But doesn't have to be.
  • When you are actually testing on the hardware.

Resources

And, that's about it!

Questions? Collaborate? Work with us? Reach out!

@vipulgupta2048

Reviews cheesecakes, closes issues & runs Mixster to "right" the docs for startups

Feedback please + Link to the slides

[Open Source Summit Japan 2023] Deep dive into Hardware in the loop testing

By Vipul Gupta

Made with Slides.com

[Open Source Summit Japan 2023] Deep dive into Hardware in the loop testing

This presentation is about using Jenkins to test hundreds of OS images, specifically focusing on balenaOS and the challenges that come with testing embedded operating systems. It was used for a talk on balenaOS's testing pipeline using Jenkins that took place at cdCon Japan 2023 in Vancouver, Canada.

  • 385

More from Vipul Gupta

  • [Constellation 2024] How Balena Releases 100's of OS releases with GitHub Actions in Hours, Not Weeks
    Vipul Gupta
    301
  • [FOSS India 2024] Pains, Processes & the Eventual Gains of Building Open-Source Hardtech projects
    Vipul Gupta
    253
  • [OSSEU 23] Building a Virtual Raspberry Pi using QEMU
    Vipul Gupta
    408
  • [PyDelhiConf '23] Deploying Python on the edge: Mistakes, pain and leanings of scaling Python applications on millions of IoT devices
    Vipul Gupta
    331