Copy of jenkins_conference_2014

Manage ovirt infrastructure with jenkins

Kiril Nesenko, Infrastructure Maintainer at oVirt

David Caro, Software Enigneer at Red Hat

Agenda

Introduction
What is oVirt
Basic view of oVirt Infrastructure
Release flow
Gerrit hooks
Jenkins jobs types
Jenkins jobs builder
Publishing artifacts to repository server
Managing rolling release
Future plans
Questions

Introduction

Who we are ?

Kiril Nesenko - Infrastructure Maintainer and release engineer. Responsible for release engineering activities for oVirt project.

David Caro - Infrastructure Maintainer and release engineer. Works at Red Hat as a Software Engineer. Responsible for all CI and systems administration for oVirt and RHEV projects.

release process

* prepare the diagram related to "Speaker notes"

publish the builds

Publisher job per version
Take a last successful build from the job and upload to the repository server
Run a cronjob on resources.ovirt.org to publish the build

* prepare a diagram

publisher job

* show the publisher screenshot

WHAT IS OVIRT

oVirt is a virtual datacenter manager that delivers powerful management of multiple virtual machines on multiple hosts. Using KVM and libvirt, oVirt can be installed on Fedora, CentOS, or Red Hat Enterprise Linux hosts to set up and manage your virtual data center.

ovirt architecture

OVIRT UI

ovirt infrastructure view

servers map

services

Provisioning: Foreman
Configuration: Puppet
Artifacts: Artifactory
Source Control: git
Code Review: gerrit
CI: Jenkins
Slaves: VMs + Bare Metal hosts
Virtualization: oVirt, OpenStack

gerrit

What is Gerrit?

GERRIT: gitweb

Code hosting (git frontend)

GERRIT HOOK

What is a hook?
For each workflow step (event) a script gets executed

Gerrit HOOKS

Default gerrit hooks flaws:

1 event <-> 1 hook
Same hook for all projects
Fire and forget, no interaction

Solution:

Build a dispatcher!

GERRIT HOOKS: TRIgger

Patch Created
Comment Added
Change Merged
Change Abandoned
Reference updated
(more on newer versions)

GERRIT HOOKS: EVENT type

By parameters passed
We will be able to detect

custom events:

Maintainer approved
Tests passed
Maintainer blocked
...

GERRIT HOOKS: Labels

What?

Label: label content

Where?

Commit message

Comment

Example:

Bug-Url: http://mytracker/bugid

Rerun-Hooks: all

GERRIT HOOKS: CHAINS

Using event name
Chain as second term if wanted
Hierarchical config files

Main one at $gerrit_dir
Per project
Per event

Hook name template:

${event}.${chain}.${hook_name}
${event}.${hook_name}
${event}

GERRIT HOOKS: EXECUTION

INFO::STARTING::change-merged
DEBUG::get_hooks::change-merged on vdsm.git/hooks
INFO::::AVAILABLE HOOKS::[
    'change-merged.set_MODIFIED',
    'change-merged.update_tracker']

Each hook can:

Break it's hook chain
Break whole execution

Everything to logfile (configurable):

Common stdout format (each section is optional):

V:VERIFIED SCORE\n
R:REVIEW SCORE\n
COMMENT

GERRIT HOOKS: REVIEW

Sum of all the reviews:

negative ones prevail over positives
no vote means no change
0 means reset negative value

Execution summary of all the hooks:

GERRIT HOOKS: Source CODE

Patches are always welcome!

Git:

git clone git://gerrit.ovirt.org/gerrit-admin

Web:

http://gerrit.ovirt.org/gitweb?p=gerrit-admin.git

GERRIT HOOKS: LIBS

Some simple generic libs available for the hooks:

Bugzilla

Gerrit

Configuration
Tools

Bash lib usage example:

source bz.sh
source gerrit.sh
gerrit.parse_params "$@"  # Sets the parameters into global vars
# get the bug ids in the commit message of the commit passed as 
# parameter (--commit SHA)
bug_ids=($(bz.get_bug_id "$commit"))

Python lib usage example:

from gerrit import Gerrit
from config import load_config
config = load_config() ## Loads the config hierarchicaly into a dict
gerrit = Gerrit(config.get('GERRIT_SRV')) ## Connect to the gerrit instance

GERRIT HOOKS: BUGZILLA

Update external tracker in bugzilla

Modify bug status when first patch sent

Modify bug status when all patches merged

Jenkins job builder

Problems with manual configuration

Hard to maintain
Hard to track
Hard to review
Hard to change common settings

Some tasks soon become quite painful:

...

Add an extra scm to jobs for projects A, B and C

New branch for project M, duplicate all it's jobs

Disable all the jobs for a project

Test a job configuration before deploying

Jenkins job builder

Jenkins job Builder to the rescue!

jenkins job builder

Some plugins that helped along the way:

Configuration deduplication:

Matrix configurations

Multijobs config changes:

Configuration slice

Configuation change management:

Job Config history

Manage jobs statuses:

Scriptler

Jenkins job builder

Developed by OpenStack folks (Kudos!!)
Configuration as code (yaml format)
Easy to review changes (on Gerrit!)
Code/config de-duplication
Include external shell/groovy/... scripts
Test before deploying
Easier to organize (per directory, per file)
Serves as backup (easily replicate on another jenkins)

Jenkins job builder

This will create the jobs:

projectA-whatever-fc19
projectA-whatever-fc20

- job-group:
    name: projectA
    dist:
        - fc19
        - fc20
    jobs:
      - '{name}-wahtever-{dist}'

- job-template:
    name: '{name}-whatever-{dist}'
    node: 'slave-{dist}'
    triggers:
      - timed: '@midnight'
    builders:
      - shell: make whatever
    publishers:
      - archive:
           artifacts: '*.log'

Running at midnigh
On slave-fc19 and slave-fc20
Executing 'make whatever'
Archiving the logs

Jenkins job builder

Oh! But I want to add a new distro, el6

- job-group:
    name: projectA
    dist:
        - fc19
        - fc20
        - el6
    jobs:
      - '{name}-wahtever-{dist}'

And a cleanup after

- job-template:
    name: '{name}-whatever-{dist}'
    node: 'slave-{dist}'
    triggers:
      - timed: '@midnight'
    builders:
      - shell: make whatever
      - shell: make clean
    publishers:
      - archive:
           artifacts: '*.log'

Jenkins job builder

Official Openstack links:

Docs:

Latest build docs

Code:

Https git repo

oVirt links

Git yaml code tree

Deploy job

Check job

Current problems/future solutions

Scarce resources

Moving to a new infrastructure
OS1 (openstack) slaves

Jobs management and control

Move all jobs to yaml
Get developers to help write/maintain jobs

Leverage manual procedures

Automate the official release + signing pkgs
Automate gerit config/project creation

Current problems/future solutions

Isolate test environments

Docker/mock on all simple builds
One slave per job on complex builds

Provisining of slaves on demand

Leverage package builds

Build on Copr (has jenkins plugin!)