jenkins_conference

Agenda

Introduction
What is oVirt
Basic view of oVirt Infrastructure
Release flow
Gerrit hooks
Jenkins jobs types
Jenkins jobs builder
Publishing builds to a repository server
Future plans
Questions

Introduction

Who are we ?

Kiril Nesenko - Infrastructure Maintainer and release engineer. Responsible for release engineering activities for oVirt project.

Eyal Edri - Infrastructure Maintainer at oVirt and CI team lead at Red Hat.

WHAT IS OVIRT

oVirt is a virtual data center manager
Delivers powerful management of multiple virtual machines on multiple hosts
Uses KVM, libvirt and vdsm.
Can be installed on Fedora, CentOS or RHEL

ovirt architecture

OVIRT UI

ovirt infrastructure view

services

Provisioning: Foreman
Configuration: Puppet
Artifacts (mirror): Artifactory
Source Control: git
Code Review: gerrit
CI: Jenkins
Slaves: VMs + Bare Metal hosts
Virtualization: oVirt, OpenStack

RELEASE PROCESS

commit life cycle

ovirt continuous delivery

One publisher per version
Use a generic script to publish the builds

publisher job

publisher job

gerrit

What is Gerrit?

GERRIT: gitweb

Code hosting (git frontend)

GERRIT HOOK

What is a hook?
For each workflow step (event) a script gets executed

Gerrit HOOKS

Default gerrit hooks flaws:

1 event <-> 1 hook
Same hook for all projects
Fire and forget, no interaction

Solution:

Build a dispatcher!

GERRIT HOOKS: TRIgger

Patch Created
Comment Added
Change Merged
Change Abandoned
Reference updated
(more on newer versions)

GERRIT HOOKS: EVENT type

By parameters passed
We will be able to detect

custom events:

Maintainer approved
Tests passed
Maintainer blocked
...

GERRIT HOOKS: Labels

What?

Label: label content

Where?

Commit message

Comment

Example:


    Bug-Url: http://mytracker/bugid


    Rerun-Hooks: all

GERRIT HOOKS: CHAINS

Using event name
Chain as second term if wanted
Hierarchical config files

Main one at $gerrit_dir
Per project
Per event

Hook name template:


    ${event}.${chain}.${hook_name}
${event}.${hook_name}
${event}

GERRIT HOOKS: EXECUTION

INFO::STARTING::change-merged
DEBUG::get_hooks::change-merged on vdsm.git/hooks
INFO::::AVAILABLE HOOKS::[
    'change-merged.set_MODIFIED',
    'change-merged.update_tracker']

Each hook can:

Break it's hook chain
Break whole execution

Everything to logfile (configurable):

Common stdout format (each section is optional):

V:VERIFIED SCORE\n
R:REVIEW SCORE\n
COMMENT

GERRIT HOOKS: REVIEW

Sum of all the reviews:

negative ones prevail over positives
no vote means no change
0 means reset negative value

Execution summary of all the hooks:

GERRIT HOOKS: Source CODE

Patches are always welcome!

Git:

git clone git://gerrit.ovirt.org/gerrit-admin

Web:

http://gerrit.ovirt.org/gitweb?p=gerrit-admin.gi t

GERRIT HOOKS: LIBS

Some simple generic libs available for the hooks:

Bugzilla

Gerrit

Configuration
Tools

Bash lib usage example:

source bz.sh
source gerrit.sh
gerrit.parse_params "$@"  # Sets the parameters into global vars
# get the bug ids in the commit message of the commit passed as 
# parameter (--commit SHA)
bug_ids=($(bz.get_bug_id "$commit"))

Python lib usage example:

from gerrit import Gerrit
from config import load_config
config = load_config() ## Loads the config hierarchicaly into a dict
gerrit = Gerrit(config.get('GERRIT_SRV')) ## Connect to the gerrit instance

GERRIT HOOKS: BUGZILLA

Update external tracker in bugzilla

Modify bug status when first patch sent

Modify bug status when all patches merged

Jenkins job builder

Problems with manual configuration

Hard to maintain
Hard to track
Hard to review
Hard to change common settings

Some tasks soon become quite painful:

...

Add an extra scm to jobs for projects A, B and C

New branch for project M, duplicate all it's jobs

Disable all the jobs for a project

Test a job configuration before deploying

Jenkins job builder

Jenkins job Builder to the rescue!

jenkins job builder

Some plugins that helped along the way:

Configuration deduplication:

Matrix configurations

Multijobs config changes:

Configuration slice

Configuation change management:

Job Config history

Manage jobs statuses:

Scriptler

Jenkins job builder

Developed by OpenStack folks (Kudos!!)
Configuration as code (yaml format)
Easy to review changes (on Gerrit!)
Code/config de-duplication
Include external shell/groovy/... scripts
Test before deploying
Easier to organize (per directory, per file)
Serves as backup (easily replicate on another jenkins)

Jenkins job builder

This will create the jobs:

projectA-whatever-fc19
projectA-whatever-fc20

- job-group:
    name: projectA
    dist:
        - fc19
        - fc20
    jobs:
      - '{name}-wahtever-{dist}'

- job-template:
    name: '{name}-whatever-{dist}'
    node: 'slave-{dist}'
    triggers:
      - timed: '@midnight'
    builders:
      - shell: make whatever
    publishers:
      - archive:
           artifacts: '*.log'

Running at midnigh
On slave-fc19 and slave-fc20
Executing 'make whatever'
Archiving the logs

Jenkins job builder

Oh! But I want to add a new distro, el6

- job-group:
    name: projectA
    dist:
        - fc19
        - fc20
        - el6
    jobs:
      - '{name}-wahtever-{dist}'

And a cleanup after

- job-template:
    name: '{name}-whatever-{dist}'
    node: 'slave-{dist}'
    triggers:
      - timed: '@midnight'
    builders:
      - shell: make whatever
      - shell: make clean
    publishers:
      - archive:
           artifacts: '*.log'

Jenkins job builder

Official Openstack links:

Docs:

Latest build docs

Code:

Https git repo

oVirt links

Git yaml code tree

Deploy job

Check job

ovirt jenkins job types

Triggered only on patch sets
Triggered only on merges
Matrix jobs
COPR jobs

poll scm vs gerrit trigger

We use gerrit trigger plugin instead of poll SCM
Allows to test patches before merge
Reduces the load on the gerrit server

matrix jobs

Allows to configure the jobs to run on different configurations

matrix reloadeD

Allows to rerun the job on a specific configuration

copr

Created by Fedora folks
Is an easy to use automatic build system providing a package repository as its output
Reduces the load on jenkins slaves
CLI + Jenkins plugin

yum -y install copr-cli
copr-cli build vdsm http://jenkins.ovirt.org/view/vdsm-4.14.10-11.git1081e76.el6.src.rpm

copr jenkins plugin

release process

Current problems/future solutions

Scarce resources

Moving to a new infrastructure
OS1 (openstack) slaves

Jobs management and control

Move all jobs to yaml
Get developers to help write/maintain jobs

Leverage manual procedures

Automate the official release + signing packages
Automate gerrit config/project creation

Current problems/future solutions

Isolate test environments

Docker/mock on all simple builds
One slave per job on complex builds

Provisioning of slaves on demand

Leverage package builds

Build on Copr (has jenkins plugin!)

questions ?

useful Links

Jenkins Job Builder Code

Jenkins Job Builder Docs

YAML code tree

contact us

Kiril Nesenko: nesenko@gmail.com

Eyal Edri: eedri@redhat.com