Agenda

  • Introduction
  • What is oVirt
  • Basic view of oVirt Infrastructure
  • Release flow
  • Gerrit hooks
  • Jenkins jobs types
  • Jenkins jobs builder
  • Publishing builds to a repository server
  • Future plans
  • Questions

Introduction

  • Who are we ?


Kiril Nesenko - Infrastructure Maintainer and release engineer.  Responsible for release engineering activities for oVirt project.

Eyal Edri - Infrastructure Maintainer at oVirt and CI team lead at Red Hat.

WHAT IS OVIRT

  • oVirt is a virtual data center manager
  • Delivers powerful management of multiple virtual machines on multiple hosts
  • Uses KVM, libvirt and vdsm.
  • Can be installed on Fedora, CentOS or RHEL




ovirt architecture

 OVIRT UI

ovirt infrastructure view


services

  • Provisioning: Foreman
  • Configuration: Puppet
  • Artifacts (mirror): Artifactory
  • Source Control: git
  • Code Review: gerrit
  • CI: Jenkins
  • Slaves: VMs + Bare Metal hosts
  • Virtualization: oVirt, OpenStack

RELEASE PROCESS


commit life cycle


ovirt continuous delivery

  • One publisher per version
  • Use a generic script to publish the builds

publisher job





publisher job




gerrit

What is Gerrit?

GERRIT: gitweb

Code hosting (git frontend)

GERRIT HOOK

What is a hook?
For each workflow step (event) a script gets executed



Gerrit HOOKS

                    Default gerrit hooks flaws:

  • 1 event <-> 1 hook
  • Same hook for all projects
  • Fire and forget, no interaction



                      Solution:

Build a dispatcher!


GERRIT HOOKS: TRIgger




  • Patch Created
  • Comment Added
  • Change Merged
  • Change Abandoned
  • Reference updated
  • (more on newer versions)


GERRIT HOOKS: EVENT type


  • By parameters passed
  • We will be able to detect
custom events:
  • Maintainer approved
  • Tests passed
  • Maintainer blocked
  • ...


GERRIT HOOKS: Labels


What?

Label: label content


Where?

Commit message

Comment

Example:


    Bug-Url: http://mytracker/bugid 



    Rerun-Hooks: all

GERRIT HOOKS: CHAINS


  • Using event name
  • Chain as second term if wanted
  • Hierarchical config files
    • Main one at $gerrit_dir
    • Per project
    • Per event








Hook name template:


    ${event}.${chain}.${hook_name}
${event}.${hook_name}
${event}

GERRIT HOOKS: EXECUTION

INFO::STARTING::change-merged
DEBUG::get_hooks::change-merged on vdsm.git/hooks
INFO::::AVAILABLE HOOKS::[
    'change-merged.set_MODIFIED',
    'change-merged.update_tracker']

Each hook can:

  • Break it's hook chain
  • Break whole execution

Everything to logfile (configurable):

Common stdout format (each section is optional):

V:VERIFIED SCORE\n
R:REVIEW SCORE\n
COMMENT

GERRIT HOOKS: REVIEW


Sum of all the reviews:

  • negative ones prevail over positives
  • no vote means no change
  • 0 means reset negative value




Execution summary of all the hooks:

GERRIT HOOKS: Source CODE










  Patches are always welcome!                 
  • Git:
    •  git clone git://gerrit.ovirt.org/gerrit-admin
  • Web:
    • http://gerrit.ovirt.org/gitweb?p=gerrit-admin.gi t

GERRIT HOOKS: LIBS

Some simple generic libs available for the hooks:

  • Bugzilla
  • Gerrit
  • Configuration
  • Tools




 Bash lib usage example:
source bz.sh
source gerrit.sh
gerrit.parse_params "$@"  # Sets the parameters into global vars
# get the bug ids in the commit message of the commit passed as 
# parameter (--commit SHA)
bug_ids=($(bz.get_bug_id "$commit"))
 Python lib usage example:
from gerrit import Gerrit
from config import load_config
config = load_config() ## Loads the config hierarchicaly into a dict
gerrit = Gerrit(config.get('GERRIT_SRV')) ## Connect to the gerrit instance

GERRIT HOOKS: BUGZILLA

Update external tracker in bugzilla

Modify bug status when first patch sent


Modify bug status when all patches merged

Jenkins job builder

Problems with manual configuration

  • Hard to maintain
  • Hard to track
  • Hard to review
  • Hard to change common settings

Some tasks soon become quite painful:
...
Add an extra scm to jobs for projects A, B and C
New branch for project M, duplicate all it's jobs
Disable all the jobs for a project
Test a job configuration before deploying

Jenkins job builder










Jenkins job Builder to the rescue!

jenkins job builder

Some plugins that helped along the way:

  • Configuration deduplication:

Matrix configurations

  • Multijobs config changes:

Configuration slice

  • Configuation change management:

Job Config history

  • Manage jobs statuses:

Scriptler

Jenkins job builder


  • Developed by OpenStack folks (Kudos!!)
  • Configuration as code (yaml format)
  • Easy to review changes (on Gerrit!)
  • Code/config de-duplication
  • Include external shell/groovy/... scripts
  • Test before deploying
  • Easier to organize (per directory, per file)
  • Serves as backup (easily replicate on another jenkins)


Jenkins job builder

  • This will create the jobs:
    • projectA-whatever-fc19
    • projectA-whatever-fc20

- job-group:
    name: projectA
    dist:
        - fc19
        - fc20
    jobs:
      - '{name}-wahtever-{dist}' 
- job-template:
    name: '{name}-whatever-{dist}'
    node: 'slave-{dist}'
    triggers:
      - timed: '@midnight'
    builders:
      - shell: make whatever
    publishers:
      - archive:
           artifacts: '*.log'
Running at midnigh
On slave-fc19 and slave-fc20
Executing 'make whatever'
Archiving the logs

Jenkins job builder

Oh! But I want to add a new distro, el6

- job-group:
    name: projectA
    dist:
        - fc19
        - fc20
        - el6
    jobs:
      - '{name}-wahtever-{dist}'






And a cleanup after 

- job-template:
    name: '{name}-whatever-{dist}'
    node: 'slave-{dist}'
    triggers:
      - timed: '@midnight'
    builders:
      - shell: make whatever
      - shell: make clean
    publishers:
      - archive:
           artifacts: '*.log'

Jenkins job builder

Official Openstack links:

Docs:

Latest build docs

Code:

Https git repo

oVirt links

Git yaml code tree

Deploy job


Check job

ovirt jenkins job types

  • Triggered only on patch sets
  • Triggered only on merges
  • Matrix jobs
  • COPR jobs


poll scm vs gerrit trigger

  • We use gerrit trigger plugin instead of poll SCM
  • Allows to test patches before merge
  • Reduces the load on the gerrit server


matrix jobs

  • Allows to configure the jobs to run on different configurations



matrix reloadeD

  • Allows to rerun the job on a specific configuration




copr

  • Created by Fedora folks
  • Is an easy to use automatic build system providing a package repository as its output
  • Reduces the load on jenkins slaves
  • CLI + Jenkins plugin
yum -y install copr-cli
copr-cli build vdsm http://jenkins.ovirt.org/view/vdsm-4.14.10-11.git1081e76.el6.src.rpm

copr jenkins plugin


release process

Current problems/future solutions

Scarce resources

  • Moving to a new infrastructure
  • OS1 (openstack) slaves


Jobs management and control
  • Move all jobs to yaml
  • Get developers to help write/maintain jobs


Leverage manual procedures
  • Automate the official release + signing packages
  • Automate gerrit config/project creation

Current problems/future solutions

Isolate test environments

  • Docker/mock on all simple builds
  • One slave per job on complex builds
    • Provisioning of slaves on demand


Leverage package builds

  • Build on Copr (has jenkins plugin!)



questions ?


useful Links

oVirt

oVirt gerrit

oVirt Jenkins

Copr

Jenkins Job Builder Code

Jenkins Job Builder Docs

YAML code tree

contact us


Kiril Nesenko: nesenko@gmail.com
Eyal Edri: eedri@redhat.com


jenkins_conference_2014

By Kiril Nesenko

Made with Slides.com

jenkins_conference_2014

  • 2,927

More from Kiril Nesenko