Manage ovirt infrastructure with jenkins




Kiril Nesenko, Infrastructure Maintainer at oVirt

David Caro, Software Enigneer at Red Hat


Agenda

  • Introduction
  • What is oVirt
  • Basic view of oVirt Infrastructure
  • Release flow
  • Gerrit hooks
  • Jenkins jobs types
  • Jenkins jobs builder
  • Publishing artifacts to repository server
  • Managing rolling release
  • Future plans
  • Questions

Introduction


  • Who we are ?


Kiril Nesenko - Infrastructure Maintainer and release engineer.  Responsible for release engineering activities for oVirt project.

David Caro - Infrastructure Maintainer and release engineer. Works at Red Hat as a Software Engineer. Responsible for all CI and systems administration for oVirt and RHEV projects.

release process

* prepare the diagram related to "Speaker notes"

publish the builds

  • Publisher job per version
  • Take a last successful build from the job and upload to the repository server
  • Run a cronjob on resources.ovirt.org to publish the build


* prepare a diagram

publisher job


* show the publisher screenshot

WHAT IS OVIRT

oVirt is a virtual datacenter manager that delivers powerful management of multiple virtual machines on multiple hosts. Using KVM and libvirt, oVirt can be installed on Fedora, CentOS, or Red Hat Enterprise Linux hosts to set up and manage your virtual data center.



ovirt architecture

 OVIRT UI

ovirt infrastructure view

servers map

services

  • Provisioning: Foreman
  • Configuration: Puppet
  • Artifacts: Artifactory
  • Source Control: git
  • Code Review: gerrit
  • CI: Jenkins
  • Slaves: VMs + Bare Metal hosts
  • Virtualization: oVirt, OpenStack



gerrit

What is Gerrit?

GERRIT: gitweb

Code hosting (git frontend)

GERRIT HOOK

What is a hook?
For each workflow step (event) a script gets executed



Gerrit HOOKS

                    Default gerrit hooks flaws:

  • 1 event <-> 1 hook
  • Same hook for all projects
  • Fire and forget, no interaction



                      Solution:

Build a dispatcher!


GERRIT HOOKS: TRIgger




  • Patch Created
  • Comment Added
  • Change Merged
  • Change Abandoned
  • Reference updated
  • (more on newer versions)


GERRIT HOOKS: EVENT type


  • By parameters passed
  • We will be able to detect
custom events:
  • Maintainer approved
  • Tests passed
  • Maintainer blocked
  • ...


GERRIT HOOKS: Labels


What?

Label: label content


Where?

Commit message

Comment

Example:

Bug-Url: http://mytracker/bugid 
Rerun-Hooks: all 

GERRIT HOOKS: CHAINS


  • Using event name
  • Chain as second term if wanted
  • Hierarchical config files
    • Main one at $gerrit_dir
    • Per project
    • Per event








Hook name template:

${event}.${chain}.${hook_name}
${event}.${hook_name}
${event} 

GERRIT HOOKS: EXECUTION

INFO::STARTING::change-merged
DEBUG::get_hooks::change-merged on vdsm.git/hooks
INFO::::AVAILABLE HOOKS::[
    'change-merged.set_MODIFIED',
    'change-merged.update_tracker'] 

Each hook can:

  • Break it's hook chain
  • Break whole execution

Everything to logfile (configurable):

Common stdout format (each section is optional):

V:VERIFIED SCORE\n
R:REVIEW SCORE\n
COMMENT

GERRIT HOOKS: REVIEW


Sum of all the reviews:

  • negative ones prevail over positives
  • no vote means no change
  • 0 means reset negative value




Execution summary of all the hooks:

GERRIT HOOKS: Source CODE










  Patches are always welcome!                 
  • Git:
    •  git clone git://gerrit.ovirt.org/gerrit-admin
  • Web:
    • http://gerrit.ovirt.org/gitweb?p=gerrit-admin.git

GERRIT HOOKS: LIBS

Some simple generic libs available for the hooks:

  • Bugzilla
  • Gerrit
  • Configuration
  • Tools




 Bash lib usage example:
source bz.sh
source gerrit.sh
gerrit.parse_params "$@"  # Sets the parameters into global vars
# get the bug ids in the commit message of the commit passed as 
# parameter (--commit SHA)
bug_ids=($(bz.get_bug_id "$commit"))
 Python lib usage example:
from gerrit import Gerrit
from config import load_config
config = load_config() ## Loads the config hierarchicaly into a dict
gerrit = Gerrit(config.get('GERRIT_SRV')) ## Connect to the gerrit instance

GERRIT HOOKS: BUGZILLA

Update external tracker in bugzilla

Modify bug status when first patch sent


Modify bug status when all patches merged

Jenkins job builder

Problems with manual configuration

  • Hard to maintain
  • Hard to track
  • Hard to review
  • Hard to change common settings

Some tasks soon become quite painful:
...
Add an extra scm to jobs for projects A, B and C
New branch for project M, duplicate all it's jobs
Disable all the jobs for a project
Test a job configuration before deploying

Jenkins job builder










Jenkins job Builder to the rescue!

jenkins job builder

Some plugins that helped along the way:

  • Configuration deduplication:

Matrix configurations

  • Multijobs config changes:

Configuration slice

  • Configuation change management:

Job Config history

  • Manage jobs statuses:

Scriptler

Jenkins job builder


  • Developed by OpenStack folks (Kudos!!)
  • Configuration as code (yaml format)
  • Easy to review changes (on Gerrit!)
  • Code/config de-duplication
  • Include external shell/groovy/... scripts
  • Test before deploying
  • Easier to organize (per directory, per file)
  • Serves as backup (easily replicate on another jenkins)


Jenkins job builder

  • This will create the jobs:
    • projectA-whatever-fc19
    • projectA-whatever-fc20

- job-group:
    name: projectA
    dist:
        - fc19
        - fc20
    jobs:
      - '{name}-wahtever-{dist}' 
- job-template:
    name: '{name}-whatever-{dist}'
    node: 'slave-{dist}'
    triggers:
      - timed: '@midnight'
    builders:
      - shell: make whatever
    publishers:
      - archive:
           artifacts: '*.log'
Running at midnigh
On slave-fc19 and slave-fc20
Executing 'make whatever'
Archiving the logs

Jenkins job builder

Oh! But I want to add a new distro, el6

- job-group:
    name: projectA
    dist:
        - fc19
        - fc20
        - el6
    jobs:
      - '{name}-wahtever-{dist}' 






And a cleanup after

- job-template:
    name: '{name}-whatever-{dist}'
    node: 'slave-{dist}'
    triggers:
      - timed: '@midnight'
    builders:
      - shell: make whatever
      - shell: make clean
    publishers:
      - archive:
           artifacts: '*.log'

Jenkins job builder

Official Openstack links:

Docs:

Latest build docs

Code:

Https git repo

oVirt links

Git yaml code tree

Deploy job


Check job

Current problems/future solutions

Scarce resources

  • Moving to a new infrastructure
  • OS1 (openstack) slaves


Jobs management and control
  • Move all jobs to yaml
  • Get developers to help write/maintain jobs


Leverage manual procedures
  • Automate the official release + signing pkgs
  • Automate gerit config/project creation

Current problems/future solutions

Isolate test environments

  • Docker/mock on all simple builds
  • One slave per job on complex builds
    • Provisining of slaves on demand


Leverage package builds

  • Build on Copr (has jenkins plugin!)



Copy of jenkins_conference_2014_david

By Kiril Nesenko

Copy of jenkins_conference_2014_david

  • 1,565