Agenda
- Introduction
- What is oVirt
- Basic view of oVirt Infrastructure
- Release flow
- Gerrit hooks
- Jenkins jobs types
- Jenkins jobs builder
- Publishing builds to a repository server
- Future plans
- Questions
Introduction
- Who are we ?
Kiril Nesenko - Infrastructure Maintainer and release engineer. Responsible for release engineering activities for oVirt project.
Eyal Edri - Infrastructure Maintainer at oVirt and CI team lead at Red Hat.
WHAT IS OVIRT
- oVirt is a virtual data center manager
- Delivers powerful management of multiple virtual machines on multiple hosts
- Uses KVM, libvirt and vdsm.
- Can be installed on Fedora, CentOS or RHEL

ovirt architecture
OVIRT UI

ovirt infrastructure view

services





- Provisioning: Foreman
- Configuration: Puppet
- Artifacts (mirror): Artifactory
- Source Control: git
- Code Review: gerrit
- CI: Jenkins
- Slaves: VMs + Bare Metal hosts
- Virtualization: oVirt, OpenStack

RELEASE PROCESS

commit life cycle

ovirt continuous delivery
- One publisher per version
- Use a generic script to publish the builds

publisher job

publisher job

gerrit
What is Gerrit?


GERRIT: gitweb
Code hosting (git frontend)

GERRIT HOOK
What is a hook?
For each workflow step (event) a script gets executed

Gerrit HOOKS
Default gerrit hooks flaws:
- 1 event <-> 1 hook
- Same hook for all projects
- Fire and forget, no interaction
Solution:
Build a dispatcher!
GERRIT HOOKS: TRIgger
- Patch Created
- Comment Added
- Change Merged
- Change Abandoned
- Reference updated
- (more on newer versions)

GERRIT HOOKS: EVENT type
- By parameters passed
-
We will be able to detect
- Maintainer approved
- Tests passed
- Maintainer blocked
- ...

GERRIT HOOKS: Labels
What?
Label: label content
Where?
Commit message
Comment
Example:
Bug-Url: http://mytracker/bugid
Rerun-Hooks: all

GERRIT HOOKS: CHAINS
- Using event name
- Chain as second term if wanted
- Hierarchical config files
- Main one at $gerrit_dir
- Per project
- Per event
Hook name template:
${event}.${chain}.${hook_name}
${event}.${hook_name}
${event}

GERRIT HOOKS: EXECUTION
INFO::STARTING::change-merged
DEBUG::get_hooks::change-merged on vdsm.git/hooks
INFO::::AVAILABLE HOOKS::[
'change-merged.set_MODIFIED',
'change-merged.update_tracker']
Each hook can:
- Break it's hook chain
- Break whole execution
Everything to logfile (configurable):
Common stdout format (each section is optional):
V:VERIFIED SCORE\n
R:REVIEW SCORE\n
COMMENT

GERRIT HOOKS: REVIEW
Sum of all the reviews:
- negative ones prevail over positives
- no vote means no change
- 0 means reset negative value

Execution summary of all the hooks:

GERRIT HOOKS: Source CODE
Patches are always welcome!
- Git:
- git clone git://gerrit.ovirt.org/gerrit-admin
- Web:
- http://gerrit.ovirt.org/gitweb?p=gerrit-admin.gi t

GERRIT HOOKS: LIBS
Some simple generic libs available for the hooks:
- Bugzilla
- Gerrit
- Configuration
- Tools
source bz.sh
source gerrit.sh
gerrit.parse_params "$@" # Sets the parameters into global vars
# get the bug ids in the commit message of the commit passed as
# parameter (--commit SHA)
bug_ids=($(bz.get_bug_id "$commit"))
from gerrit import Gerrit
from config import load_config
config = load_config() ## Loads the config hierarchicaly into a dict
gerrit = Gerrit(config.get('GERRIT_SRV')) ## Connect to the gerrit instance
GERRIT HOOKS: BUGZILLA
Update external tracker in bugzilla

Modify bug status when first patch sent

Modify bug status when all patches merged

Jenkins job builder
Problems with manual configuration
- Hard to maintain
- Hard to track
- Hard to review
- Hard to change common settings
...
Add an extra scm to jobs for projects A, B and C
New branch for project M, duplicate all it's jobs
Disable all the jobs for a project
Test a job configuration before deploying
Jenkins job builder

Jenkins job Builder to the rescue!
jenkins job builder
Some plugins that helped along the way:
- Configuration deduplication:
Matrix configurations
- Multijobs config changes:
Configuration slice
- Configuation change management:
Job Config history
- Manage jobs statuses:
Scriptler
Jenkins job builder
- Developed by OpenStack folks (Kudos!!)
- Configuration as code (yaml format)
- Easy to review changes (on Gerrit!)
- Code/config de-duplication
- Include external shell/groovy/... scripts
- Test before deploying
- Easier to organize (per directory, per file)
- Serves as backup (easily replicate on another jenkins)
Jenkins job builder
- This will create the jobs:
- projectA-whatever-fc19
- projectA-whatever-fc20
- job-group:
name: projectA
dist:
- fc19
- fc20
jobs:
- '{name}-wahtever-{dist}'
- job-template:
name: '{name}-whatever-{dist}'
node: 'slave-{dist}'
triggers:
- timed: '@midnight'
builders:
- shell: make whatever
publishers:
- archive:
artifacts: '*.log'
On slave-fc19 and slave-fc20
Executing 'make whatever'
Archiving the logs
Jenkins job builder
Oh! But I want to add a new distro, el6
- job-group:
name: projectA
dist:
- fc19
- fc20
- el6
jobs:
- '{name}-wahtever-{dist}'
And a cleanup after
- job-template:
name: '{name}-whatever-{dist}'
node: 'slave-{dist}'
triggers:
- timed: '@midnight'
builders:
- shell: make whatever
- shell: make clean
publishers:
- archive:
artifacts: '*.log'
Jenkins job builder
Official Openstack links:
Docs:
Code:
oVirt links
ovirt jenkins job types
- Triggered only on patch sets
- Triggered only on merges
- Matrix jobs
- COPR jobs
poll scm vs gerrit trigger
- We use gerrit trigger plugin instead of poll SCM
- Allows to test patches before merge
- Reduces the load on the gerrit server

matrix jobs
- Allows to configure the jobs to run on different configurations

matrix reloadeD
-
Allows to rerun the job on a specific configuration

copr
- Created by Fedora folks
- Is an easy to use automatic build system providing a package repository as its output
- Reduces the load on jenkins slaves
- CLI + Jenkins plugin
yum -y install copr-cli
copr-cli build vdsm http://jenkins.ovirt.org/view/vdsm-4.14.10-11.git1081e76.el6.src.rpm
copr jenkins plugin

release process

Current problems/future solutions
Scarce resources
- Moving to a new infrastructure
- OS1 (openstack) slaves
Jobs management and control
- Move all jobs to yaml
- Get developers to help write/maintain jobs
Leverage manual procedures
- Automate the official release + signing packages
- Automate gerrit config/project creation
Current problems/future solutions
Isolate test environments
- Docker/mock on all simple builds
- One slave per job on complex builds
-
Provisioning of slaves on demand
Leverage package builds
- Build on Copr (has jenkins plugin!)
questions ?

useful Links
contact us

jenkins_conference_2014
By Kiril Nesenko
jenkins_conference_2014
- 2,948