Manage ovirt infrastructure with jenkins
Kiril Nesenko, Infrastructure Maintainer at oVirt
David Caro, Software Enigneer at Red Hat
Agenda
- Introduction
- What is oVirt
- Basic view of oVirt Infrastructure
- Release flow
- Gerrit hooks
- Jenkins jobs types
- Jenkins jobs builder
- Publishing artifacts to repository server
- Managing rolling release
- Future plans
- Questions
Introduction
- Who we are ?
Kiril Nesenko - Infrastructure Maintainer and release engineer. Responsible for release engineering activities for oVirt project.
David Caro - Infrastructure Maintainer and release engineer. Works at Red Hat as a Software Engineer. Responsible for all CI and systems administration for oVirt and RHEV projects.
release process
* prepare the diagram related to "Speaker notes"
publish the builds
- Publisher job per version
- Take a last successful build from the job and upload to the repository server
- Run a cronjob on resources.ovirt.org to publish the build
* prepare a diagram
publisher job
* show the publisher screenshot
WHAT IS OVIRT
oVirt is a virtual datacenter manager that delivers powerful management of multiple virtual machines on multiple hosts. Using KVM and libvirt, oVirt can be installed on Fedora, CentOS, or Red Hat Enterprise Linux hosts to set up and manage your virtual data center.
ovirt architecture
OVIRT UI
ovirt infrastructure view
servers map
services
- Provisioning: Foreman
- Configuration: Puppet
- Artifacts: Artifactory
- Source Control: git
- Code Review: gerrit
- CI: Jenkins
- Slaves: VMs + Bare Metal hosts
- Virtualization: oVirt, OpenStack
gerrit
What is Gerrit?
GERRIT: gitweb
Code hosting (git frontend)
GERRIT HOOK
What is a hook?
For each workflow step (event) a script gets executed
Gerrit HOOKS
Default gerrit hooks flaws:
- 1 event <-> 1 hook
- Same hook for all projects
- Fire and forget, no interaction
Solution:
Build a dispatcher!
GERRIT HOOKS: TRIgger
- Patch Created
- Comment Added
- Change Merged
- Change Abandoned
- Reference updated
- (more on newer versions)
GERRIT HOOKS: EVENT type
- By parameters passed
- We will be able to detect
- Maintainer approved
- Tests passed
- Maintainer blocked
- ...
GERRIT HOOKS: Labels
What?
Label: label content
Where?
Commit message
Comment
Example:
Bug-Url: http://mytracker/bugid
Rerun-Hooks: all
GERRIT HOOKS: CHAINS
- Using event name
- Chain as second term if wanted
- Hierarchical config files
- Main one at $gerrit_dir
- Per project
- Per event
Hook name template:
${event}.${chain}.${hook_name}
${event}.${hook_name}
${event}
GERRIT HOOKS: EXECUTION
INFO::STARTING::change-merged
DEBUG::get_hooks::change-merged on vdsm.git/hooks
INFO::::AVAILABLE HOOKS::[
'change-merged.set_MODIFIED',
'change-merged.update_tracker']
Each hook can:
- Break it's hook chain
- Break whole execution
Everything to logfile (configurable):
Common stdout format (each section is optional):
V:VERIFIED SCORE\n
R:REVIEW SCORE\n
COMMENT
GERRIT HOOKS: REVIEW
Sum of all the reviews:
- negative ones prevail over positives
- no vote means no change
- 0 means reset negative value
Execution summary of all the hooks:
GERRIT HOOKS: Source CODE
Patches are always welcome!
- Git:
- git clone git://gerrit.ovirt.org/gerrit-admin
- Web:
- http://gerrit.ovirt.org/gitweb?p=gerrit-admin.git
GERRIT HOOKS: LIBS
Some simple generic libs available for the hooks:
- Bugzilla
- Gerrit
- Configuration
- Tools
source bz.sh
source gerrit.sh
gerrit.parse_params "$@" # Sets the parameters into global vars
# get the bug ids in the commit message of the commit passed as
# parameter (--commit SHA)
bug_ids=($(bz.get_bug_id "$commit"))
from gerrit import Gerrit
from config import load_config
config = load_config() ## Loads the config hierarchicaly into a dict
gerrit = Gerrit(config.get('GERRIT_SRV')) ## Connect to the gerrit instance
GERRIT HOOKS: BUGZILLA
Update external tracker in bugzilla
Modify bug status when first patch sent
Modify bug status when all patches merged
Jenkins job builder
Problems with manual configuration
- Hard to maintain
- Hard to track
- Hard to review
- Hard to change common settings
...
Add an extra scm to jobs for projects A, B and C
New branch for project M, duplicate all it's jobs
Disable all the jobs for a project
Test a job configuration before deploying
Jenkins job builder
Jenkins job Builder to the rescue!
jenkins job builder
Some plugins that helped along the way:
- Configuration deduplication:
Matrix configurations
- Multijobs config changes:
Configuration slice
- Configuation change management:
Job Config history
- Manage jobs statuses:
Scriptler
Jenkins job builder
- Developed by OpenStack folks (Kudos!!)
- Configuration as code (yaml format)
- Easy to review changes (on Gerrit!)
- Code/config de-duplication
- Include external shell/groovy/... scripts
- Test before deploying
- Easier to organize (per directory, per file)
- Serves as backup (easily replicate on another jenkins)
Jenkins job builder
- This will create the jobs:
- projectA-whatever-fc19
- projectA-whatever-fc20
- job-group:
name: projectA
dist:
- fc19
- fc20
jobs:
- '{name}-wahtever-{dist}'
- job-template:
name: '{name}-whatever-{dist}'
node: 'slave-{dist}'
triggers:
- timed: '@midnight'
builders:
- shell: make whatever
publishers:
- archive:
artifacts: '*.log'
On slave-fc19 and slave-fc20
Executing 'make whatever'
Archiving the logs
Jenkins job builder
Oh! But I want to add a new distro, el6
- job-group:
name: projectA
dist:
- fc19
- fc20
- el6
jobs:
- '{name}-wahtever-{dist}'
And a cleanup after
- job-template:
name: '{name}-whatever-{dist}'
node: 'slave-{dist}'
triggers:
- timed: '@midnight'
builders:
- shell: make whatever
- shell: make clean
publishers:
- archive:
artifacts: '*.log'
Jenkins job builder
Official Openstack links:
Docs:
Code:
oVirt links
Current problems/future solutions
Scarce resources
- Moving to a new infrastructure
- OS1 (openstack) slaves
Jobs management and control
- Move all jobs to yaml
- Get developers to help write/maintain jobs
Leverage manual procedures
- Automate the official release + signing pkgs
- Automate gerit config/project creation
Current problems/future solutions
Isolate test environments
- Docker/mock on all simple builds
- One slave per job on complex builds
- Provisining of slaves on demand
Leverage package builds
- Build on Copr (has jenkins plugin!)
Copy of jenkins_conference_2014_david
By Kiril Nesenko
Copy of jenkins_conference_2014_david
- 1,565