Integrating Upstream Projects into OPNFV

Daniel Farrell
Red Hat SDN Team
@dfarrell07

VERY QUICK: INTRO TO ME

Daniel Farrell
Software Engineer, Red Hat SDN Team

VERY QUICK: INTRO TO ME

PTL:
  • OpenDaylight Integration/Packaging
  • OPNFV CPerf
 

VERY QUICK: INTRO TO ME

Committer:
  • OpenDaylight Integration/Test
  • CentOS NFV SIG

VERY QUICK: INTRO TO ME

OpenDaylight TSC

VERY QUICK: INTRO TO ME

tl;dr - OpenDaylight side of ODL+OPNFV integration

Overview

OVERVIEW

What is CD?

OVERVIEW

New requirements imposed by OPNFV on upstream projects

OVERVIEW

How OpenDaylight integrates into OPNFV
(guide for other projects)

CD?

CD?

"Continuous Delivery" or "Continuous Deployment"?

CD?

"Delivery": Delivering artifact for consumption

CD?

"Deployment": Deploy a delivered artifact

CD?

Upstream projects Deliver

CD?

Consumers (like OPNFV) Deploy

CD?

"Continuous": Project-specific
Per-[patch, commit, build-able artifact, day, semver bump...]

OPNFV: CREATING NEW REQUIREMENTS

OPNFV: CREATING NEW REQUIREMENTS

Upstream -> Installers -> Test Projects -> Upstream

OPNFV: CREATING NEW REQUIREMENTS

Upstream -> Installers -> Test Projects -> Upstream
^^Frequency of this cycle is important!

OPNFV: CREATING NEW REQUIREMENTS

Upstream -> Installers
1. Upstream projects must release frequently

OPNFV: CREATING NEW REQUIREMENTS

Upstream -> Installers
2. Upstream must provide install/config tools

New Req: Frequency

NEW REQ: FREQUENCY

1. Upstream projects need to release frequently
^^How frequently?

NEW REQ: FREQUENCY

OPNFV CI is limited by rate of integrated project releases

NEW REQ: FREQUENCY

OPNFV can't test new artifacts faster than they are released

NEW REQ: FREQUENCY

Slow releases -> re-testing same artifacts

NEW REQ: FREQUENCY

Quick releases -> Quick feedback about breakages

NEW REQ: FREQUENCY

New Theorem:
As OPNFV approaches CI, consumed project approach CD

NEW REQ: FREQUENCY

Affects all projects consumed by OPNFV
(ODL, ONOS, OpenContrail, OVS, DPDK...)

New Req: ARTIFACTS

NEW REQ: ARTIFACTS

Modern deployments require complex release artifacts

NEW REQ: ARTIFACTS

Tarballs and hope are not enough

NEW REQ: ARTIFACTS

Upstream must provide well-defined knobs for OPNFV's config

NEW REQ: ARTIFACTS

OPNFV minimum: Install via config management tool
(Puppet, Ansible)

NEW REQ: ARTIFACTS

OPNFV installers use ~Puppet for automated installs

NEW REQ: ARTIFACTS

Each integrated project needs to provide a config tool

NEW REQ: ARTIFACTS

Config tools typically build on packages
(RPM, .deb)

NEW REQ: ARTIFACTS

Config mgmt tools can be used to build pre-configured images
(Vagrant base boxes, Docker images)

NEW REQ: ARTIFACTS

Affects all projects consumed by OPNFV
(ODL, ONOS, OpenContrail, OVS, DPDK...)

ODL's CD Pipeline

ODL'S CD PIPELINE

ThreeFour-layer stack

ODL'S CD PIPELINE

Layer 0: Source builds

ODL'S CD PIPELINE

Offical releases, daily autoreleases

ODL'S CD PIPELINE

Layer 1: Packaging layer

ODL'S CD PIPELINE

RPM, .deb

ODL'S CD PIPELINE

Layer 2: Configuration Management Layer

ODL'S CD PIPELINE

Puppet module, Ansible role, Juju charms

ODL'S CD PIPELINE

Layer 3: Pre-Configured Images Layer

ODL'S CD PIPELINE

Containers, VMs
(Docker, Vagrant base boxes)

Source Builds

SOURCE BUILDS

Offical releases

SOURCE BUILDS

nexus.opendaylight.org/.../opendaylight.release/...

SOURCE BUILDS

Daily autorelease builds

SOURCE BUILDS

nexus.opendaylight.org/.../staging/...

PACKAGE LAYER

PACKAGE LAYER

Handles system-level dependencies
(Java...)

PACKAGE LAYER

Lays down the basic files

PACKAGE LAYER

Creates required users/groups

PACKAGE LAYER

Sets permissions

PACKAGE LAYER

Configures systemd

PACKAGE LAYER

Current: RPMs

PACKAGE LAYER

Under-dev: DEBs

RPM

RPM

CentOS Community Build System

RPM

Example

RPM

[opendaylight-40-release]
name=CentOS CBS OpenDaylight Beryllium repository
baseurl=http://cbs.centos.org/repos/nfv7-opendaylight-40-release/$basearch/os/
enabled=1
gpgcheck=0

RPM

Install ODL's repo
[~]$ sudo curl -o /etc/yum.repos.d/opendaylight-40-release.repo \
                  <URL to repo config>

RPM

Install ODL
[~]$ sudo yum install -y opendaylight
<snip>
Installed:
  opendaylight.noarch 0:4.0.0-1.el7
Complete!

RPM

Systemd integration
[~]$ sudo systemctl start opendaylight

RPM

Systemd integration
[~]$ sudo systemctl is-active opendaylight
active

RPM

RPM

[/opt/opendaylight/bin]$ ./client
<snip>
opendaylight-user@root>feature:install ...

RPM

#endexample

Deb

Deb

LF intern building now

CONFIG MANAGEMENT LAYER

CONFIG MANAGEMENT LAYER

Installs ODL via package layer

CONFIG MANAGEMENT LAYER

Edits files to apply configuration
(Karaf features, port mappings, L3, logging, HA...)

CONFIG MANAGEMENT LAYER

Opens ports in FirewalD

CONFIG MANAGEMENT LAYER

Manages ODL's systemd service

PUPPET

Puppet

Puppet Forge

Puppet

Well-developed API

Puppet

We work to make contributing easy.

Puppet

OS Support

Puppet

Community: Ubuntu support

Puppet

Supports tarball-based installs

Puppet

class { 'opendaylight':
  install_method => 'tarball',
  tarball_url    => '<URL to your custom tarball>',
  unitfile_url   => '<URL to your custom unitfile>',
}

Puppet

Future: Tarballs -> .deb packages

Puppet

Well tested

PUPPET

rspec-puppet
(100% coverage, not worth discussion)

Puppet

Beaker

Puppet

Deploy against real VMs

Puppet

Verify resulting state

Puppet

Repeat for various configs

Puppet

At a high level...

Puppet

In detail...

Puppet

Bringing machine 'centos-7' up with 'virtualbox' provider...

centos-7 12:45:09$ yum install -y puppet

Puppet

centos-7 12:45:46$ puppet apply <test manifest>

Puppet

centos-7 12:49:37$ /bin/sh -c systemctl\ is-active\ opendaylight
active

centos-7 executed in 0.04 seconds
      should be running

Puppet

centos-7 12:49:37$ /bin/sh -c id\ odl
uid=1001(odl) gid=1001(odl) groups=1001(odl)

centos-7 executed in 0.04 seconds
      should exist

Puppet

centos-7 executed in 0.05 seconds
            should match /^featuresBoot=standard,ssh/

Puppet

Finished in 5 minutes 22 seconds (files took 1 minute 10.87 seconds to load)
132 examples, 0 failures

Puppet

Beaker is pretty great

Puppet

Example: Vagrant provisioner

Puppet

At a high level...

Puppet

  # Box that installs ODL via Puppet RPM method on CentOS 7
  config.vm.define "cent7_pup_rpm" do |cent7_pup_rpm|
    # Build Vagrant box based on CentOS 7
    cent7_pup_rpm.vm.box = "centos/7"

    # Add EPEL repo for access to Puppet et al
    cent7_pup_rpm.vm.provision "shell", inline: "yum install -y epel-release"

    # Install Puppet
    cent7_pup_rpm.vm.provision "shell", inline: "yum install -y puppet"

    # Install OpenDaylight using its Puppet module
    cent7_pup_rpm.vm.provision "puppet" do |puppet|
      puppet.module_path = ["modules"]
      puppet.manifest_file = "odl_install.pp"
    end
  end

Puppet

In detail...

Puppet

      puppet.manifest_file = "odl_install.pp"

Puppet

[~/vagrant-opendaylight]$ cat manifests/odl_install.pp
class { 'opendaylight':
  extra_features => ['odl-ovsdb-sfc'],
}

Puppet

Provision the box

Puppet

[~/vagrant-opendaylight]$ vagrant up cent7_pup_rpm

Puppet

ODL installed
Notice: /Stage[main]/Opendaylight::Install/Yumrepo[opendaylight]/ensure: created
Notice: /Stage[main]/Opendaylight::Install/Package[opendaylight]/ensure: created

Puppet

Features configured
/Stage[main]/Opendaylight::Config/File[org.apache.karaf.features.cfg]/content: content changed '<hash>' to '<different hash>'

Puppet

Explore

Puppet

[~/vagrant-opendaylight]$ vagrant ssh cent7_pup_rpm
[vagrant@localhost ~]$

Puppet

[vagrant@localhost ~]$ sudo systemctl is-active opendaylight
active

Puppet

[vagrant@localhost ~]$ ssh -p 8101 karaf@localhost
# password "karaf"

Puppet


Puppet

opendaylight-user@root>config:list | grep featuresBoot\ =
   featuresBoot = <default features>,odl-ovsdb-sfc

Puppet

opendaylight-user@root>feature:list | grep odl-ovsdb-sfc
odl-ovsdb-sfc | 1.2.1-Beryllium | x | odl-ovsdb-sfc-1.2.1-Beryllium <snip>

Puppet

#endexample

ANSIBLE

Ansible

Ansible Galaxy

Ansible

Example: Vagrant provisioner

Ansible

At a high level...

Ansible

  # Box that installs ODL via its Ansible role on CentOS 7
  config.vm.define "cent7_ansible" do |cent7_ansible|
    # Build Vagrant box based on CentOS 7
    cent7_ansible.vm.box = "centos/7"

    # Install ODL using the Ansible provisioner
    cent7_ansible.vm.provision "ansible" do |ansible|
      # Path to Ansible playbook that installs ODL using ODL's Ansible role
      ansible.playbook = "provisioning/playbook.yml"
    end
  end

Ansible

In detail...

Ansible

      ansible.playbook = "provisioning/playbook.yml"

Ansible

[~/vagrant-opendaylight]$ cat provisioning/playbook.yml
---
- hosts: all
  sudo: yes
  roles:
    - opendaylight

ansible

Provision the box

Ansible

[~/vagrant-opendaylight]$ vagrant up cent7_ansible

ansible

At a high level...

Ansible

In detail...

Ansible

TASK: [opendaylight | Add ODL yum repo] ********************************** 
changed: [cent7_ansible]
TASK: [opendaylight | Install ODL via RPM repo] **************************
changed: [cent7_ansible]

Ansible

TASK: [opendaylight | Check if FirewallD service is running] ************* 
ok: [cent7_ansible]
TASK: [opendaylight | Open ODL NB REST port via FirewallD] *************** 
skipping: [cent7_ansible]

Ansible

TASK: [opendaylight | Start ODL systemd service] ************************* 
changed: [cent7_ansible]

Ansible

Explore

Ansible

[~/vagrant-opendaylight]$ vagrant ssh cent7_ansible
[vagrant@localhost ~]$

Ansible

[vagrant@localhost ~]$ sudo systemctl is-active opendaylight
active

Ansible

Example: Config change

Ansible

[~/vagrant-opendaylight]$ cat provisioning/playbook.yml
---
- hosts: all
  sudo: yes
  roles:
    - role: opendaylight
      extra_features: ['odl-ovsdb-sfc']

Ansible

[~/vagrant-opendaylight]$ vagrant provision cent7_ansible

ansible

At a high level...

Ansible

In detail...

Ansible

TASK: [opendaylight | Configure ODL Karaf features] ********************** 
changed: [cent7_ansible]

Ansible

NOTIFIED: [opendaylight | Stop ODL] ************************************** 
changed: [cent7_ansible]

NOTIFIED: [opendaylight | Cleanup Karaf] ********************************* 
changed: [cent7_ansible]

NOTIFIED: [opendaylight | Start ODL] ************************************* 
changed: [cent7_ansible]

Ansible

Explore

Ansible

[~/vagrant-opendaylight]$ vagrant ssh cent7_ansible

Ansible

[vagrant@localhost ~]$ sudo systemctl is-active opendaylight
active

Ansible

[vagrant@localhost ~]$ /vagrant/scripts/connect.sh


Ansible


Ansible

opendaylight-user@root>feature:list | grep odl-ovsdb-sfc
odl-ovsdb-sfc | 1.2.1-Beryllium | x | odl-ovsdb-sfc-1.2.1-Beryllium <snip>

Juju

JUJU

Exist, but not pulling recent ODL builds

Juju

Use upstream .deb once we have it

PRE-CONFIGURED IMAGES

PRE-CONFIGURED IMAGES

Environment with ODL installed and configured

PRE-CONFIGURED IMAGES

Built using config management layer

PRE-CONFIGURED IMAGES

VM images
(Vagrant base boxes)

PRE-CONFIGURED IMAGES

Containers
(Docker)

PRE-CONFIGURED IMAGES

Built with Packer

PACKER

Packer

Creates ODL's Vagrant base box + Docker image

Packer

Everything's in VCS

Packer

Common Ansible path
(powerful config, free updates)

CONTAINERS

Containers

Example: One-liner Karaf shell

Containers

$ docker run -ti dfarrell07/odl:4.0.0 /opt/opendaylight/bin/karaf

Containers

$ docker ps
CONTACONTAINER ID   IMAGE                  COMMAND
b2869594fab3        dfarrell07/odl:4.0.0   "/opt/opendaylight/bin/karaf"

Containers

#endexamples

VAGRANT BASE BOX

Vagrant Base Box

Best user experience?

Vagrant Base Box

Example

Vagrant Base Box

[~/sandbox]$ vagrant init -m opendaylight/odl
[~/sandbox]$ cat Vagrantfile
Vagrant.configure(2) do |config|
  config.vm.box = "opendaylight/odl"
end

Vagrant Base Box

[~/sandbox]$ vagrant up

Vagrant Base Box

    default: URL: https://atlas.hashicorp.com/opendaylight/odl
==> default: Adding box 'opendaylight/odl' (v4.0.0)

Vagrant Base Box

[~/sandbox]$ vagrant status
Current machine states:

default                   running (libvirt)

Vagrant Base Box

[~/sandbox]$ vagrant ssh

Vagrant Base Box

[vagrant@localhost ~]$ sudo systemctl is-active opendaylight
active

Vagrant Base Box

#endexample

Challenges

CHALLENGES

Challenge: From-source RPM builds

CHALLENGES

Problem: *Really* hard to package Java projects from source

CHALLENGES

Blocks: Fuel wants some stable JARs, latest others

CHALLENGES

Needs: Dedicated, expert resources

CHALLENGES

Challenge: Fully automated RPM builds

CHALLENGES

Problem: Push to CBS requires GPG auth

CHALLENGES

Blocks: Automated daily RPMs

CHALLENGES

Needs: Tooling upgrade (dlrn); RPM repo on ODL Nexus

CONCLUSION

CONCLUSION

Rate of OPNFV's CI is limited by consumed project's CD

CONCLUSION

Upstreams must move towards CD

CONCLUSION

OPNFV can't own all install/config logic

CONCLUSION

Upstream projects must provide config mgmt tools for OPNFV

CONCLUSION

ODL is pretty good example to follow

Contact

Daniel Farrell
Twitter (optimal):

Others:
dfarrell@redhat.com
dfarrell07 on IRC, GitHub

Integrating Upstream Projects into OPNFV

By Daniel Farrell

Made with Slides.com

Integrating Upstream Projects into OPNFV

Talk given at OPNFV Summit 2016

  • 1,829

More from Daniel Farrell