OpenDaylight Performance

Daniel Farrell, Red Hat

Objectives

Describe WCBench tool
Share ODL perf test results
Describe state of tools for perf testing controllers
Start conversation about where we want to go tool-wise

VERY QUICK: INTRO TO ME

Daniel Farrell, SDN Intern Red Hat

Very Quick: Intro to ME

Code!

Worked on OpenFlow, OpenStack and OpenDaylight

VERY QUICK: INTRO TO ME

Robots!

VERY QUICK: INTRO TO ME

(Craft) Beer!

VERY QUICK: INTRO TO ME

Biking!

VERY QUICK: INTRO TO ME

SCUBA!

CBENCH OVERVIEW

Controller benchmark tool, written by Rob Sherwood
Very simple:

Algorithm:
    pretend to be n switches (n=16 is default)
    create n openflow sessions to the controller
    if latency mode (default):
        for each session:
            1) send up a packet in
            2) wait for a matching flow mod to come back
            3) repeat
            4) count how many times #1-3 happen per sec
<snip>

CBENCH ISSUES

Not actively maintained

CBENCH ISSUES

Poor documentation

# Total number of lines in primary file
[~/oflops/cbench]$ cat cbench.c | wc -l
426
# Total number of comments
[~/oflops/cbench]$ cat cbench.c | grep '\/\*[^\*]\|\/\/' | wc -l
13

CBench Issues

No unit tests

CBENCH ISSUES

Dumps (unstructured) results to stdout

        printf("RESULT: %d switches %d tests "
            "min/max/avg/stdev = %.2lf/%.2lf/%.2lf/%.2lf responses/s\n",
                i+1,
                counted_tests,
                min, max, avg, std_dev);

CBench Issues

Written in C

Much code could be replaced by Python stdlib calls
Not readable, harder to maintain

CBENCH ISSUES

Single-threaded

(Bigger) CBench Issues

Limited code path

Not realistic

Wrapped CBench (WCBEnch)

CBench, wrapped in stuff that makes it useful

WRAPPED CBENCH (WCBENCH)

Three main scripts

WRAPPED CBENCH (WCBENCH)

wcbench.sh

# Install CBench and OpenDaylight, make required config changes
[fedora@dfarrell-wcbench wcbench]$ ./wcbench.sh -ci
CBench is not installed
Installing CBench dependencies
Cloning CBench repo
Cloning openflow source code
Building oflops/configure file
Building CBench
CBench is installed
Successfully installed CBench
Installing OpenDaylight dependencies
Downloading last successful ODL build
Unzipping last successful ODL build
Downloading openflowplugin
Removing simpleforwarding plugin
Removing arphandler plugin

WRAPPED CBENCH (WCBENCH)

wcbench.sh

# Start OpenDaylight, issue required config
[fedora@dfarrell-wcbench wcbench]$ ./wcbench.sh -o
Starting OpenDaylight
Giving ODL 90 seconds to get up and running
80 seconds remaining
<snip>
0 seconds remaining
Installing telnet, as it's required for issuing ODL config.
Issuing `dropAllPacketsRpc on` command via telnet to localhost:2400
Trying ::1...
Connected to localhost.
Escape character is '^]'.
osgi> dropAllPacketsRpc on
DropAllFlows transitions to on
osgi> Connection closed by foreign host.

WRAPPED CBENCH (WCBENCH)

wcbench.sh

# Two minute CBench run
[fedora@dfarrell-wcbench wcbench]$ ./wcbench.sh -t 2 -r
Set MS_PER_TEST to 120000, TESTS_PER_SWITCH to 1, CBENCH_WARMUP to 0
Collecting pre-test stats
Running CBench against ODL on localhost:6633
Collecting post-test stats
Collecting time-irrelevant stats
Average responses/second: 22384.52
/home/fedora/results.csv not found or empty, building fresh one

WRAPPED CBENCH (WCBENCH)

wcbench.sh

ODL+WCBench on same machine or different ones

WRAPPED CBENCH (WCBENCH)

wcbench.sh

Usage ./wcbench.sh [options]

Setup and/or run CBench and/or OpenDaylight.

OPTIONS:
    -h Show this message
    -c Install CBench
    -t <time> Run CBench for given number of minutes
    -r Run CBench against OpenDaylight
    -i Install ODL from last successful build
    -p <processors> Pin ODL to given number of processors
    -o Run ODL from last successful build
    -k Kill OpenDaylight
    -d Delete local ODL and CBench code

WRAPPED CBENCH (WCBENCH)

loop_wcbench.sh

# Two minute CBench run, restart OpenDaylight between runs
[fedora@dfarrell-wcbench wcbench]$ ./loop_wcbench.sh -t 2 -r
# I'm not going to let this run, you get the idea

Wrapped CBench (WCBench)

Usage ./loop_wcbench.sh [options]

Run WCBench against OpenDaylight in a loop.

OPTIONS:
    -h Show this help message
    -l Loop WCBench runs without restarting ODL
    -r Loop WCBench runs, restart ODL between runs
    -t <time> Run WCBench for a given number of minutes
    -p <processors> Pin ODL to given number of processors

WRAPPED CBENCH (WCBENCH)

stats.py

# CBench flows/sec stats
./stats.py -s flows
{'flows': {'max': 15036,
           'mean': 8958.426,
           'min': 4981,
           'relstddev': 32.032,
           'stddev': 2869.584},
 'sample_size': 115}

WRAPPED CBENCH (WCBENCH)

stats.py

Flows/sec, RAM, iowait, steal time, load

(many others, 22 total)

WRAPPED CBENCH (WCBENCH)

stats.py

# Command for graphs of flows/sec and used RAM stats
./stats.py -g flows ram

WRAPPED CBENCH (WCBENCH)

WRAPPED CBENCH (WCBENCH)

Awesome README

WRAPPED CBENCH (WCBENCH)

Good code style

TLDP/Google Shell Guide

[~/wcbench]$ ./bash8.py wcbench.sh loop_wcbench.sh
[~/wcbench]$ pep8 stats.py

WRAPPED CBENCH (WCBENCH)

Awesome in-line docs

# Total number of lines
[~/wcbench]$ cat wcbench.sh | wc -l
749
# Total number of lines that contain comments
[~/wcbench]$ cat wcbench.sh | grep "^# \| # " | wc -l
280

Test Results

Good Test Results

NUM_SWITCHES=16

./loop_cbench.sh -t 2 -r

{'cbench_relative_std_dev': 4.918,
 'cbench_standard_deviation': 269.428}

CBench Flail

NUM_SWITCHES=256

./loop_cbench.sh -p 2 -t 2 -r

{'cbench_relative_std_dev': 13.723,
 'cbench_standard_deviation': 532.303}

CBench Flail

Dropping connections?

Handshake issue?

Needs packet-level analysis

DEcreasing PERFORMANCE

./loop_cbench.sh -l

Performance DEGRADATIOn

Bug 1395

Targeted for fix in Helium

Increasing RAM Usage

RAM Usage

Bug 1591

Jenkins Integration

Future Work

Doing: Fix perf issues I described
TODO: Build/find better tools (coverage!)

Better Tools

Future work

BETTER TOOL WISHLIST

Open Source, of course

Better Tool Wishlist

Cover more code paths

BETTER TOOL WISHLIST

Documentation

(Bonus if it covers *why* vs just what/how)

BETTER TOOL WISHLIST

Readable code

(Thought-out var names, elegant->simple design, formatting)

BETTER TOOL WISHLIST

Appropriate language choice

BETTER TOOL WISHLIST

Unit tests

(I wish it could go without saying)

BETTER TOOL WISHLIST

Community-enabling distribution

(GitHub, not tarballs, emailed scripts or VMs)

BETTER TOOL WISHLIST

Support for multiple controllers

Existing Tools

KC-CBENCH

kC-CBench

OpenFlow 1.3 improvements added to CBench

KC-CBench

Less-than-ideal fork :(

JAN'S SCRIPTS

Jan's Scripts

Load up controller with flows via REST
Delete flows via REST

JAN'S SCRIPTS

Very readable code

JAN'S SCRIPTS

Distributed dev-to-dev via email :(

JAN'S SCRIPTS

No tests

OFNET

OFNet

Feature-rich controller testing tool by Ganesh Shankar

OFNet

Easy to model topologies

OFNet

Easy to simulate various types of traffic

OFNet

Real-time stats graphs

OFNET

Distributed dev-to-dev via VM/tarball :(

The only link to OFNet on the Internet

OFNet

"Throw code over wall" approach :(

OFNet

Poor docs

^^As you can see, those don't exist

OFNet

Ganesh is working on the docs, he tells me

OFNet

No tests

^^Tiny file that's not actually a test

OFNet

Coded in C :(

TestON

TestOn

Supports various SDN devices (yay!)

TestON

Coded like C (but it's Python)

IXIA

Closed source :(

SPIRENT

Closed source :(

SPIRENT

Maybe support OF switch simulation?

No docs, but informal assurance to LuisGomez

Others?

Do you know of any other perf tools for controllers?

(contact slide soon)

Proposed New Tool

PROPOSED NEW TOOL

Bring sanity to the controller perf testing landscape

PROPOSED NEW TOOL

Start with focus on ODL, iterate to generalize

PROPOSED NEW TOOL

As it generalizes, bring in devs from other controller projects

PROPOSED NEW TOOL

Start with IETF-recommended tests

PROPOSED NEW TOOL

Iterate to generalize test framework

PROPOSED NEW TOOL

Honor "Better Tools Wishlist"

PROPOSED NEW TOOL

Python for fast development and readability

(build elegantly first, optimize critical path if needed)

PROPOSED NEW TOOL

Solid docs, especially Sphinx-style docstrings

PROPOSED NEW TOOL

All the obvious things

(tests, GitHub, Open Source, PEP8)

PROPOSED NEW TOOL

In research phase

Input solicited!

Contact

Daniel Farrell

SDN Intern, Red Hat

Let's have a beer/coffee and talk about SDN/ODL/Linux/etc!

919.576.0112

^^I expect a pile of texts before I sit down

dfarrell07 [IRC, Gmail, GitHub]

dfarrell.me

OpenDaylight Performance

By Daniel Farrell

OpenDaylight Performance

Presentation for LinuxCon+CloudOpen 2014, OpenDaylight Mini Summit.

3,893

Daniel Farrell

Software Engineer, Red Hat SDN Team

dfarrell07