OpenDaylight Performance

Daniel Farrell, Red Hat

Objectives

Part 0: Syncing up

Describe WCBench
Sync up on the rest of the tools I know about

OBJECTIVES

PART 0: Roadmap of TODOs

Propose perf framework
Discussions

TOOL WISHLIST

Open Source, of course

Tool Wishlist

Cover many code paths

TOOL WISHLIST

Documentation

(Bonus if it covers *why* vs just what/how)

TOOL WISHLIST

Readable code

(Thought-out var names, elegant->simple design, formatting)

TOOL WISHLIST

Appropriate language choice

TOOL WISHLIST

Unit tests

(I wish it could go without saying)

TOOL WISHLIST

Community-enabling distribution

(GitHub, not tarballs, emailed scripts or VMs)

TOOL WISHLIST

Support for multiple controllers

Existing Tools

Quick Dive into W/CBench

CBENCH OVERVIEW

Controller benchmark tool, written by Rob Sherwood
Very simple:

Algorithm:
    pretend to be n switches (n=16 is default)
    create n openflow sessions to the controller
    if latency mode (default):
        for each session:
            1) send up a packet in
            2) wait for a matching flow mod to come back
            3) repeat
            4) count how many times #1-3 happen per sec
<snip>

CBENCH ISSUES

Not actively maintained

CBENCH ISSUES

Poor documentation

# Total number of lines in primary file
[~/oflops/cbench]$ cat cbench.c | wc -l
426
# Total number of comments
[~/oflops/cbench]$ cat cbench.c | grep '\/\*[^\*]\|\/\/' | wc -l
13

CBench Issues

No unit tests

CBENCH ISSUES

Dumps (unstructured) results to stdout

        printf("RESULT: %d switches %d tests "
            "min/max/avg/stdev = %.2lf/%.2lf/%.2lf/%.2lf responses/s\n",
                i+1,
                counted_tests,
                min, max, avg, std_dev);

CBench Issues

Written in C

Much code could be replaced by Python stdlib calls
Not readable, harder to maintain

CBENCH ISSUES

Single-threaded

(Bigger) CBench Issues

Limited code path

Not realistic

Wrapped CBench (WCBEnch)

CBench, wrapped in stuff that makes it useful

WRAPPED CBENCH (WCBENCH)

Three main scripts

WRAPPED CBENCH (WCBENCH)

wcbench.sh

# Install CBench and OpenDaylight, make required config changes
[fedora@dfarrell-wcbench wcbench]$ ./wcbench.sh -ci
CBench is not installed
Installing CBench dependencies
Cloning CBench repo
Cloning openflow source code
Building oflops/configure file
Building CBench
CBench is installed
Successfully installed CBench
Installing OpenDaylight dependencies
Downloading last successful ODL build
Unzipping last successful ODL build
Downloading openflowplugin
Removing simpleforwarding plugin
Removing arphandler plugin

WRAPPED CBENCH (WCBENCH)

wcbench.sh

# Start OpenDaylight, issue required config
[fedora@dfarrell-wcbench wcbench]$ ./wcbench.sh -o
Starting OpenDaylight
Giving ODL 90 seconds to get up and running
80 seconds remaining
<snip>
0 seconds remaining
Installing telnet, as it's required for issuing ODL config.
Issuing `dropAllPacketsRpc on` command via telnet to localhost:2400
Trying ::1...
Connected to localhost.
Escape character is '^]'.
osgi> dropAllPacketsRpc on
DropAllFlows transitions to on
osgi> Connection closed by foreign host.

WRAPPED CBENCH (WCBENCH)

wcbench.sh

# Two minute CBench run
[fedora@dfarrell-wcbench wcbench]$ ./wcbench.sh -t 2 -r
Set MS_PER_TEST to 120000, TESTS_PER_SWITCH to 1, CBENCH_WARMUP to 0
Collecting pre-test stats
Running CBench against ODL on localhost:6633
Collecting post-test stats
Collecting time-irrelevant stats
Average responses/second: 22384.52
/home/fedora/results.csv not found or empty, building fresh one

WRAPPED CBENCH (WCBENCH)

wcbench.sh

ODL+WCBench on same machine or different ones

WRAPPED CBENCH (WCBENCH)

wcbench.sh

Usage ./wcbench.sh [options]

Setup and/or run CBench and/or OpenDaylight.

OPTIONS:
    -h Show this message
    -c Install CBench
    -t <time> Run CBench for given number of minutes
    -r Run CBench against OpenDaylight
    -i Install ODL from last successful build
    -p <processors> Pin ODL to given number of processors
    -o Run ODL from last successful build
    -k Kill OpenDaylight
    -d Delete local ODL and CBench code

WRAPPED CBENCH (WCBENCH)

loop_wcbench.sh

# Two minute CBench run, restart OpenDaylight between runs
[fedora@dfarrell-wcbench wcbench]$ ./loop_wcbench.sh -t 2 -r
# I'm not going to let this run, you get the idea

Wrapped CBench (WCBench)

Usage ./loop_wcbench.sh [options]

Run WCBench against OpenDaylight in a loop.

OPTIONS:
    -h Show this help message
    -l Loop WCBench runs without restarting ODL
    -r Loop WCBench runs, restart ODL between runs
    -t <time> Run WCBench for a given number of minutes
    -p <processors> Pin ODL to given number of processors

WRAPPED CBENCH (WCBENCH)

stats.py

# CBench flows/sec stats
./stats.py -s flows
{'flows': {'max': 15036,
           'mean': 8958.426,
           'min': 4981,
           'relstddev': 32.032,
           'stddev': 2869.584},
 'sample_size': 115}

WRAPPED CBENCH (WCBENCH)

stats.py

Flows/sec, RAM, iowait, steal time, load

(many others, 22 total)

WRAPPED CBENCH (WCBENCH)

stats.py

# Command for graphs of flows/sec and used RAM stats
./stats.py -g flows ram

WRAPPED CBENCH (WCBENCH)

WRAPPED CBENCH (WCBENCH)

Solid README

WRAPPED CBENCH (WCBENCH)

Good code style

TLDP/Google Shell Guide

[~/wcbench]$ ./bash8.py wcbench.sh loop_wcbench.sh
[~/wcbench]$ pep8 stats.py

WRAPPED CBENCH (WCBENCH)

Solid in-line docs

# Total number of lines
[~/wcbench]$ cat wcbench.sh | wc -l
749
# Total number of lines that contain comments
[~/wcbench]$ cat wcbench.sh | grep "^# \| # " | wc -l
280

Tool Ecosystem Syncup

KC-CBENCH

kC-CBench

OpenFlow 1.3 improvements added to CBench

KC-CBench

Less-than-ideal fork :(

OFNET

OFNet

Feature-rich controller testing tool by Ganesh Shankar

OFNet

Easy to model topologies

OFNet

Easy to simulate various types of traffic

OFNet

Real-time stats graphs

OFNET

Distributed dev-to-dev via VM/tarball :(

The only link to OFNet on the Internet

OFNet

"Throw code over wall" approach :(

OFNet

Poor docs

^^As you can see, those don't exist

OFNet

Ganesh is working on the docs, he tells me

OFNet

No tests

^^Tiny file that's not actually a test

OFNet

Coded in C :(

OFNET

Enough features to pay attention?

TestON

TestOn

Supports various SDN devices (yay!)

TestON

Coded like C (but it's Python)

JAN'S SCRIPTS

Jan's Scripts

Load up controller with flows via REST
Delete flows via REST

JAN'S SCRIPTS

There's an updated version that I haven't seen

JAN'S SCRIPTS

Please, let's put them in a repo

NETCONF SCRIPTS

NETCONF scripts

Just learned these exist

NETCONF SCRIPTS

Let's put them in a repo

DATASTORE SCRIPTS

Datastore Scripts

Just heard about them

DATASTORE SCRIPTS

More info? Are they shared anywhere?

IXIA

Phil Rob tells me they are working on support

IXIA

Anyone know anything else about this?

SPIRENT

Maybe support OF switch simulation?

No docs, but informal assurance to LuisGomez

SPIRENT

Anyone know anything else about this?

Others?

Do you know of any other perf tools for controllers?

(contact slide soon)

Proposed Framework

PROPOSED FRAMEWORK

Properly architect controller perf testing landscape

PROPOSED FRAMEWORK

Enable code reuse

PROPOSED FRAMEWORK

Clarify test options for community

PROPOSED FRAMEWORK

Start with focus on ODL, iterate to generalize

PROPOSED FRAMEWORK

As it generalizes, bring in devs from other controller projects

PROPOSED FRAMEWORK

Start with IETF-recommended tests

PROPOSED FRAMEWORK

Honor "Better Tools Wishlist"

PROPOSED FRAMEWORK

Python for fast development and readability

(build elegantly first, optimize critical path if needed)

PROPOSED FRAMEWORK

Solid docs: Sphinx-style docstrings, guides

PROPOSED FRAMEWORK

All the obvious things

(tests, GitHub, Open Source, PEP8)

PROPOSED FRAMEWORK

In research phase

Input solicited!

Discussion!

What use cases do we want to perf test?
Projects/tools we should know about for framework?
There's more here, but I ran out of time. @CASP3R?

Contact

Daniel Farrell

SDN Intern, Red Hat

Let's have a beer/coffee and talk about SDN/ODL/Linux/etc!

919.576.0112

dfarrell07 [IRC, Gmail, GitHub, Twitter]

dfarrell.me

OpenDaylight Performance for ODL Dev Forum

By Daniel Farrell

OpenDaylight Performance for ODL Dev Forum

Unconf deck for OpenDaylight Developer Design Forum, 9/2014

4,338

Daniel Farrell

Software Engineer, Red Hat SDN Team

dfarrell07