Introducing

Epoxy

A flexible test framework

Red Hat QA meet-up
Camping Baldovec, Czech Republic
June 2013

Nikolai Kondrashov | spbnick

spbnick@gmail.com

Nikolai Kondrashov

spbnick

Sr. QA Engineer at Red Hat IDM QE.

Formerly Sr. System Software Engineer
at NVIDIA Tegra graphics team.

Open Source developer.

Founder of DIGImend project - improving
Linux support for generic graphics tablets.

EPOXY

Bash test framework

An alternative to BeakerLib

3 KLOC in Bash, Lua and Python

Nearing first release, already useful

Personal spare-time project

a definition

A suite, a test phase, a test
— all verify an assertion

"Assertion" is often used
in place of "assertion verification"

Thus we will use "assertion" to mean
any of "suite", "test phase", or "test"

Why not BeakerLib?

Limits assertion structuring

Produces noisy output

Can't verify a subset of assertions

Ignores setup/teardown failures

Limited structuring

Three levels only:
"suite", "phase", "assertion"

Either have huge flat suites
or integrate by external means

Either way hard to read and maintain

Three is not enough

Suite

#!/bin/bash

# Phase
rlPhaseStartTest "defaults"
# Assertion
rlRun client_sudo_user_requires_auth 0 "defaults_without"
# Assertion
rlRun client_sudo_user_is_allowed 0 "defaults_with"
rlPhaseEnd

# Phase
rlPhaseStartTest "config"
# Assertion
rlRun client_sudo_user_is_allowed 0 "config_all_options"
rlPhaseEnd

How about something more complex?

Noisy output

Mixes test results with
script output while running

Uses heavy decoration around
result messages as a workaround

This makes it harder to track progress
and spot failures and their causes

Spot the failure

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: simple_access_001: where simple_allow_groups = simple group1
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

spawn ssh -o StrictHostKeyChecking=no -l simple userB localhost
simple userB@localhost's password:
Last login: Thu Jun 27 10:19:34 2013 from localhost.localdomain
[simple userB@client-rhel6 ~]$ :: [   PASS   ] :: Authentication successfull, as expected
spawn ssh -o StrictHostKeyChecking=no -l simple userE localhost
simple userE@localhost's password:
Permission denied, please try again.
simple userE@localhost's password: :: [   FAIL   ] :: ERROR: Authentiation failed, not as expected
Jun 27 10:38:16 client-rhel6 sshd[484]: pam_sss(sshd:account): Access denied for user simple userD: 6 (Permission denied)
:: [   PASS   ] :: Running 'cat /var/log/secure | grep "pam_sss(sshd:account): Access denied for user simple[ ]userD: 6 (Permission denied)"'
:: [   PASS   ] :: Authentication successfull connection closed, as expected
:: [   PASS   ] :: Running '> /var/log/secure'
Jun 27 10:38:22 client-rhel6 sshd[508]: pam_sss(sshd:account): Access denied for user simple userF: 6 (Permission denied)
:: [   PASS   ] :: Running 'cat /var/log/secure | grep "pam_sss(sshd:account): Access denied for user simple[ ]userF: 6 (Permission denied)"'
:: [   PASS   ] :: Authentication successfull connection closed, as expected

Monolithic suites

No way to request running only
a few tests, skipping others

As suite size increases, the
edit/test loop becomes painfully slow

The only way around is to edit code,
which leads to mistakes

Go make a coffee

# time SERVER="server.sss-test.test" CLIENT="client-rhel6.sss-test.test" make

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: defaults
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [   PASS   ] :: defaults_without
:: [   PASS   ] :: defaults_with
.  .  .
:: [   PASS   ] :: attrs_command_paranoid_non_equal2_match_negative
:: [   PASS   ] :: attrs_command_paranoid_non_equal2_mismatch
:: [ 18:34:04 ] ::  JOURNAL XML: /var/tmp/beakerlib-y2ii2jZ/journal.xml
:: [ 18:34:04 ] ::  JOURNAL TXT: /var/tmp/beakerlib-y2ii2jZ/journal.txt
result_server not set, assuming developer mode.
Setting client-rhel6.sss-test.test to state DONE

real    10m28.666s  
user    0m10.454s 
sys     0m8.929s

10 minutes

Got milk?

Setup/teardown neglect

Proceeds with execution even if
a setup/teardown command failed

This obscures failure causes,
complicates debugging

The only way around is to verify each
command status, as "set -e" is not supported

Wait, what?

:: [   FAIL   ] :: Running 'cat /var/log/secure | grep "pam_sss(sshd:account): Access denied for user simple[ ]userC: 6 (Permission denied)"' (Expected 0, got 1)

50 lines back...

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: simple_group_Setup
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

ldap_bind: Invalid credentials (49)
:: [   FAIL   ] :: Running 'ldapmodify -x -h server.sss-test.test -D "cn=Manager,dc=example,dc=com" -w Secret124 -a -f /tmp/setup.ldif' (Expected 0, got 49)
:: [ 10:43:23 ] ::  Server Setup:
:: [ 10:43:23 ] ::  simple group1 => simple UserA simple UserB simple Group3
:: [ 10:43:23 ] ::  simple Group2 => simple UserC simple UserD
:: [ 10:43:23 ] ::  simple Group3 => simple UserE
:: [ 10:43:23 ] ::  simple UserF (belongs to no group/its own group)
:: [ 10:43:23 ] ::

More definitions

Epoxy has only two kinds of assertions:
"suite" and "test"

A suite contains only setup/teardown
code and other suites and tests

A test contains only verification code

A suite is considered passing only
if all its sub-suites and tests pass

How does it look?

#!/bin/bash

# Add Epoxy module directory to PATH
eval "`ep_env || echo exit 1`"
# Source epoxy modules
. ep.sh
# Initialize the suite shell, process command line arguments
ep_suite_init "$@"

# Setup
declare TMP_DIR="`mktemp -d`"
# Add a teardown command
ep_teardown_push rm -Rf "$TMP_DIR"

# More setup and teardown
pushd "$TMP_DIR" >/dev/null
ep_teardown_push eval 'popd >/dev/null'

But where's the meat?

That was only the beginning

# Run a test
ep_test "touch_present" command -v "test"
# Run another test
ep_test "rm_present" command -v "rm"

# Run a sub-suite
ep_suite_begin "file"; (
    ep_suite_init
    # Sub-suite teardown
    ep_teardown_push rm -f "file1" "file2"
    # A test
    ep_test "create" touch "file1"
    # Setup
    touch "file2"
    # Another test
    ep_test "remove" rm "file2"
); ep_suite_end

So Why Epoxy instead?

Allows infinite assertion nesting

Produces terse, clear output

Allows selection of assertions to verify

Handles setup/teardown failures

Assertion nesting

Assertions can be nested indefinitely
and referred to by path names

Each assertion can be implemented either
as an executable, a shell function, or a subshell

Transitions between implementations are easy

This simplifies implementing, integrating
and splitting arbitrarily large suites.

It takes all kinds

function test_function()
{
    true
}
function suite_function()
{
    ep_test "executable_test" true
    ep_test_sh "function_test" test_function
}
ep_suite_begin "subshell_suite"; (
    ep_suite_init
    ep_suite_sh "function_suite" suite_function
    ep_test_begin "subshell_test"; (
        ep_test_init
        true
    ); ep_test_end
); ep_suite_end

and combines them

#!/bin/bash

eval "`ep_env || echo exit 1`"
. ep.sh
ep_suite_init "$@"

declare depth="${EP_SUITE_ARGS[0]-0}"
declare i

if ((depth >= 3)); then
    exit
fi

for ((i = 1; i <= 3; i++)); do
    ep_suite "suite$i" "$0" -- "$((depth + 1))"
done

Into recursion!

CLEAR OUTPUT

Epoxy produces structured plain text log

Status messages and script output
lines are clearly separated

This log is "cooked" by default,
outputting one assertion per line

Log filtering is supported, along with
conversion to BeakerLib log format

Eat'em raw

$ ./example -r
STRUCT BEGIN ''
STRUCT BEGIN '/file'
STRUCT BEGIN '/file/create'
STRUCT END   '/file/create' PASSED
STRUCT BEGIN '/file/remove'
STRUCT END   '/file/remove' PASSED
STRUCT BEGIN '/file/stress'
STRUCT BEGIN '/file/stress/1'
OUTPUT Creating 1
STRUCT END   '/file/stress/1' PASSED
STRUCT BEGIN '/file/stress/2'
OUTPUT Creating 2
STRUCT END   '/file/stress/2' PASSED
STRUCT BEGIN '/file/stress/3'
OUTPUT Creating 3
STRUCT END   '/file/stress/3' PASSED
OUTPUT Removing 1
OUTPUT Removing 2
OUTPUT Removing 3
STRUCT END   '/file/stress' PASSED
STRUCT END   '/file' PASSED
STRUCT END   '' PASSED

Or cook'em

$ ./example
/file/create PASSED
/file/remove PASSED
/file/stress/1 PASSED
/file/stress/2 PASSED
/file/stress/3 PASSED
/file/stress PASSED
/file PASSED
PASSED

Suppress THem

$ ./example -f-b2
/file/create PASSED
/file/remove PASSED
/file/stress PASSED
/file PASSED
PASSED

Or convert them

$ ./example -r | ep_log_beakerlib

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: /file
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [   PASS   ] :: /file/create
:: [   PASS   ] :: /file/remove
:: [   PASS   ] :: /file/stress
:: [ 17:05:37 ] ::  JOURNAL XML: /var/tmp/beakerlib-XpmJyEb/journal.xml
:: [ 17:05:37 ] ::  JOURNAL TXT: /var/tmp/beakerlib-XpmJyEb/journal.txt

Test selection

Suite executables accept path patterns
matching sub-suites/tests to run

Patterns can be supplied via either
command line or environment variables

This reduces edit/test loop overhead,
speeds up development

Simplifies communicating specific test runs

Cut to THE CHASE

Run as is

$ ./example_long
/file/create PASSED
/file/remove PASSED
/file/stress/1 PASSED
/file/stress/2 PASSED
/file/stress/3 PASSED
/file/stress/4 PASSED
/file/stress/5 PASSED
/file/stress PASSED
/file PASSED
/dir/create PASSED
/dir/remove PASSED
/dir/stress/1 PASSED
/dir/stress/2 PASSED
/dir/stress/3 PASSED
/dir/stress/4 PASSED
/dir/stress/5 PASSED
/dir/stress PASSED
/dir PASSED
PASSED

Give some paths

$ ./example_long /file/create /dir/remove
/file/create PASSED
/file PASSED
/dir/remove PASSED
/dir PASSED
PASSED

Or use glob patterns

$ ./example_long /file/@(create|remove|stress/[123])
/file/create PASSED
/file/remove PASSED
/file/stress/1 PASSED
/file/stress/2 PASSED
/file/stress/3 PASSED
/file/stress PASSED
/file PASSED
PASSED

Setup/teardown CARE

All code runs with "set -e"

All the code in suites is considered
setup/teardown code

If any setup command fails, the suite
stops and proceeds to teardown

If any teardown command fails
the whole suite stack terminates

No chance for another error

The setup

ep_test dusk true
ep_test bar true
ep_test vampires true
shotgun
ep_test dawn true

The outcome

/dusk PASSED
/bar PASSED
/vampires PASSED
ERRORED "./dusk_dawn: line 10: shotgun: command not found"

Don't panic

ep_suite_begin Earth; (
    ep_suite_init
    ep_suite_begin Arthur; (
        ep_suite_init
        ep_teardown_push "lifting ray"
        ep_test bulldozers true
        touch earth
        ep_test bar true
        ep_test beer true
        ep_test vogons true
    ); ep_suite_end
    ep_suite_begin Ford; (
        ep_suite_init
        ep_test towel true
    ); ep_suite_end
); ep_suite_end

Actually...
do
panic!

/Earth/Arthur/bulldozers PASSED
/Earth/Arthur/bar PASSED
/Earth/Arthur/beer PASSED
/Earth/Arthur/vogons PASSED
/Earth/Arthur PANICKED "ep_teardown.sh: line 55: lifting ray: command not found"
/Earth PANICKED
PANICKED

Why else Epoxy?

Supports disabling/enabling
assertions

Supports waiving/claiming
assertion verification results

A failure reason string can be
attached to any assertion

What does it do already?

Tests itself (1.3 KLOC)

Tests Carton (2.3 KLOC)

Works on Beaker

Why not Epoxy?

Requires more attentive coding and
deeper understanding of Bash

The "set -e" behavior is quirky and
sometimes non-obvious

No special purpose assertions
— implement as a library on top

What's still missing?

TCMS (Nitrate) integration

Suite/test code consistency verification

Minor fixes

Extensive testing

What's next?

Trace logging, similar to "set -x"

Implicit and explicit breakpoints

Clean SIGINT/SIGTERM handling

Remote and parallel assertions

Where to get?

GitHub

https://github.com/spbnick/epoxy

Red Hat internal RPM repo

https://url.corp.redhat.com/sssd-qe-devel-rpm

(follow the redirect)

How to help?

Install and use

Clone and hack

And finally...

Tell your manager that Red Hat needs it,
so work on it is approved and
can proceed faster

Thank you!

This presentation on slid.es:

https://slid.es/spbnick/epoxy

Licensed under

Creative Commons Attribution License

Epoxy - a flexible test framework

By Nikolai Kondrashov

Epoxy - a flexible test framework

7,196

Nikolai Kondrashov

spbnick

Introducing

Epoxy

A flexible test framework

Nikolai Kondrashov

EPOXY

a definition

Why not BeakerLib?

Limited structuring

Three is not enough

Noisy output

Spot the failure

Monolithic suites

Go make a coffee

Setup/teardown neglect

Wait, what?

More definitions

How does it look?

That was only the beginning

So Why Epoxy instead?

Assertion nesting

It takes all kinds

and combines them

CLEAR OUTPUT

Eat'em raw

Or cook'em

Suppress THem

Or convert them

Test selection

Cut to THE CHASE

Setup/teardown CARE

No chance for another error

Don't panic

Why else Epoxy?

What does it do already?

Why not Epoxy?

What's still missing?

What's next?

Where to get?

How to help?

Thank you!

Epoxy - a flexible test framework

More from Nikolai Kondrashov