Introducing
Epoxy
A flexible test framework
Red Hat QA meet-up
Camping Baldovec, Czech Republic
June 2013
Nikolai Kondrashov
spbnick
Sr. QA Engineer at Red Hat IDM QE.
Formerly Sr. System Software Engineer
at NVIDIA Tegra graphics team.
Open Source developer.
Founder of DIGImend project - improving
Linux support for generic graphics tablets.
EPOXY
Bash test framework
An alternative to BeakerLib
3 KLOC in Bash, Lua and Python
Nearing first release, already useful
Personal spare-time project
a definition
A suite, a test phase, a test
— all verify an assertion
— all verify an assertion
"Assertion" is often used
in place of "assertion verification"
in place of "assertion verification"
Thus we will use "assertion" to mean
any of "suite", "test phase", or "test"
any of "suite", "test phase", or "test"
Why not BeakerLib?
Limits assertion structuring
Produces noisy output
Can't verify a subset of assertions
Ignores setup/teardown failures
Limited structuring
Three levels only:
"suite", "phase", "assertion"
"suite", "phase", "assertion"
Either have huge flat suites
or integrate by external means
or integrate by external means
Either way hard to read and maintain
Three is not enough
Suite
#!/bin/bash
# Phase
rlPhaseStartTest "defaults"
# Assertion
rlRun client_sudo_user_requires_auth 0 "defaults_without"
# Assertion
rlRun client_sudo_user_is_allowed 0 "defaults_with"
rlPhaseEnd
# Phase
rlPhaseStartTest "config"
# Assertion
rlRun client_sudo_user_is_allowed 0 "config_all_options"
rlPhaseEnd
How about something more complex?
Noisy output
Mixes test results with
script output while running
script output while running
Uses heavy decoration around
result messages as a workaround
result messages as a workaround
This makes it harder to track progress
and spot failures and their causes
and spot failures and their causes
Spot the failure
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: simple_access_001: where simple_allow_groups = simple group1 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: spawn ssh -o StrictHostKeyChecking=no -l simple userB localhost simple userB@localhost's password: Last login: Thu Jun 27 10:19:34 2013 from localhost.localdomain [simple userB@client-rhel6 ~]$ :: [ PASS ] :: Authentication successfull, as expected spawn ssh -o StrictHostKeyChecking=no -l simple userE localhost simple userE@localhost's password: Permission denied, please try again. simple userE@localhost's password: :: [ FAIL ] :: ERROR: Authentiation failed, not as expected Jun 27 10:38:16 client-rhel6 sshd[484]: pam_sss(sshd:account): Access denied for user simple userD: 6 (Permission denied) :: [ PASS ] :: Running 'cat /var/log/secure | grep "pam_sss(sshd:account): Access denied for user simple[ ]userD: 6 (Permission denied)"' :: [ PASS ] :: Authentication successfull connection closed, as expected :: [ PASS ] :: Running '> /var/log/secure' Jun 27 10:38:22 client-rhel6 sshd[508]: pam_sss(sshd:account): Access denied for user simple userF: 6 (Permission denied) :: [ PASS ] :: Running 'cat /var/log/secure | grep "pam_sss(sshd:account): Access denied for user simple[ ]userF: 6 (Permission denied)"' :: [ PASS ] :: Authentication successfull connection closed, as expected
Monolithic suites
No way to request running only
a few tests, skipping others
a few tests, skipping others
As suite size increases, the
edit/test loop becomes painfully slow
edit/test loop becomes painfully slow
The only way around is to edit code,
which leads to mistakes
which leads to mistakes
Go make a coffee
# time SERVER="server.sss-test.test" CLIENT="client-rhel6.sss-test.test" make :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: defaults :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ PASS ] :: defaults_without :: [ PASS ] :: defaults_with . . . :: [ PASS ] :: attrs_command_paranoid_non_equal2_match_negative :: [ PASS ] :: attrs_command_paranoid_non_equal2_mismatch :: [ 18:34:04 ] :: JOURNAL XML: /var/tmp/beakerlib-y2ii2jZ/journal.xml :: [ 18:34:04 ] :: JOURNAL TXT: /var/tmp/beakerlib-y2ii2jZ/journal.txt result_server not set, assuming developer mode. Setting client-rhel6.sss-test.test to state DONE real 10m28.666s user 0m10.454s sys 0m8.929s
10 minutes
Got milk?
Setup/teardown neglect
Proceeds with execution even if
a setup/teardown command failed
a setup/teardown command failed
This obscures failure causes,
complicates debugging
complicates debugging
The only way around is to verify each
command status, as "set -e" is not supported
command status, as "set -e" is not supported
Wait, what?
:: [ FAIL ] :: Running 'cat /var/log/secure | grep "pam_sss(sshd:account): Access denied for user simple[ ]userC: 6 (Permission denied)"' (Expected 0, got 1)
50 lines back...
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: simple_group_Setup :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: ldap_bind: Invalid credentials (49) :: [ FAIL ] :: Running 'ldapmodify -x -h server.sss-test.test -D "cn=Manager,dc=example,dc=com" -w Secret124 -a -f /tmp/setup.ldif' (Expected 0, got 49) :: [ 10:43:23 ] :: Server Setup: :: [ 10:43:23 ] :: simple group1 => simple UserA simple UserB simple Group3 :: [ 10:43:23 ] :: simple Group2 => simple UserC simple UserD :: [ 10:43:23 ] :: simple Group3 => simple UserE :: [ 10:43:23 ] :: simple UserF (belongs to no group/its own group) :: [ 10:43:23 ] ::
More definitions
Epoxy has only two kinds of assertions:
"suite" and "test"
"suite" and "test"
A suite contains only setup/teardown
code and other suites and tests
code and other suites and tests
A test contains only verification code
A suite is considered passing only
if all its sub-suites and tests pass
if all its sub-suites and tests pass
How does it look?
#!/bin/bash
# Add Epoxy module directory to PATH
eval "`ep_env || echo exit 1`"
# Source epoxy modules
. ep.sh
# Initialize the suite shell, process command line arguments
ep_suite_init "$@"
# Setup
declare TMP_DIR="`mktemp -d`"
# Add a teardown command
ep_teardown_push rm -Rf "$TMP_DIR"
# More setup and teardown
pushd "$TMP_DIR" >/dev/null
ep_teardown_push eval 'popd >/dev/null'But where's the meat?
That was only the beginning
# Run a test
ep_test "touch_present" command -v "test"
# Run another test
ep_test "rm_present" command -v "rm"
# Run a sub-suite
ep_suite_begin "file"; (
ep_suite_init
# Sub-suite teardown
ep_teardown_push rm -f "file1" "file2"
# A test
ep_test "create" touch "file1"
# Setup
touch "file2"
# Another test
ep_test "remove" rm "file2"
); ep_suite_end So Why Epoxy instead?
Allows infinite assertion nesting
Produces terse, clear output
Allows selection of assertions to verify
Handles setup/teardown failures
Assertion nesting
Assertions can be nested indefinitely
and referred to by path names
and referred to by path names
Each assertion can be implemented either
as an executable, a shell function, or a subshell
as an executable, a shell function, or a subshell
Transitions between implementations are easy
This simplifies implementing, integrating
and splitting arbitrarily large suites.
and splitting arbitrarily large suites.
It takes all kinds
function test_function()
{
true
}
function suite_function()
{
ep_test "executable_test" true
ep_test_sh "function_test" test_function
}
ep_suite_begin "subshell_suite"; (
ep_suite_init
ep_suite_sh "function_suite" suite_function
ep_test_begin "subshell_test"; (
ep_test_init
true
); ep_test_end
); ep_suite_end
and combines them
#!/bin/bash
eval "`ep_env || echo exit 1`"
. ep.sh
ep_suite_init "$@"
declare depth="${EP_SUITE_ARGS[0]-0}"
declare i
if ((depth >= 3)); then
exit
fi
for ((i = 1; i <= 3; i++)); do
ep_suite "suite$i" "$0" -- "$((depth + 1))"
done
Into recursion!
CLEAR OUTPUT
Epoxy produces structured plain text log
Status messages and script output
lines are clearly separated
lines are clearly separated
This log is "cooked" by default,
outputting one assertion per line
outputting one assertion per line
Log filtering is supported, along with
conversion to BeakerLib log format
conversion to BeakerLib log format
Eat'em raw
$ ./example -r STRUCT BEGIN '' STRUCT BEGIN '/file' STRUCT BEGIN '/file/create' STRUCT END '/file/create' PASSED STRUCT BEGIN '/file/remove' STRUCT END '/file/remove' PASSED STRUCT BEGIN '/file/stress' STRUCT BEGIN '/file/stress/1' OUTPUT Creating 1 STRUCT END '/file/stress/1' PASSED STRUCT BEGIN '/file/stress/2' OUTPUT Creating 2 STRUCT END '/file/stress/2' PASSED STRUCT BEGIN '/file/stress/3' OUTPUT Creating 3 STRUCT END '/file/stress/3' PASSED OUTPUT Removing 1 OUTPUT Removing 2 OUTPUT Removing 3 STRUCT END '/file/stress' PASSED STRUCT END '/file' PASSED STRUCT END '' PASSED
Or cook'em
$ ./example /file/create PASSED /file/remove PASSED /file/stress/1 PASSED /file/stress/2 PASSED /file/stress/3 PASSED /file/stress PASSED /file PASSED PASSED
Suppress THem
$ ./example -f-b2 /file/create PASSED /file/remove PASSED /file/stress PASSED /file PASSED PASSED
Or convert them
$ ./example -r | ep_log_beakerlib :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: /file :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ PASS ] :: /file/create :: [ PASS ] :: /file/remove :: [ PASS ] :: /file/stress :: [ 17:05:37 ] :: JOURNAL XML: /var/tmp/beakerlib-XpmJyEb/journal.xml :: [ 17:05:37 ] :: JOURNAL TXT: /var/tmp/beakerlib-XpmJyEb/journal.txt
Test selection
Suite executables accept path patterns
matching sub-suites/tests to run
matching sub-suites/tests to run
Patterns can be supplied via either
command line or environment variables
command line or environment variables
This reduces edit/test loop overhead,
speeds up development
speeds up development
Simplifies communicating specific test runs
Cut to THE CHASE
Run as is
$ ./example_long /file/create PASSED /file/remove PASSED /file/stress/1 PASSED /file/stress/2 PASSED /file/stress/3 PASSED /file/stress/4 PASSED /file/stress/5 PASSED /file/stress PASSED /file PASSED /dir/create PASSED /dir/remove PASSED /dir/stress/1 PASSED /dir/stress/2 PASSED /dir/stress/3 PASSED /dir/stress/4 PASSED /dir/stress/5 PASSED /dir/stress PASSED /dir PASSED PASSED
Give some paths
$ ./example_long /file/create /dir/remove /file/create PASSED /file PASSED /dir/remove PASSED /dir PASSED PASSED
Or use glob patterns
$ ./example_long /file/@(create|remove|stress/[123]) /file/create PASSED /file/remove PASSED /file/stress/1 PASSED /file/stress/2 PASSED /file/stress/3 PASSED /file/stress PASSED /file PASSED PASSED
Setup/teardown CARE
All code runs with "set -e"
All the code in suites is considered
setup/teardown code
setup/teardown code
If any setup command fails, the suite
stops and proceeds to teardown
stops and proceeds to teardown
If any teardown command fails
the whole suite stack terminates
the whole suite stack terminates
No chance for another error
The setup
ep_test dusk true
ep_test bar true
ep_test vampires true
shotgun
ep_test dawn true /dusk PASSED
/bar PASSED
/vampires PASSED
ERRORED "./dusk_dawn: line 10: shotgun: command not found" Don't panic
ep_suite_begin Earth; ( ep_suite_init ep_suite_begin Arthur; ( ep_suite_init ep_teardown_push "lifting ray" ep_test bulldozers true touch earth ep_test bar true ep_test beer true ep_test vogons true ); ep_suite_endep_suite_begin Ford; ( ep_suite_init ep_test towel true ); ep_suite_end ); ep_suite_end
Actually...
do
panic!
do
panic!
/Earth/Arthur/bulldozers PASSED /Earth/Arthur/bar PASSED /Earth/Arthur/beer PASSED /Earth/Arthur/vogons PASSED /Earth/Arthur PANICKED "ep_teardown.sh: line 55: lifting ray: command not found" /Earth PANICKED PANICKED
Why else Epoxy?
Supports disabling/enabling
assertions
assertions
Supports waiving/claiming
assertion verification results
assertion verification results
A failure reason string can be
attached to any assertion
attached to any assertion
What does it do already?
Tests itself (1.3 KLOC)
Tests Carton (2.3 KLOC)
Why not Epoxy?
Requires more attentive coding and
deeper understanding of Bash
deeper understanding of Bash
The "set -e" behavior is quirky and
sometimes non-obvious
sometimes non-obvious
No special purpose assertions
— implement as a library on top
— implement as a library on top
What's still missing?
TCMS (Nitrate) integration
Suite/test code consistency verification
Minor fixes
Extensive testing
What's next?
Trace logging, similar to "set -x"
Implicit and explicit breakpoints
Clean SIGINT/SIGTERM handling
Remote and parallel assertions
Where to get?
Red Hat internal RPM repo
(follow the redirect)
How to help?
Install and use
Clone and hack
And finally...
Tell your manager that Red Hat needs it,
so work on it is approved and
can proceed faster
so work on it is approved and
can proceed faster
Thank you!
This presentation on slid.es:
Licensed under