@giorgionatili
@giorgionatili
@giorgionatili
int d; // elapsed time in daysint elapsedTimeInDays;VS
@giorgionatili
namespace stringutil {
std::string tail(const std::string& word) {
if (word.length() == 0) return "";
return word.substr(1);
}
}TEST(AString, AllTheLettersAfterTheHeadAsShlouldBeTheTail) {
ASSERT_THAT(tail("xyz"), Eq("yz"));
}
TEST(AString, TheTailOfAnEmptyStringShouldBeEmpty) {
ASSERT_THAT(tail(""), Eq(""));
}
TEST(AString, TheTailOfASingleCharacterStringShouldBeEmpty) {
ASSERT_THAT(tail("X"), Eq(""));
}
@giorgionatili
TEST(AString, AllTheLettersAfterTheHeadAsShlouldBeTheTail) {
ASSERT_THAT(tail("xyz"), Eq("yz"));
}
TEST(AString, TheTailOfAnEmptyStringShouldBeEmpty) {
ASSERT_THAT(tail(""), Eq(""));
}
TEST(AString, TheTailOfASingleCharacterStringShouldBeEmpty) {
tail("X");
}
@giorgionatili
bool checkEvenOdd(int num){
return num % 2 == 0 ? true : false;
}bool isDigit(char *c_array){
for (int k = 0; k < strlen(c_array); k++) {
if ((int)c_array[k]<(int)'0' ||
(int)c_array[k]>(int)'9') {
return false;
}
}
return true;
}@giorgionatili
Each transformation results in a new program, called mutant, that differs from the original
Detecting and rejecting such a modification by the existing tests is denoted as killing a mutant
Test suite effectiveness is measured by its ability to detect those mutants
The mutation score is the ratio of killed mutants to the total number of mutants
This is where mutation testing comes into play!
// Initial code:
if(a < b) {
c = 10;
} else {
c = 20;
}
// Changed code:
if(a < b) {
d = 10;
} else {
d = 20;
}// Initial code:
int mod = 1000000007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;
// Mutated code:
int mod = 1007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;// Initial code:
if(a < b) {
c = 10;
} else {
c = 20;
}
// Mutated code:
if(a > b) {
c = 10;
} else {
c = 20;
}int greatestCommonDenominator(int x, int y) {
int tmp;
while(y != 0) {
tmp = x % y; // The % operator can be replaced
x = y; // with +,-,*,/,%,**
y = tmp;
}
return x;
}int greatestCommonDenominator(int x, int y) {
int tmp;
while(y != 0) { // The != operator can be
tmp = x % y; // replaced by <,>,<=,>=,=, !=
x = y;
y = tmp;
}
return x;
}if(a && b)
// Potential mutations
if(a || b)
if(a & b)
if(a | b)
if(a ^ b)
if(false)
if(true)
if(a)
if(b)
@giorgionatili
Java, JVM
C, C++
Javascript
Rust
Swift
Ruby
PHP
C#
Closure
Python
Scala
Java, JVM
C, C++
Javascript
Rust
Swift
Ruby
PHP
C#
Closure
Python
Scala
mull-cxx -test-framework=GoogleTest -mutators=conditional \
-reporters=Elements -report-dir=./report \
-report-name=MULL-TEST-ONE \
-workers=4 -compdb-path compile_commands \
-disable-cache=0 \
./bin/core-test$ ./mull-cxx --help--mutators=<value> - Choose mutators:
=all - default, experimental
=arithmetic - cxx_arithmetic_add_to_sub, cxx_arithmetic_add_assign_to_sub_assign, cxx_arithmetic_post_inc_to_post_dec, cxx_arithmetic_pre_inc_to_pre_dec, cxx_arithmetic_sub_to_add, cxx_arithmetic_sub_assign_to_add_assign, cxx_arithmetic_post_dec_to_post_inc, cxx_arithmetic_pre_dec_to_pre_inc, cxx_arithmetic_mul_to_div, cxx_arithmetic_mul_assign_to_div_assign, cxx_arithmetic_div_to_mul, cxx_arithmetic_div_assign_to_mul_assign, cxx_arithmetic_rem_to_div, cxx_arithmetic_rem_assign_to_div_assign
=bitwise - cxx_bitwise_lshift_to_rshift, cxx_bitwise_lshift_assign_to_rshift_assign, cxx_bitwise_rshift_to_lshift, cxx_bitwise_rshift_assign_to_lshift_assign, cxx_bitwise_and_to_or, cxx_bitwise_and_assign_to_or_assign, cxx_bitwise_or_to_and, cxx_bitwise_or_assign_to_and_assign, cxx_bitwise_xor_to_or, cxx_bitwise_xor_assign_to_or_assign
=conditional - and_or_replacement_mutator, negate_mutator, conditionals_boundary_mutator, negate_relational
=conditionals_boundary_mutator - cxx_relational_le_to_lt, cxx_relational_lt_to_le, cxx_relational_ge_to_gt, cxx_relational_gt_to_ge
=constant - scalar_value_mutator
=cxx - conditionals_boundary_mutator, negate_relational, arithmetic, numbers
=default - cxx_arithmetic_add_to_sub, negate_mutator, remove_void_function_mutator
=experimental - and_or_replacement_mutator, numbers, replace_call_mutator, scalar_value_mutator, conditionals_boundary_mutator, negate_relational, arithmetic, bitwise
=functions - replace_call_mutator, remove_void_function_mutator
=math - cxx_arithmetic_add_to_sub, cxx_arithmetic_sub_to_add, cxx_arithmetic_mul_to_div, cxx_arithmetic_div_to_mul
=negate_relational - cxx_relational_gt_to_le, cxx_relational_ge_to_lt, cxx_relational_lt_to_ge, cxx_relational_le_to_gt, cxx_relational_eq_to_ne, cxx_relational_ne_to_eq
=numbers - cxx_number_init_const, cxx_number_assign_const
=and_or_replacement_mutator - Replaces && with ||, || with &&
=cxx_arithmetic_add_assign_to_sub_assign - Replaces += with -=
=cxx_arithmetic_add_to_sub - Replaces + with -
=cxx_arithmetic_div_assign_to_mul_assign - Replaces /= with *=
=cxx_arithmetic_div_to_mul - Replaces / with *
=cxx_arithmetic_mul_assign_to_div_assign - Replaces *= with /=
=cxx_arithmetic_mul_to_div - Replaces * with /
=cxx_arithmetic_post_dec_to_post_inc - Replaces x-- with x++
=cxx_arithmetic_post_inc_to_post_dec - Replaces x++ with x--
=cxx_arithmetic_pre_dec_to_pre_inc - Replaces --x with ++x
=cxx_arithmetic_pre_inc_to_pre_dec - Replaces ++x with --x
=cxx_arithmetic_rem_assign_to_div_assign - Replaces %= with /=
=cxx_arithmetic_rem_to_div - Replaces % with /
=cxx_arithmetic_sub_assign_to_add_assign - Replaces -= with +=
=cxx_arithmetic_sub_to_add - Replaces - with +
=cxx_bitwise_and_assign_to_or_assign - Replaces &= with |=
=cxx_bitwise_and_to_or - Replaces & with |
=cxx_bitwise_lshift_assign_to_rshift_assign - Replaces <<= with >>=
=cxx_bitwise_lshift_to_rshift - Replaces << with >>
=cxx_bitwise_or_assign_to_and_assign - Replaces |= with &=
=cxx_bitwise_or_to_and - Replaces | with &
=cxx_bitwise_rshift_assign_to_lshift_assign - Replaces >>= with <<=
=cxx_bitwise_rshift_to_lshift - Replaces << with >>
=cxx_bitwise_xor_assign_to_or_assign - Replaces ^= with |=
=cxx_bitwise_xor_to_or - Replaces ^ with |
=cxx_number_assign_const - Replaces 'a = b' with 'a = 42'
=cxx_number_init_const - Replaces 'T a = b' with 'T a = 42'
=cxx_relational_eq_to_ne - Replaces == with !=
=cxx_relational_ge_to_gt - Replaces >= with >
=cxx_relational_ge_to_lt - Replaces >= with <
=cxx_relational_gt_to_ge - Replaces > with >=
=cxx_relational_gt_to_le - Replaces > with <=
=cxx_relational_le_to_gt - Replaces <= with >
=cxx_relational_le_to_lt - Replaces <= with <
=cxx_relational_lt_to_ge - Replaces < with >=
=cxx_relational_lt_to_le - Replaces < with <=
=cxx_relational_ne_to_eq - Replaces != with ==
=negate_mutator - Negates conditionals !x to x and x to !x
=remove_void_function_mutator - Removes calls to a function returning void
=replace_call_mutator - Replaces call to a function with 42
=scalar_value_mutator - Replaces zeros with 42, and non-zeros with 0@giorgionatili
mull-cxx -test-framework=GoogleTest -mutators=math \
-reporters=Elements -report-dir=./report -report-name=TEST \
-workers=4 -compdb-path compile_cmd.json -disable-cache=0 \
-compilation-flags="\
-isystem /opt/clang+llvm-9.0.0/include/c++/v1 \
-isystem /opt/clang+llvm-9.0.0/lib/clang/9.0.0/include \
-isystem /usr/include" \
./bin/core-testLoading bitcode files (threads: 4): 4/4. Finished in 267ms.
Compiling instrumented code (threads: 4): 4/4. Finished in 11ms.
Loading dynamic libraries (threads: 1): 1/1. Finished in 0ms.
Searching tests (threads: 1): 1/1. Finished in 2ms.
Preparing original test run (threads: 1): 1/1. Finished in 145ms.
Running original tests (threads: 4): 30/30. Finished in 187ms.
Applying function filter: no debug info (threads: 4): 3496/3496. Finished in 14ms.
Applying function filter: file path (threads: 4): 3313/3313. Finished in 22ms.
Instruction selection (threads: 4): 3313/3313. Finished in 23ms.
Searching mutants across functions (threads: 4): 3313/3313. Finished in 369ms.
Applying filter: no debug info (threads: 4): 12355/12355. Finished in 12ms.
Applying filter: file path (threads: 4): 12355/12355. Finished in 35ms.
Applying filter: junk (threads: 4): 12355/12355. Finished in 3657ms.
Prepare mutations (threads: 1): 1/1. Finished in 0ms.
Cloning functions for mutation (threads: 4): 4/4. Finished in 769ms.
Removing original functions (threads: 4): 4/4. Finished in 194ms.
Redirect mutated functions (threads: 4): 4/4. Finished in 11ms.
Applying mutations (threads: 1): 409/409. Finished in 11ms.
Compiling original code (threads: 4): 4/4. Finished in 3625ms.
Running mutants (threads: 4): 409/409. Finished in 4586ms.(fmt / include / fmt / core.h with math mutators)
(fmt / include / fmt / core.h with math mutators)
Compiling instrumented code (threads: 4): 4/4. Finished in 4612ms.
Loading dynamic libraries (threads: 1): 1/1. Finished in 0ms.
Searching tests (threads: 1): 1/1. Finished in 1ms.
Preparing original test run (threads: 1): 1/1. Finished in 86ms.
Running original tests (threads: 4): 30/30. Finished in 203ms.
Applying function filter: no debug info (threads: 4): 3496/3496. Finished in 15ms.
Applying function filter: file path (threads: 4): 3313/3313. Finished in 23ms.
Instruction selection (threads: 4): 3313/3313. Finished in 21ms.
Searching mutants across functions (threads: 4): 3313/3313. Finished in 608ms.
Applying filter: no debug info (threads: 4): 20586/20586. Finished in 15ms.
Applying filter: file path (threads: 4): 20586/20586. Finished in 58ms.
Applying filter: junk (threads: 4): 20586/20586. Finished in 3969ms.
Prepare mutations (threads: 1): 1/1. Finished in 1ms.
Cloning functions for mutation (threads: 4): 4/4. Finished in 1040ms.
Removing original functions (threads: 4): 4/4. Finished in 204ms.
Redirect mutated functions (threads: 4): 4/4. Finished in 13ms.
Applying mutations (threads: 1): 446/446. Finished in 10ms.
Compiling original code (threads: 4): 4/4. Finished in 3808ms.
Running mutants (threads: 4): 446/446. Finished in 5704ms.
Total execution time: 21046mstotal = 300 known bugs
labeled = 100 categorized bugs
found = 100 total bugs discovered with mutation
labeledFound = 30 existing bugs discovered with mutation
labeled
--------------------
unknown
labeledFound
--------------------------
notLabeledFound
=
unknown > 200 potential unknown bugs
@giorgionatili
Java, JVM
C, C++
Javascript
Rust
Swift
Ruby
PHP
C#
Closure
Python
Scala
@giorgionatili
By Giorgio Natili
Mutation testing is a technique used to evaluate the quality of existing test suites that implies modifying a program logic in small ways such as negating conditionals, changing a logical connector, and so on to introduce faults. When the application code changes, it should produce different results and cause the unit tests to fail. If a unit test does not fail in this situation, it may indicate that there is an issue in the test suite. Mutation testing typically utilizes a set of language-specific source code transformations, called operators, to introduce faults, but these operators may not be sufficient to generate enough mutations to cover the specific domain of the system under test. Join Giorgio in this session to learn more about mutation testing and how to create operators to generate mutations specific to a custom domain.
