Mutation Tests in Ruby

Paweł Duda

https://slides.com/pawelduda/mutation-tests-in-ruby/

How do I know
my tests don't suck?

Write tests to test tests

https://github.com/mbj/mutant

mutant --use rspec Rectangle

gem install mutant-rspec

How can this test be improved to kill all mutations?

Ruby code

Ruby AST

Mutated Ruby AST (x total combinations)

parser

mutant

unparser

Mutated Ruby code (x total combinations)

Run tests against each mutated output

Expect each to fail at least one example

General tips

Don't run Mutant against your entire codebase at once
(mutant --since master --use rspec Namespace::Class)
- It might take hours (and crash somewhere in the middle)
- You will drown in output
Don't aim to reach 100% coverage "at all costs"
- Try to ask yourself whether improving the coverage by few
  % is worth the time spent on it
- 100% now does not mean 100% in the future anyway

Stricter code = less side effects

foo[:bar]

foo.fetch(:bar)

age.to_i

Integer(age)

Date.parse(date)

Date.iso8601(date)

2.2.3 :001 > person = { name: 'John' }
=> {:name=>"John"}

2.2.3 :002 > person[:address]
=> nil

2.2.3 :003 > person.fetch(:address)
KeyError: key not found: :address
   from (irb):3:in `fetch'
   from (irb):3
   from /home/pawelduda/.rvm/rubies/ruby-2.2.3/bin/irb:11:in `<main>'

2.2.3 :001 > nil.to_i
=> 0

2.2.3 :002 > Integer(nil)
TypeError: can't convert nil into Integer
   from (irb):2:in `Integer'
   from (irb):2
   from /home/pawelduda/.rvm/rubies/ruby-2.2.3/bin/irb:11:in `<main>'

2.2.3 :003 > Date.parse('Trust me, I am random string with a number 10')
=> #<Date: 2016-03-10 ((2457458j,0s,0n),+0s,2299161j)>

2.2.3 :004 > Date.iso8601('Trust me, I am random string with a number 10')
ArgumentError: invalid date
   from (irb):4:in `iso8601'
   from (irb):4
   from /home/pawelduda/.rvm/rubies/ruby-2.2.3/bin/irb:11:in `<main>'

2.2.3 :005 > Date.iso8601('2016-03-14')

=> #<Date: 2016-03-14 ((2457462j,0s,0n),+0s,2299161j)>

Get started with Mutant

https://github.com/mbj/mutant

/home/pawelduda/.rvm/gems/ruby-2.2.4/bundler/gems/mutant-2c71dbcfafa2/lib/mutant/integration/rspec.rb:121:in `match': invalid byte sequence in UTF-8 (ArgumentError)

Module#name from: XPath::HTML returned name(.). Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil
Module#name from: XPath returned name(.). Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil

Bug with unescaped string sequences

Questions?