Mutation Tests in Ruby
Paweł Duda
https://slides.com/pawelduda/mutation-tests-in-ruby/
How do I know
my tests don't suck?
Write tests to test tests
https://github.com/mbj/mutant
mutant --use rspec Rectangle
gem install mutant-rspec
How can this test be improved to kill all mutations?
Ruby code
Ruby AST
Mutated Ruby AST (x total combinations)
parser
mutant
unparser
Mutated Ruby code (x total combinations)
Run tests against each mutated output
Expect each to fail at least one example
General tips
- Don't run Mutant against your entire codebase at once
(mutant --since master --use rspec Namespace::Class)- It might take hours (and crash somewhere in the middle)
- You will drown in output
- Don't aim to reach 100% coverage "at all costs"
- Try to ask yourself whether improving the coverage by few
% is worth the time spent on it - 100% now does not mean 100% in the future anyway
- Try to ask yourself whether improving the coverage by few
Stricter code = less side effects
foo[:bar]
foo.fetch(:bar)
age.to_i
Integer(age)
Date.parse(date)
Date.iso8601(date)
vs
vs
vs
2.2.3 :001 > person = { name: 'John' }
=> {:name=>"John"}
2.2.3 :002 > person[:address]
=> nil
2.2.3 :003 > person.fetch(:address)
KeyError: key not found: :address
from (irb):3:in `fetch'
from (irb):3
from /home/pawelduda/.rvm/rubies/ruby-2.2.3/bin/irb:11:in `<main>'
2.2.3 :001 > nil.to_i
=> 0
2.2.3 :002 > Integer(nil)
TypeError: can't convert nil into Integer
from (irb):2:in `Integer'
from (irb):2
from /home/pawelduda/.rvm/rubies/ruby-2.2.3/bin/irb:11:in `<main>'
2.2.3 :003 > Date.parse('Trust me, I am random string with a number 10')
=> #<Date: 2016-03-10 ((2457458j,0s,0n),+0s,2299161j)>
2.2.3 :004 > Date.iso8601('Trust me, I am random string with a number 10')
ArgumentError: invalid date
from (irb):4:in `iso8601'
from (irb):4
from /home/pawelduda/.rvm/rubies/ruby-2.2.3/bin/irb:11:in `<main>'
2.2.3 :005 > Date.iso8601('2016-03-14')
=> #<Date: 2016-03-14 ((2457462j,0s,0n),+0s,2299161j)>
Get started with Mutant
https://github.com/mbj/mutant
/home/pawelduda/.rvm/gems/ruby-2.2.4/bundler/gems/mutant-2c71dbcfafa2/lib/mutant/integration/rspec.rb:121:in `match': invalid byte sequence in UTF-8 (ArgumentError)
Module#name from: XPath::HTML returned name(.). Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil
Module#name from: XPath returned name(.). Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil
Bug with unescaped string sequences
Questions?
Mutation Tests in Ruby
By Paweł Duda
