Michael Kutz
Quality Engineer at REWE digital, Conference Speaker about QA & Agile, Founder of Agile QA Cologne meetup, Freelance QA Consultant
@MichaKutz
@mkutz@mstdn.social
Why Measure Quality?
Why do we need to talk about this?
add new feature
improve quality
Goal:
Make better informed decisions about quality
What Could Possibly Go Wrong?
Goodhart's Law
When a measure becomes a target, it ceases to be a good measure.
Negative Impact on Motivation/Collaboration
How to Find Metrics?
Goal → Question → Metric
Are we going in the right direction?
Where are we?
Which Quality?
How defective?
How well liked?
How functional?
Number of Found Bugs
in Staging vs Production
How defective?
How effective is the test process?
Broken Service Level Objectives
How defective?
How defective?
How well liked?
How functional?
Number of customer service complaints/contacts
How well liked?
How many people complaining?
User Surveys or Platform Ratings
How well liked?
How defective?
How well liked?
How functional?
How defective?
How functional?
How well liked?
How well protected?
How maintainable?
How confident is the team?
Code Coverage
@Test void strike() { var game = new Game(); var firstRollPins = 10; var secondRollPins = 5; var thirdRollPins = 3; game.roll(firstRollPins); game.roll(secondRollPins); game.roll(thirdRollPins); var score = game.score(); assertThat(score) .isEqualTo( firstRollPins + (secondRollPins + thirdRollPins) * 2); }
@Test void strike() { var game = new Game(); var firstRollPins = 10; var secondRollPins = 5; var thirdRollPins = 3; game.roll(firstRollPins); game.roll(secondRollPins); game.roll(thirdRollPins); var score = game.score(); assertThat(score) .isEqualTo( firstRollPins + (secondRollPins + thirdRollPins) * 2); }
@Test void strike() { var game = new Game(); var firstRollPins = 10; var secondRollPins = 5; var thirdRollPins = 3; game.roll(firstRollPins); game.roll(secondRollPins); game.roll(thirdRollPins); var score = game.score(); assertThat(score) .isEqualTo( firstRollPins + (secondRollPins + thirdRollPins) * 2); }
public int score() { int score = firstRoll + secondRoll; if (previous != null) { if (previous.strike) { score *= 2; } else if (previous.spare) { score += firstRoll; } } return score; }
@Test void strike() { var game = new Game(); var firstRollPins = 10; var secondRollPins = 5; var thirdRollPins = 3; game.roll(firstRollPins); game.roll(secondRollPins); game.roll(thirdRollPins); var score = game.score(); // assertThat(score) // .isEqualTo( // firstRollPins + // (secondRollPins + thirdRollPins) * 2); }
How well protected?
How much code is not executed by tests?
Mutation Testing: Surviving Mutations
@Test void strike() { var game = new Game(); var firstRollPins = 10; var secondRollPins = 5; var thirdRollPins = 3; game.roll(firstRollPins); game.roll(secondRollPins); game.roll(thirdRollPins); var score = game.score(); assertThat(score .isEqualTo( firstRollPins + (secondRollPins + thirdRollPins) * 2); }
public int score() { int score = firstRoll + secondRoll; if (previous != null) { if (previous.strike) { score *= 2; } else if (previous.spare) { score += firstRoll; } } return score; }
public int score() { int score = firstRoll + secondRoll; if (previous != null) { if (previous.strike) { score *= 2; } else if (previous.spare) { score += firstRoll; } } return score; }
AssertionFailedError:
expected: 26
but was: 14How well protected?
How well protected?
How maintainable?
How confident is the team?
Team Surveys
How confident is the team?
How effective can you work with the code?
How confident are you to deploy to production?
What would you need to improve the above answers?
How well protected?
How maintainable?
How confident is the team?
How maintainable?
How confident is the team?
How well protected?
How productive?
How safe?
How fast?
Velocity
v=tdelivery−tcommit1Pestimate
v = \frac{P_{estimate}}{t_{delivery} - t_{commit1}}
How fast?
How good are our estimations?
Delivery Lead Time
Δtdelivery=tdelivery−tcommit1
\Delta t_{delivery} = t_{delivery} - t_{commit1}
How fast?
How productive?
How safe?
How fast?
Change Fail Rate
08:07:23 Deploy basket-service ✓ 09:06:11 Migrate checkout-service DB ✓ 09:56:54 Deploy checkout-service ✓
08:07:23 Deploy basket-service ✓ 09:06:11 Migrate checkout-service DB ✓ 09:56:54 Deploy checkout-service ✓ 10:19:44 Deploy order-managment-service ✗ 10:39:27 Rollback order-managemet-service ✓
08:07:23 Deploy basket-service ✓ 09:06:11 Migrate checkout-service DB ✓ 09:56:54 Deploy checkout-service ✓ 10:19:44 Deploy order-managment-service ✗ 10:39:27 Rollback order-managemet-service ✓ 11:09:59 Update database cluster ✓ 12:27:32 Migrate order-managemet-service DB ✓ 13:19:22 Deploy order-managemet-service ✓
08:07:23 Deploy basket-service ✓ 09:06:11 Migrate checkout-service DB ✓ 09:56:54 Deploy checkout-service ✓ 10:19:44 Deploy order-managment-service ✗ 10:39:27 Rollback order-managemet-service ✓ 11:09:59 Update database cluster ✓ 12:27:32 Migrate order-managemet-service DB ✓ 13:19:22 Deploy order-managemet-service ✓ 14:45:55 Update database-cluster ✗ 15:50:49 Update database-cluster ✓
How safe?
Mean Time to Restore Service
08:07:23 Deploy basket-service ✓ 09:06:11 Migrate checkout-service DB ✓ 09:56:54 Deploy checkout-service ✓ 10:19:44 Deploy order-managment-service ✗ 10:39:27 Rollback order-managemet-service ✓ 11:09:59 Update database cluster ✓ 12:27:32 Migrate order-managemet-service DB ✓ 13:19:22 Deploy order-managemet-service ✓ 14:45:55 Update database-cluster ✗ 15:50:49 Update database-cluster ✓ 16:39:11 Deploy product-service ✓ 17:44:56 Deploy customer-data-service ✓
How safe?
How productive?
How safe?
How fast?
How productive?
How safe?
How fast?
How defective?
How well liked?
How functional?
How well protected?
How maintainable?
How confident is the team?
How productive?
How safe?
How fast?
@MichaKutz
@mkutz@mstdn.social
Not everything that counts can be counted,
and not everything that can be counted counts.
This talk as an article 👇
Hi, I'm Michael Kutz.
I'm a developer… who prefers to continuously produce working software over bug fixing phases.
Hence, I'm also a tester and test automation expert.
I also enjoy working in an organization where decisions are made by those who can own them.
Hence, I like working closely with management.
In each of my jobs so far, at some point, somebody tried to measure the quality of the software we were producing…
the result of their efforts was always less than satisfying.
The latest attempt led me to finally sit down think why that is.
This talk is the result of that thinking…
@MichaKutz @mkutz@mstdn.social
How (Not) to Measure Quality
By Michael Kutz
How (Not) to Measure Quality
Measuring quality is hard, defining what quality is is difficult, being aware of why you measure is fundamentally important. Learn how to choose and combine metrics to create something valuable.
- 7,917
