@MichaKutz

@mkutz@mstdn.social

Why Measure Quality?

improve quality

Goal:

Make better informed decisions about quality

What Could Possibly Go Wrong?

Goodhart's Law

When a measure becomes a target, it ceases to be a good measure.

How to Find Metrics?

Goal → Question → Metric

Are we going in the right direction?

Where are we?

Which Quality?

Code Coverage

@Test
void strike() {
var game = new Game();
var firstRollPins = 10;
var secondRollPins = 5;
var thirdRollPins = 3;
game.roll(firstRollPins);
game.roll(secondRollPins);
game.roll(thirdRollPins);

var score = game.score();

assertThat(score)
.isEqualTo(
firstRollPins +
(secondRollPins + thirdRollPins) * 2);
}
public int score() {
int score = firstRoll + secondRoll;
if (previous != null) {
if (previous.strike) {
score *= 2;
} else if (previous.spare) {
score += firstRoll;
}
}
return score;
}
@Test
void strike() {
var game = new Game();
var firstRollPins = 10;
var secondRollPins = 5;
var thirdRollPins = 3;
game.roll(firstRollPins);
game.roll(secondRollPins);
game.roll(thirdRollPins);

var score = game.score();

// assertThat(score)
//  .isEqualTo(
//    firstRollPins +
//    (secondRollPins + thirdRollPins) * 2);
}

Mutation Testing: Surviving Mutations

@Test
void strike() {
var game = new Game();
var firstRollPins = 10;
var secondRollPins = 5;
var thirdRollPins = 3;
game.roll(firstRollPins);
game.roll(secondRollPins);
game.roll(thirdRollPins);

var score = game.score();

assertThat(score
.isEqualTo(
firstRollPins +
(secondRollPins + thirdRollPins) * 2);
}
public int score() {
int score = firstRoll + secondRoll;
if (previous != null) {
if (previous.strike) {
score *= 2;
} else if (previous.spare) {
score += firstRoll;
}
}
return score;
}
AssertionFailedError:
expected: 26
but was: 14

Team Surveys

How confident is the team?

How effective can you work with the code?

How confident are you to deploy to production?

What would you need to improve the above answers?

Static Code Analysis: Complexity

public int score() {
return firstRoll + secondRoll;
}
public int score() {
int score = firstRoll + secondRoll;
if (previous != null && previous.spare) {
score += firstRoll;
}
return score;
}
public int score() {
int score = firstRoll + secondRoll;
if (previous != null) {
if (previous.strike) {
score *= 2;
} else if (previous.spare) {
score += firstRoll;
}
}
return score;
}
previous
.spare
score *= 2;
previous
.strike
score += firstRoll;
previous != null
  && previous.spare
int score = firstRoll
+ secondRoll;
return score;

Static Code Analysis: Code Smells

# Code Smells

- long methods,
- huge classes,
- many parameters,
- code duplicates,
- methods with complexity > 7,
- …

Velocity

v = \frac{P_{estimate}}{t_{delivery} - t_{commit1}}

How good are our estimations?

\Delta t_{delivery} = t_{delivery} - t_{commit1}

Change Fail Rate

08:07:23 Deploy basket-service ✓
09:06:11 Migrate checkout-service DB ✓
09:56:54 Deploy checkout-service ✓


08:07:23 Deploy basket-service ✓
09:06:11 Migrate checkout-service DB ✓
09:56:54 Deploy checkout-service ✓
10:19:44 Deploy order-managment-service ✗
10:39:27 Rollback order-managemet-service ✓


08:07:23 Deploy basket-service ✓
09:06:11 Migrate checkout-service DB ✓
09:56:54 Deploy checkout-service ✓
10:19:44 Deploy order-managment-service ✗
10:39:27 Rollback order-managemet-service ✓
11:09:59 Update database cluster ✓
12:27:32 Migrate order-managemet-service DB ✓
13:19:22 Deploy order-managemet-service ✓


08:07:23 Deploy basket-service ✓
09:06:11 Migrate checkout-service DB ✓
09:56:54 Deploy checkout-service ✓
10:19:44 Deploy order-managment-service ✗
10:39:27 Rollback order-managemet-service ✓
11:09:59 Update database cluster ✓
12:27:32 Migrate order-managemet-service DB ✓
13:19:22 Deploy order-managemet-service ✓
14:45:55 Update database-cluster ✗
15:50:49 Update database-cluster ✓


08:07:23 Deploy basket-service ✓
09:06:11 Migrate checkout-service DB ✓
09:56:54 Deploy checkout-service ✓
10:19:44 Deploy order-managment-service ✗
10:39:27 Rollback order-managemet-service ✓
11:09:59 Update database cluster ✓
12:27:32 Migrate order-managemet-service DB ✓
13:19:22 Deploy order-managemet-service ✓
14:45:55 Update database-cluster ✗
15:50:49 Update database-cluster ✓
16:39:11 Deploy product-service ✓
17:44:56 Deploy customer-data-service ✓

Mean Time to Restore Service

08:07:23 Deploy basket-service ✓
09:06:11 Migrate checkout-service DB ✓
09:56:54 Deploy checkout-service ✓
10:19:44 Deploy order-managment-service ✗
10:39:27 Rollback order-managemet-service ✓
11:09:59 Update database cluster ✓
12:27:32 Migrate order-managemet-service DB ✓
13:19:22 Deploy order-managemet-service ✓
14:45:55 Update database-cluster ✗
15:50:49 Update database-cluster ✓
16:39:11 Deploy product-service ✓
17:44:56 Deploy customer-data-service ✓

How fast?

How fast?

@MichaKutz

@mkutz@mstdn.social

Not everything that counts can be counted,
and not everything that can be counted counts.

By Michael Kutz

How (Not) to Measure Quality

Measuring quality is hard, defining what quality is is difficult, being aware of why you measure is fundamentally important. Learn how to choose and combine metrics to create something valuable.

• 6,257