How Go calculates code coverage
David Chou @ Golang Taiwan
CC-BY-SA-3.0-TW
@ Umbo Computer Vision 回家吃自己🏠
@ Golang Taiwan Co-organizer 🙋♂️
Software engineer, DevOps, and Gopher 👨💻
Go's coverage-based fuzzing
Fuzzing is the process of sending intentionally invalid data to a product in the hopes of triggering an error.
- H.D. Moore
What is fuzzing test?
Fuzzing test
-
Start from a set of initial inputs
-
Continuously manipulate inputs
-
Semi-random input from various mutation
-
Discover new code coverage based on instrumentation
Go's official fuzzing solution
-
Official proposal [link]
-
Coverage-based fuzzing
-
Write fuzz function just like test function
-
func FuzzFoo(f *testing.F)
-
-
Integrate with go command
-
go test -fuzz
-
-
Plan to land in 1.18
Code coverage in Go
go test -cover
go test -fuzz
A simple example
func CountAverage(num []byte) int {
sum := byte(0)
for _, v := range num {
sum += v
}
return int(sum) / len(num)
}func TestCountAverage(t *testing.T) {
tests := []struct {
name string
num []byte
want int
}{
{
num: []byte{1, 2, 3, 4, 5},
want: 3,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := CountAverage(tt.num)
assert.EqualValues(t, tt.want, got)
})
}
}$ go test -cover
PASS
coverage: 100.0% of statementsGo test coverage
Coverage visualization
$ go test -coverprofile=coverage.out
$ go tool cover -html=coverage.out
Go tool cover
$ go tool cover
Usage of 'go tool cover':
Given a coverage profile produced by 'go test':
go test -coverprofile=c.out
Open a web browser displaying annotated source code:
go tool cover -html=c.out
Write out an HTML file instead of launching a web browser:
go tool cover -html=c.out -o coverage.html
Display coverage percentages to stdout for each function:
go tool cover -func=c.out
Finally, to generate modified source code with coverage annotations
(what go test -cover does):
go tool cover -mode=set -var=CoverageVariableName program.go
package go_fuzzing_playground
func CountAverage(num []byte) int {GoCover.Count[0] = 1;
sum := byte(0)
for _, v := range num {GoCover.Count[2] = 1;
sum += v
}
GoCover.Count[1] = 1;return int(sum) / len(num)
}
var GoCover = struct {
Count [3]uint32
Pos [3 * 3]uint32
NumStmt [3]uint16
} {
Pos: [3 * 3]uint32{
3, 5, 0x180023, // [0]
8, 8, 0x1c0002, // [1]
5, 7, 0x30018, // [2]
},
NumStmt: [3]uint16{
2, // 0
1, // 1
1, // 2
},
}$ go tool cover -mode=set ./count_average.go
{StartLine, EndLine, ColumnInfo} [link]
Number of statements
package go_fuzzing_playground
func CountAverage(num []byte) int {GoCover.Count[0] = 1;
sum := byte(0)
for _, v := range num {GoCover.Count[2] = 1;
sum += v
}
GoCover.Count[1] = 1;return int(sum) / len(num)
}
var GoCover = struct {
Count [3]uint32
Pos [3 * 3]uint32
NumStmt [3]uint16
} {
Pos: [3 * 3]uint32{
3, 5, 0x180023, // [0]
8, 8, 0x1c0002, // [1]
5, 7, 0x30018, // [2]
},
NumStmt: [3]uint16{
2, // 0
1, // 1
1, // 2
},
}block0
block1
block2
$ go test ./ -coverprofile=coverage.out && cat coverage.out
ok 0.003s coverage: 100.0% of statements
mode: set
count_average.go:3.35,5.24 2 1
count_average.go:8.2, 8.28 1 1
count_average.go:5.24,7.3 1 1$ go test ./ -coverprofile=coverage.out && cat coverage.out
-
Coverage format:
-
name.go: Line.Column, Line.Column NumStmt Count
[link]
-
-
100% = (2x1 + 1x1 + 1x1) / (2 + 1 + 1)
Go test coverage
-
Source to source transform
-
Add instrument code before compiling
-
Basic block coverage
Basic block v.s. Branch coverage
block0
block1
block2
block0
block1
block2
Basic block: 100%
Branch coverage: 67%
package go_fuzzing_playground
func CountAverage(num []byte) int {GoCover.Count[0] = 1;
sum := byte(0)
for _, v := range num {GoCover.Count[2] = 1;
sum += v
}
GoCover.Count[1] = 1;return int(sum) / len(num)
}block0
block1
block2
How "go test -fuzz" works
Go fuzz coverage
Go fuzz coverage
Compiler instrumentation
// edge inserts coverage instrumentation for libfuzzer.
func (o *orderState) edge() {
// Create a new uint8 counter to be allocated in section
// __libfuzzer_extra_counters.
counter := staticinit.StaticName(types.Types[types.TUINT8])
counter.SetLibfuzzerExtraCounter(true)
// counter += 1
incr := ir.NewAssignOpStmt(base.Pos, ir.OADD, counter, ir.NewInt(1))
o.append(incr)
}edge() inserts coverage instrumentation
func (o *orderState) stmt(n ir.Node) {
switch n.Op() {
...
case ir.OFOR:
edge()
case ir.OIF:
edge()
case ir.ORANGE:
edge()
case ir.OSELECT:
edge()
case ir.OSWITCH:
edge()
case OANDAND, OOROR:
edge()
...
}
}compiler adds edge() into each edge
// _counters and _ecounters mark the start and end, respectively, of where
// the 8-bit coverage counters reside in memory. They're known to cmd/link,
// which specially assigns their addresses for this purpose.
var _counters, _ecounters [0]byte
func coverage() []byte {
addr := unsafe.Pointer(&_counters)
size := uintptr(unsafe.Pointer(&_ecounters)) - uintptr(addr)
var res []byte
*(*unsafeheader.Slice)(unsafe.Pointer(&res)) = unsafeheader.Slice{
Data: addr,
Len: int(size),
Cap: int(size),
}
return res
}coverage() returns the coverage counters
Go fuzz coverage
$ objdump go-fuzzing.test -h
go-fuzzing.test: file format elf64-x86-64
Sections:
Idx Name Size VMA LMA File off Algn
0 .text 0027890e 0000000000401000 0000000000401000 00001000 2**5
CONTENTS, ALLOC, LOAD, READONLY, CODE
...
19 .data 0000a550 0000000000883da0 0000000000883da0 00483da0 2**5
CONTENTS, ALLOC, LOAD, DATA
20 .bss 00031708 000000000088e300 000000000088e300 0048e300 2**5
ALLOC
21 .noptrbss 00006fc0 00000000008bfa20 00000000008bfa20 004bfa20 2**5
ALLOC
22 __libfuzzer_extra_counters 000052a4 00000000008c69e0 00000000008c69e0 004c69e0 2**0
ALLOC
Bonus: libfuzzer_extra_counters
-
libfuzzer is a well-known LLVM fuzzy engine
-
It could take coverage number from libfuzzer_extra_counters variable
-
It means Go could use external fuzzy engine [link]
$ go build -gcflags=all=-d=fuzzing -buildmode=c-archive -o pngfuzz.a .
$ clang -o png.fuzzer pngfuzz.a -fsanitize=fuzzerBonus: libfuzzer_extra_counters
Compiler instrumentation
Source to source transform
-
Go test coverage
-
Add instrument code before compiling
-
Basic block coverage
-
Difficult to adopt branch coverage
-
Go fuzz coverage
-
Add instrument code during compiling
-
Basic block coverage
-
Easier to adopt branch coverage
-
Could expose code coverage during execution
Amazon CTO Dr. Werner Vogels
How Go calculates code coverage
By Ting-Li Chou
