How Go calculates code coverage

David Chou @ Golang Taiwan

CC-BY-SA-3.0-TW

@ Umbo Computer Vision 回家吃自己🏠

@ Golang Taiwan Co-organizer 🙋‍♂️

Software engineer, DevOps, and Gopher 👨‍💻

david74.chou @ gmail

david74.chou @ facebook

david74chou @ telegram


Blog: https://blog.david74.dev

Go's coverage-based fuzzing

Fuzzing is the process of sending intentionally invalid data to a product in the hopes of triggering an error.
- H.D. Moore

What is fuzzing test?

Fuzzing test

  • Start from a set of initial inputs

  • Continuously manipulate inputs

  • Semi-random input from various mutation

  • Discover new code coverage based on instrumentation

Go's official fuzzing solution

  • Official proposal [link]

  • Coverage-based fuzzing

  • Write fuzz function just like test function

    • func FuzzFoo(f *testing.F)

  • ​Integrate with  go command

    • go test -fuzz

  • Plan to land in 1.18

Code coverage in Go

go test -cover
go test -fuzz

A simple example

func CountAverage(num []byte) int {
	sum := byte(0)
	for _, v := range num {
		sum += v
	}
	return int(sum) / len(num)
}
func TestCountAverage(t *testing.T) {
	tests := []struct {
		name string
		num []byte
		want int
	}{
		{
			num: []byte{1, 2, 3, 4, 5},
			want: 3,
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			got := CountAverage(tt.num)
			assert.EqualValues(t, tt.want, got)
		})
	}
}
$ go test -cover
PASS
coverage: 100.0% of statements

Go test coverage

Coverage visualization

$ go test -coverprofile=coverage.out
$ go tool cover -html=coverage.out

Go tool cover

$ go tool cover
Usage of 'go tool cover':
Given a coverage profile produced by 'go test':
        go test -coverprofile=c.out

Open a web browser displaying annotated source code:
        go tool cover -html=c.out

Write out an HTML file instead of launching a web browser:
        go tool cover -html=c.out -o coverage.html

Display coverage percentages to stdout for each function:
        go tool cover -func=c.out

Finally, to generate modified source code with coverage annotations
(what go test -cover does):
        go tool cover -mode=set -var=CoverageVariableName program.go

package go_fuzzing_playground

func CountAverage(num []byte) int {GoCover.Count[0] = 1;
	sum := byte(0)
	for _, v := range num {GoCover.Count[2] = 1;
		sum += v
	}
	GoCover.Count[1] = 1;return int(sum) / len(num)
}

var GoCover = struct {
	Count     [3]uint32
	Pos       [3 * 3]uint32
	NumStmt   [3]uint16
} {
	Pos: [3 * 3]uint32{
		3, 5, 0x180023, // [0]
		8, 8, 0x1c0002, // [1]
		5, 7, 0x30018, // [2]
	},
	NumStmt: [3]uint16{
		2, // 0
		1, // 1
		1, // 2
	},
}

$ go tool cover -mode=set ./count_average.go

{StartLine, EndLine, ColumnInfo} [link]

Number of statements

package go_fuzzing_playground

func CountAverage(num []byte) int {GoCover.Count[0] = 1;
	sum := byte(0)
	for _, v := range num {GoCover.Count[2] = 1;
		sum += v
	}
	GoCover.Count[1] = 1;return int(sum) / len(num)
}

var GoCover = struct {
	Count     [3]uint32
	Pos       [3 * 3]uint32
	NumStmt   [3]uint16
} {
	Pos: [3 * 3]uint32{
		3, 5, 0x180023, // [0]
		8, 8, 0x1c0002, // [1]
		5, 7, 0x30018, // [2]
	},
	NumStmt: [3]uint16{
		2, // 0
		1, // 1
		1, // 2
	},
}

block0

block1

block2

$ go test ./ -coverprofile=coverage.out && cat coverage.out
ok            0.003s  coverage: 100.0% of statements
mode: set
count_average.go:3.35,5.24 2 1
count_average.go:8.2, 8.28 1 1
count_average.go:5.24,7.3  1 1

$ go test ./ -coverprofile=coverage.out && cat coverage.out

  • Coverage format:

    • name.go: Line.Column, Line.Column NumStmt Count
      [link]

  • 100% = (2x1 + 1x1 + 1x1) / (2 + 1 + 1)

Go test coverage

  • Source to source transform

  • Add instrument code before compiling

  • Basic block coverage

Basic block v.s. Branch coverage

block0

block1

block2

block0

block1

block2

Basic block: 100%

Branch coverage: 67%

package go_fuzzing_playground

func CountAverage(num []byte) int {GoCover.Count[0] = 1;
	sum := byte(0)
	for _, v := range num {GoCover.Count[2] = 1;
		sum += v
	}
	GoCover.Count[1] = 1;return int(sum) / len(num)
}

block0

block1

block2

How "go test -fuzz" works

Go fuzz coverage

Go fuzz coverage

  • Compiler instrumentation

  • Add instrument code during compiling

    • [1] and [2] by mdempsky

Compiler instrumentation

// edge inserts coverage instrumentation for libfuzzer.
func (o *orderState) edge() {
	// Create a new uint8 counter to be allocated in section
	// __libfuzzer_extra_counters.
	counter := staticinit.StaticName(types.Types[types.TUINT8])
	counter.SetLibfuzzerExtraCounter(true)

	// counter += 1
	incr := ir.NewAssignOpStmt(base.Pos, ir.OADD, counter, ir.NewInt(1))
	o.append(incr)
}

edge() inserts coverage instrumentation

func (o *orderState) stmt(n ir.Node) {
    switch n.Op() {
    ...
    case ir.OFOR:
        edge()
    case ir.OIF:
        edge()
    case ir.ORANGE:
        edge()
    case ir.OSELECT:
        edge()
    case ir.OSWITCH:
        edge()
    case OANDAND, OOROR:
        edge()
    ...
    }
}

compiler adds edge() into each edge

// _counters and _ecounters mark the start and end, respectively, of where
// the 8-bit coverage counters reside in memory. They're known to cmd/link,
// which specially assigns their addresses for this purpose.
var _counters, _ecounters [0]byte

func coverage() []byte {
	addr := unsafe.Pointer(&_counters)
	size := uintptr(unsafe.Pointer(&_ecounters)) - uintptr(addr)

	var res []byte
	*(*unsafeheader.Slice)(unsafe.Pointer(&res)) = unsafeheader.Slice{
		Data: addr,
		Len:  int(size),
		Cap:  int(size),
	}
	return res
}

coverage() returns the coverage counters

Go fuzz coverage

  • Still using basic block, but this could be improved in the future

  • Compared to source-to-source, it's much easier to implement branch coverage with compiler instrumentation

    • go-fuzz uses s2s and has lots of corner cases: miscompile, crash, invalid codes. E.g., [1], [2]

$ objdump go-fuzzing.test -h

go-fuzzing.test:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         0027890e  0000000000401000  0000000000401000  00001000  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
...
 19 .data         0000a550  0000000000883da0  0000000000883da0  00483da0  2**5
                  CONTENTS, ALLOC, LOAD, DATA
 20 .bss          00031708  000000000088e300  000000000088e300  0048e300  2**5
                  ALLOC
 21 .noptrbss     00006fc0  00000000008bfa20  00000000008bfa20  004bfa20  2**5
                  ALLOC
 22 __libfuzzer_extra_counters 000052a4  00000000008c69e0  00000000008c69e0  004c69e0  2**0
                  ALLOC

Bonus: libfuzzer_extra_counters

  • libfuzzer is a well-known LLVM fuzzy engine

  • It could take coverage number from libfuzzer_extra_counters variable

  • It means Go could use external fuzzy engine [link]

$ go build -gcflags=all=-d=fuzzing -buildmode=c-archive -o pngfuzz.a .
$ clang -o png.fuzzer pngfuzz.a -fsanitize=fuzzer

Bonus: libfuzzer_extra_counters

Compiler instrumentation

Source to source transform

  • Go test coverage

  • Add instrument code before compiling

  • Basic block coverage

  • Difficult to adopt branch coverage

  • Go fuzz coverage

  • Add instrument code during compiling

  • Basic block coverage

  • Easier to adopt branch coverage

  • Could expose code coverage during execution

Amazon CTO Dr. Werner Vogels

Made with Slides.com