Debugging Go in Kubernetes

Debugging Go programs
running in a Kubernetes Pod
using Delve with GoLand

by Mike Schinkel on May 4th, 2023

github.com/mikeschinkel

Or more accurately:

About Me

25+ years involved with dev as

A developer,
A trainer of devs, and
A startup CEO selling to devs.

Contractor with these teams:

SingleStore K8s Operator
HPE Greenlake On-Prem
Numerous lesser-known others

Currently:

Free-agent dev: Go/K8s/CI-CD
New YouTuber —
"Homelabbing for Developers"

youtube.com/@gearboxworks

Go programming experience
Familiar with K8s YAML
Can read Dockerfiles
Can read Bash scripts
Knowledge of Makefiles

Prerequisites & Assumptions

Find the code
on Github

github.com/mikeschinkel/go-debuggable-k3d-pod

Open-source tools we'll use

K3s — k3s.io

A lightweight distro of K8s

1.

2.

k3d — k3d.io

Runs a K3s cluster in a Docker container

3.

Delve — github.com/go-delve

The defacto-standard debugger for Go

Source for our Go "App"

package main

import (
	"fmt"
	"time"
)

const (
	DateFormat = "2006-01-02 3:04:05PM"
	DelayTime  = 5
)

// Debug with Go Remote = localhost:8765
func main() {
	fmt.Println("Starting Debuggable Pod")
	for {
		dt := time.Now().Format(DateFormat)
		fmt.Printf("\n[%s] Hello World!", dt)
		time.Sleep(DelayTime * time.Second)
	}
}

# PRESENTING CODE

The go.mod

module debuggable-pod

go 1.19

# PRESENTING CODE

The k3d.yaml

---
apiVersion: k3d.io/v1alpha4
kind: Simple
servers: 1
kubeAPI:
  hostPort: "6443"
image: rancher/k3s:v1.23.8-k3s1
registries:
  use:
    - k3d-registry.localhost:5000
options:
  k3d:
    wait: true
    timeout: "60s"
ports:
  - port: 8765:30800
    nodeFilters:
      - server:0

# PRESENTING CODE

The pod.yaml — 1st part

---
apiVersion: v1
kind: Pod
metadata:
  name: debuggable-pod
  labels:
    debugger: dlv
  annotations:
    container.apparmor.security.beta.kubernetes.io/debuggable-app-container: unconfined
spec:
  restartPolicy: Always
  containers:
  - name: debuggable-app-container
    image: k3d-registry.localhost:5000/debuggable-app-image
    imagePullPolicy: Always
    securityContext:
      capabilities:
        add:
        - SYS_PTRACE
    ports:
    - containerPort: 32345
      hostPort: 32345
      protocol: TCP

# PRESENTING CODE

The pod.yaml — 2nd part

---
apiVersion: v1
kind: Service
metadata:
  name: dlv-port-service
spec:
  type: NodePort
  selector:
    debugger: dlv
  ports:
  - name: dlv-port
    protocol: TCP
    port: 32345
    nodePort: 30800

# PRESENTING CODE

The Dockerfile for the Pod

FROM golang:1.19-alpine AS builder

ENV CGO_ENABLED=0

RUN apk update \
    && apk add --no-cache git \
    && go install github.com/go-delve/delve/cmd/dlv@v1.9.1

WORKDIR /app

COPY . /app

RUN go build -o debuggable-go-app -gcflags="all=-N -l" /app/main.go

EXPOSE 32345 32345

CMD [ "dlv", \
    "--listen=:32345", \
    "--headless=true", \
    "--api-version=2", \
    "--accept-multiclient", \
    "exec", "/app/debuggable-go-app"]

# PRESENTING CODE

Components/Ports

Preparing the Example

git clone https://github.com/mikeschinkel/go-debuggable-k3d-pod

make install  # Installs required software

make init     # Initializes k3d registry and cluster

make build    # Builds the Go App and Docker container

make deploy   # Push container to K8s registry, apply Pod YAML

# PRESENTING CODE

Demo #1: Commands to Try

# DEMONSTRATION

Commands	Notice
docker ps	Images, ports for proxy
kubectl get all	Names, service ports
kubectl exec -it pod/debuggable-pod -- sh
top	PID 1's command
ls -al	files
kubectl logs pods/debuggable-pod	Output

Demo #2: Debugging in the IDE

# DEMONSTRATION

What did we learn? Ports!

Component	Port
The K8s Cluster	8765
The K8s Pod	30800
The Go App	32345

You can use any available port numbers here, they just need to be used in the right places.

Questions?

github.com/mikeschinkel/go-debuggable-k3d-pod

(Again, find the code here:)

Thank you!

Who am I, again?	Mike Schinkel
Code	github.com/mikeschinkel/go-debuggable-k3d-pod
These Slides	slides.com/mikeschinkel/debugging-go-apps-in-k8s
YouTube channel (Coming soon)	www.youtube.com/@gearboxworks
Available for contract work	mike@newclarity.net
My LinkedIn	www.linkedin.com/in/mikeschinkel