Web APIs in Go

Rainer Stropek | @rstropek

Introduction

Rainer Stropek

Passionate developer since 25+ years
Microsoft MVP, Regional Director
Trainer, Teacher, Mentor
💕 community

https://rainerstropek.me

Getting Started

Setup Environment

Install latest version of Go
- go version
Latest version of Visual Studio Code
- You can use different IDE if you want
- Training will be shown with VSCode
Up-to-date web browser
VSCode extensions

Setup folders

Create empty folder hero-manager
Initialize module
- go mod init heroes.<yourname>.com
- Read more about modules
Create project structure
- mkdir -p bin cmd/api internal migrations www
  touch cmd/api/main.go
- Read more about project layout of larger projects
Open folder in VSCode (code .)
Add a "Hello World" project to cmd/api/main.go and try it
- go run ./cmd/api
- Try also debugging in VSCode

requests.http

@host=http://localhost:4000

###
GET {{host}}/v1/healthcheck

###
POST {{host}}/v1/heroes

{
    "name": "Superman",
    "firstSeen": "1935-01-01T00:00:00Z",
    "canFly": true,
    "realName": "Clark Kent",
    "abilities": [ "super strong", "can disguise with glasses" ]
}

###
GET {{host}}/v1/heroes/1

###
# Test error handling
GET {{host}}/v1/somethingThatDoesNotExist

###
# Test error handling
POST {{host}}/v1/healthcheck

###
POST {{host}}/v1/generate
###

###
PUT {{host}}/v1/heroes/1

{
    "name": "Homelander",
    "firstSeen": "2020-01-01T00:00:00Z",
    "canFly": true,
    "abilities": [ "super strong" ]
}

###
DELETE {{host}}/v1/heroes/1

###
GET {{host}}/v1/heroes?name=Or&abilities=foo&page=1&page_size=3&sort=name

###
GET {{host}}/v1/heroes?page=2&page_size=3&sort=name

Basic Web Server

main.go

package main

import (
    "flag"
    "fmt"
    "log"
    "net/http"
    "os"
    "time"
)

const version = "1.0.0"

type config struct {
    port int
    env  string
}

type application struct {
    config config
    logger *log.Logger
}

func main() {
    var cfg config

    flag.IntVar(&cfg.port, "port", 4000, "API server port")
    flag.StringVar(&cfg.env, "env", "development", "Environment (development|staging|production)")
    flag.Parse()

    logger := log.New(os.Stdout, "", log.Ldate | log.Ltime)

    app := &application{
        config: cfg,
        logger: logger,
    }

    mux := http.NewServeMux()
    mux.HandleFunc("/v1/healthcheck", app.healthcheckHandler)

	srv := &http.Server{
        Addr:         fmt.Sprintf(":%d", cfg.port),
        Handler:      mux,
        IdleTimeout:  time.Minute,
        ReadTimeout:  10 * time.Second,
        WriteTimeout: 30 * time.Second,
    }

    logger.Printf("starting %s server on %s", cfg.env, srv.Addr)
    err := srv.ListenAndServe()
    logger.Fatal(err)
}

healthcheck.go

package main

import (
    "fmt"
    "net/http"
)

// Declare a handler which writes a plain-text response with information about the 
// application status, operating environment and version.
func (app *application) healthcheckHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "status: available")
    fmt.Fprintf(w, "environment: %s\n", app.config.env)
    fmt.Fprintf(w, "version: %s\n", version)
}

Notes

Command-line flags for passing config parameters
- We use default flag package
- Try go run ./cmd/api -help
- Many different modules 🔗 (e.g. viper)
- Consider config files and/or env variables (os.Getenv("...")) 🔗
Handler functions are similar to controllers
- Executing app logic
- Write HTTP responses
Router (aka servemux) maps URL patterns to handlers
- Many different routers 🔗 (e.g. httprouter)
- We start with the router from Go std lib http.ServeMux
  - not powerful, but no deps

Notes

Use application struct as a hub for all app features
- Many different packages 🔗 (e.g. wire)
Web server built-in in Go
- No need for external server like Nginx or Apache
Logging
- We use default log package (simple, but no deps)
- Many different modules 🔗 (e.g. logrus)
Discussion
- Approaches for API versioning

http.Handler

Handler with ServeHTTP method
Handler function with same signature
Note: Requests are handled concurrently
- All incoming HTTP requests are served in their own goroutine

type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

API Routes

Switch Router

Add httprouter
- Popular, fast, I am familiar with it
- Note config options 🔗
- By far not the only option! 🔗
- go get github.com/julienschmidt/httprouter
Add routes
- touch cmd/api/routes.go
  touch cmd/api/heroes.go
  touch cmd/api/helpers.go

routes.go

package main

import (
    "net/http"

    "github.com/julienschmidt/httprouter"
)

func (app *application) routes() *httprouter.Router {
    router := httprouter.New()

    router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)
    router.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
    router.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)

    // Return the httprouter instance.
    return router
}

Updated main.go

func main() {
    var cfg config

    flag.IntVar(&cfg.port, "port", 4000, "API server port")
    flag.StringVar(&cfg.env, "env", "development", "Environment (development|staging|production)")
    flag.Parse()

    logger := log.New(os.Stdout, "", log.Ldate|log.Ltime)

    app := &application{
        config: cfg,
        logger: logger,
    }

    srv := &http.Server{
        Addr:         fmt.Sprintf(":%d", cfg.port),
        Handler:      app.routes(),
        IdleTimeout:  time.Minute,
        ReadTimeout:  10 * time.Second,
        WriteTimeout: 30 * time.Second,
    }

    logger.Printf("starting %s server on %s", cfg.env, srv.Addr)
    err := srv.ListenAndServe()
    logger.Fatal(err)
}

heroes.go

package main

import (
    "fmt"
    "net/http"
    "strconv" 

    "github.com/julienschmidt/httprouter" 
)

func (app *application) createHeroHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "create a new hero")
}

func (app *application) showHeroHandler(w http.ResponseWriter, r *http.Request) {
    params := httprouter.ParamsFromContext(r.Context())

    id, err := strconv.ParseInt(params.ByName("id"), 10, 64)
    if err != nil || id < 1 {
        http.NotFound(w, r)
        return
    }

    fmt.Fprintf(w, "show the details of hero %d\n", id)
}

Test

go run ./cmd/api
Try test requests

@host=http://localhost:4000

###
GET {{host}}/v1/healthcheck

###
POST {{host}}/v1/heroes

###
GET {{host}}/v1/heroes/42

helpers.go

package main

import (
    "errors"
    "net/http"
    "strconv"

    "github.com/julienschmidt/httprouter"
)

func (app *application) readIDParam(r *http.Request) (int64, error) {
    params := httprouter.ParamsFromContext(r.Context())

    id, err := strconv.ParseInt(params.ByName("id"), 10, 64)
    if err != nil || id < 1 {
        return 0, errors.New("invalid id parameter")
    }

    return id, nil
}

Updated heroes.go

func (app *application) showHeroHandler(w http.ResponseWriter, r *http.Request) {
	id, err := app.readIDParam(r)
	if err != nil {
		http.NotFound(w, r)
		return
	}

	fmt.Fprintf(w, "show the details of hero %d\n", id)
}

Sending JSON

Updated healthcheck.go

package main

import (
    "fmt"
    "net/http"
)

func (app *application) healthcheckHandler(w http.ResponseWriter, r *http.Request) {
    // Fixed-Format JSON
    js := `{"status": "available", "environment": %q, "version": %q}`
    js = fmt.Sprintf(js, app.config.env, version)

    w.Header().Set("Content-Type", "application/json")

    w.Write([]byte(js))
}

Generating JSON

Options
- Call json.Marshal() (done in this example)
- Use a json.Encoder
Note and discuss: Empty interface
Good read: The Ultimate Guide to JSON in Go
Update project
- mkdir internal/data
  touch internal/data/heroes.go

func Marshal(v interface{}) ([]byte, error)

Updated healthcheck.go

package main

import (
    "encoding/json"
    "net/http"
)

func (app *application) healthcheckHandler(w http.ResponseWriter, r *http.Request) {
    data := map[string]string{
        "status":      "available",
        "environment": app.config.env,
        "version":     version,
    }

    js, err := json.Marshal(data)
    if err != nil {
        app.logger.Println(err)
        http.Error(w, "The server encountered a problem and could not process your request", http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "application/json")

    w.Write(js)
}

Updated helpers.go

func (app *application) writeJSON(w http.ResponseWriter, status int, data interface{}, headers http.Header) error {
	js, err := json.Marshal(data)
	if err != nil {
		return err
	}

	for key, value := range headers {
		w.Header()[key] = value
	}

	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(status)
	w.Write(js)

	return nil
}

Updated healthcheck.go

func (app *application) healthcheckHandler(w http.ResponseWriter, r *http.Request) {
	data := map[string]string{
		"status":      "available",
		"environment": app.config.env,
		"version":     version,
	}

	err := app.writeJSON(w, http.StatusOK, data, nil)
	if err != nil {
		app.logger.Println(err)
		http.Error(w, "The server encountered a problem and could not process your request", http.StatusInternalServerError)
	}
}

internal/data/heroes.go

package data

import (
	"time"
)

// Note: All members are exported (i.e. start with capital letter),
//       required for JSON marshalling

type Hero struct {
	ID        int64     // Unique integer ID for the hero
	FirstSeen time.Time // Timestamp for when the hero was first seen
	Name      string    // Hero Name
	CanFly    bool      // Indicates whether the hero can fly
	RealName  string    // Real-world name of the hero (e.g. Clark Kent)
	Abilities []string  // Slice of abilities for the hero (super strength, laser beams from eyes, etc.)
	Version   int32     // The version number starts at 1 and will be incremented each time the hero information is updated
}

Updated heroes.go

func (app *application) showHeroHandler(w http.ResponseWriter, r *http.Request) {
	id, err := app.readIDParam(r)
	if err != nil {
		http.NotFound(w, r)
		return
	}

	hero := data.Hero{
		ID:        id,
		Name:      "Superman",
		FirstSeen: time.Date(1935, time.Month(1), 1, 0, 0, 0, 0, time.UTC),
		RealName:  "Clark Kent",
		CanFly:    true,
		Abilities: []string{"super strong", "can disguise with glasses"},
		Version:   1,
	}

	err = app.writeJSON(w, http.StatusOK, hero, nil)
	if err != nil {
		app.logger.Println(err)
		http.Error(w, "The server encountered a problem and could not process your request", http.StatusInternalServerError)
	}
}

Configure JSON

Use struct tags
- Note omitempty
- Note hyphen (-)
Updated heroes.go
Note: Possible to do custom JSON serialization by providing MarshalJSON method
- Example: time.Time is a struct, but marshalled to RFC3339

type Hero struct {
	ID        int64     `json:"id"`
	FirstSeen time.Time `json:"firstSeen"`
	Name      string    `json:"name"`
	CanFly    bool      `json:"canFly"`
	RealName  string    `json:"realName,omitempty"`
	Abilities []string  `json:"abilities,omitempty"`
	Version   int32     `json:"version"`
}

Updated internal/data/heroes.go

type Hero struct {
	ID        int64     `json:"id"`
	FirstSeen time.Time `json:"firstSeen"`
	Name      string    `json:"name"`
	CanFly    bool      `json:"canFly"`
	RealName  string    `json:"realName,omitempty"`
	Abilities []string  `json:"-"`
	Version   int32     `json:"version"`
}

func (h Hero) MarshalJSON() ([]byte, error) {
	var abilities string

	if h.Abilities != nil {
		abilities = strings.Join(h.Abilities, ", ")
	}

	type HeroAlias Hero

	aux := struct {
		HeroAlias
		Abilities string `json:"abilities,omitempty"`
	}{
		HeroAlias: HeroAlias(h),
		Abilities: abilities,
	}

	return json.Marshal(aux)
}

Error Messages

touch cmd/api/errors.go

errors.go

package main

import (
    "fmt"
    "net/http"
)

func (app *application) logError(r *http.Request, err error) {
    app.logger.Println(err)
}

func (app *application) errorResponse(w http.ResponseWriter, r *http.Request, status int, message interface{}) {
    err := app.writeJSON(w, status, message, nil)
    if err != nil {
        app.logError(r, err)
        w.WriteHeader(500)
    }
}

func (app *application) serverErrorResponse(w http.ResponseWriter, r *http.Request, err error) {
    app.logError(r, err)

    message := "the server encountered a problem and could not process your request"
    app.errorResponse(w, r, http.StatusInternalServerError, message)
}

func (app *application) notFoundResponse(w http.ResponseWriter, r *http.Request) {
    message := "the requested resource could not be found"
    app.errorResponse(w, r, http.StatusNotFound, message)
}

func (app *application) methodNotAllowedResponse(w http.ResponseWriter, r *http.Request) {
    message := fmt.Sprintf("the %s method is not supported for this resource", r.Method)
    app.errorResponse(w, r, http.StatusMethodNotAllowed, message)
}

func (app *application) badRequestResponse(w http.ResponseWriter, r *http.Request, err error) {
    app.errorResponse(w, r, http.StatusBadRequest, err.Error())
}

func (app *application) failedValidationResponse(w http.ResponseWriter, r *http.Request, errors map[string]string) {
	app.errorResponse(w, r, http.StatusUnprocessableEntity, errors)
}

Updated healthcheck.go

func (app *application) healthcheckHandler(w http.ResponseWriter, r *http.Request) {
	data := map[string]string{
		"status":      "available",
		"environment": app.config.env,
		"version":     version,
	}

	err := app.writeJSON(w, http.StatusOK, data, nil)
	if err != nil {
		app.serverErrorResponse(w, r, err)
	}
}

Updated heroes.go

func (app *application) showHeroHandler(w http.ResponseWriter, r *http.Request) {
	id, err := app.readIDParam(r)
	if err != nil {
		http.NotFound(w, r)
		return
	}

	hero := data.Hero{
		ID:        id,
		Name:      "Superman",
		FirstSeen: time.Date(1935, time.Month(1), 1, 0, 0, 0, 0, time.UTC),
		RealName:  "Clark Kent",
		CanFly:    true,
		Abilities: []string{"super strong", "can disguise with glasses"},
		Version:   1,
	}

	err = app.writeJSON(w, http.StatusOK, hero, nil)
	if err != nil {
		app.serverErrorResponse(w, r, err)
	}
}

Updated routes.go

func (app *application) routes() *httprouter.Router {
	router := httprouter.New()

	router.NotFound = http.HandlerFunc(app.notFoundResponse)
	router.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)

	router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)
	router.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)

	// Return the httprouter instance.
	return router
}

When to panic()?

Go by example:
A panic typically means something went unexpectedly wrong. Mostly we use it to fail fast on errors that shouldn’t occur during normal operation and that we aren’t prepared to handle gracefully.

Parsing JSON

Options
- Call json.Unmarshal()
- Use a json.Decoder (done in this example)
Again: Don't forget to export members (starting with capital letter)
Note: Possible to do custom JSON serialization by providing UnmarshalJSON method

type Unmarshaler interface {
    UnmarshalJSON([]byte) error
}

Updated heroes.go

func (app *application) createHeroHandler(w http.ResponseWriter, r *http.Request) {
	var input struct {
		Name      string    `json:"name"`
		FirstSeen time.Time `json:"firstSeen"`
		CanFly    bool      `json:"canFly"`
		RealName  string    `json:"realName,omitempty"`
		Abilities []string  `json:"abilities"`
	}

	err := json.NewDecoder(r.Body).Decode(&input)
	if err != nil {
		app.errorResponse(w, r, http.StatusBadRequest, err.Error())
		return
	}

	fmt.Fprintf(w, "%+v\n", input)
}

Updated helpers.go

func (app *application) readJSON(w http.ResponseWriter, r *http.Request, dst interface{}) error {
	// Limit the size of the request body to 1MB
	maxBytes := 1_048_576
	r.Body = http.MaxBytesReader(w, r.Body, int64(maxBytes))

	dec := json.NewDecoder(r.Body)
	dec.DisallowUnknownFields() // Remove if not wanted

	err := dec.Decode(dst)
	if err != nil {
		var syntaxError *json.SyntaxError
		var unmarshalTypeError *json.UnmarshalTypeError
		var invalidUnmarshalError *json.InvalidUnmarshalError

		switch {
		// Return a plain-english error message
		case errors.As(err, &syntaxError):
			return fmt.Errorf("body contains badly-formed JSON (at character %d)", syntaxError.Offset)

		case errors.Is(err, io.ErrUnexpectedEOF):
			return errors.New("body contains badly-formed JSON")

		// JSON value is the wrong type for the target destination.
		case errors.As(err, &unmarshalTypeError):
			if unmarshalTypeError.Field != "" {
				return fmt.Errorf("body contains incorrect JSON type for field %q", unmarshalTypeError.Field)
			}
			return fmt.Errorf("body contains incorrect JSON type (at character %d)", unmarshalTypeError.Offset)

		// Request body is empty
		case errors.Is(err, io.EOF):
			return errors.New("body must not be empty")

		// JSON contains a field which cannot be mapped to the target destination
		case strings.HasPrefix(err.Error(), "json: unknown field "):
			fieldName := strings.TrimPrefix(err.Error(), "json: unknown field ")
			return fmt.Errorf("body contains unknown key %s", fieldName)

		case err.Error() == "http: request body too large":
			return fmt.Errorf("body must not be larger than %d bytes", maxBytes)

		case errors.As(err, &invalidUnmarshalError):
			panic(err) // This should NEVER happen

		// Return the error message as-is.
		default:
			return err
		}
	}

	// If the request body only contained a single JSON value this will
    // return an io.EOF error. So if we get anything else, we know that there is
    // additional data in the request body and we return our own custom error message.
    err = dec.Decode(&struct{}{})
    if err != io.EOF {
        return errors.New("body must only contain a single JSON value")
    }

	return nil
}

Updated heroes.go

func (app *application) createHeroHandler(w http.ResponseWriter, r *http.Request) {
	var input struct {
		Name      string    `json:"name"`
		FirstSeen time.Time `json:"firstSeen"`
		CanFly    bool      `json:"canFly"`
		RealName  string    `json:"realName,omitempty"`
		Abilities []string  `json:"abilities"`
	}

	err := app.readJSON(w, r, &input)
	if err != nil {
		app.badRequestResponse(w, r, err) 
		return
	}

	fmt.Fprintf(w, "%+v\n", input)
}

Validating Input

Validating

Many different packages 🔗
For training purposes, we are going to build a small validator helper on our own
- mkdir internal/validator
  touch internal/validator/validator.go

validator.go

package validator

import (
	"regexp"
)

var (
	EmailRX = regexp.MustCompile("^[a-zA-Z0-9.!#$%&'*+\\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$")
)

type Validator struct {
	Errors map[string]string
}

func New() *Validator {
	return &Validator{Errors: make(map[string]string)}
}

func (v *Validator) Valid() bool {
	return len(v.Errors) == 0
}

func (v *Validator) AddError(key, message string) {
	if _, exists := v.Errors[key]; !exists {
		v.Errors[key] = message
	}
}

func (v *Validator) Check(ok bool, key, message string) {
	if !ok {
		v.AddError(key, message)
	}
}

func In(value string, list ...string) bool {
	for i := range list {
		if value == list[i] {
			return true
		}
	}
	return false
}

func Matches(value string, rx *regexp.Regexp) bool {
	return rx.MatchString(value)
}

func Unique(values []string) bool {
	uniqueValues := make(map[string]bool)

	for _, value := range values {
		uniqueValues[value] = true
	}

	return len(values) == len(uniqueValues)
}

Updated heroes.go

func (app *application) createHeroHandler(w http.ResponseWriter, r *http.Request) {
	var input struct {
		Name      string    `json:"name"`
		FirstSeen time.Time `json:"firstSeen"`
		CanFly    bool      `json:"canFly"`
		RealName  string    `json:"realName,omitempty"`
		Abilities []string  `json:"abilities"`
	}

	err := app.readJSON(w, r, &input)
	if err != nil {
		app.badRequestResponse(w, r, err)
		return
	}

	v := validator.New()

	v.Check(input.Name != "", "name", "must be provided")
	v.Check(len(input.Name) <= 100, "name", "must not be more than 100 bytes long")

	v.Check(input.Abilities != nil, "abilities", "must be provided")
	v.Check(len(input.Abilities) >= 1, "abilities", "must contain at least 1 ability")
	v.Check(len(input.Abilities) <= 5, "abilities", "must not contain more than 5 abilities")
	v.Check(validator.Unique(input.Abilities), "abilities", "must not contain duplicate values")

	if !v.Valid() {
		app.failedValidationResponse(w, r, v.Errors)
		return
	}

	fmt.Fprintf(w, "%+v\n", input)
}

Updated internal/data/heroes.go

func ValidateHero(v *validator.Validator, hero *Hero) {
	v.Check(hero.Name != "", "name", "must be provided")
	v.Check(len(hero.Name) <= 100, "name", "must not be more than 100 bytes long")

	v.Check(hero.Abilities != nil, "abilities", "must be provided")
	v.Check(len(hero.Abilities) >= 1, "abilities", "must contain at least 1 ability")
	v.Check(len(hero.Abilities) <= 5, "abilities", "must not contain more than 5 abilities")
	v.Check(validator.Unique(hero.Abilities), "abilities", "must not contain duplicate values")
}

Updated heroes.go

func (app *application) createHeroHandler(w http.ResponseWriter, r *http.Request) {
	var input struct {
		Name      string    `json:"name"`
		FirstSeen time.Time `json:"firstSeen"`
		CanFly    bool      `json:"canFly"`
		RealName  string    `json:"realName,omitempty"`
		Abilities []string  `json:"abilities"`
	}

	err := app.readJSON(w, r, &input)
	if err != nil {
		app.badRequestResponse(w, r, err)
		return
	}

	hero := &data.Hero{
		Name:      input.Name,
		FirstSeen: input.FirstSeen,
		CanFly:    input.CanFly,
		RealName:  input.RealName,
		Abilities: input.Abilities,
	}

	v := validator.New()

	if data.ValidateHero(v, hero); !v.Valid() {
		app.failedValidationResponse(w, r, v.Errors)
		return
	}

	fmt.Fprintf(w, "%+v\n", input)
}

Setting up the Database

Setup Postgres

We are going to use PostgreSQL here
- database/sql package with Postgres pg driver
Options
- Install it locally 🔗
- Run it in Docker
- go get github.com/lib/pq

docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres
docker exec -it some-postgres /bin/bash
	psql -U postgres
    SELECT current_user;
    CREATE DATABASE heroes;
    \c heroes
    CREATE ROLE heroes WITH LOGIN PASSWORD 'mysecretpassword';
    CREATE EXTENSION IF NOT EXISTS citext;
    \q
docker exec -it some-postgres /bin/bash
	psql --host=localhost --dbname=heroes --username=heroes
    SELECT current_user;
export HEROES_DB_DSN='postgres://heroes:mysecretpassword@localhost/heroes?sslmode=disable'

Setup Postgres

We use a connection pool
- See sql.Open
- Auto-maintenance of idle and in-use connections
- Graceful error handling
- Many configuration options 🔗

What about ORMs?

Many options available 🔗
- Example: GORM

Updated main.go

package main

import (
	"context"
	"database/sql"

	"flag"
	"fmt"
	"log"
	"net/http"
	"os"
	"time"

	_ "github.com/lib/pq"
)

const version = "1.0.0"

type config struct {
	port int
	env  string
	db   struct {
		dsn          string
		maxOpenConns int
		maxIdleConns int
		maxIdleTime  string
	}
}

type application struct {
	config config
	logger *log.Logger
}

func main() {
	var cfg config

	flag.IntVar(&cfg.port, "port", 4000, "API server port")
	flag.StringVar(&cfg.env, "env", "development", "Environment (development|staging|production)")
	flag.StringVar(&cfg.db.dsn, "db-dsn", os.Getenv("HEROES_DB_DSN"), "PostgreSQL DSN")
	flag.IntVar(&cfg.db.maxOpenConns, "db-max-open-conns", 25, "PostgreSQL max open connections")
	flag.IntVar(&cfg.db.maxIdleConns, "db-max-idle-conns", 25, "PostgreSQL max idle connections")
	flag.StringVar(&cfg.db.maxIdleTime, "db-max-idle-time", "15m", "PostgreSQL max connection idle time")
	flag.Parse()

	logger := log.New(os.Stdout, "", log.Ldate|log.Ltime)

	db, err := openDB(cfg)
	if err != nil {
		logger.Fatal(err)
	}

	defer db.Close()
	logger.Printf("database connection pool established")

	app := &application{
		config: cfg,
		logger: logger,
	}

	srv := &http.Server{
		Addr:         fmt.Sprintf(":%d", cfg.port),
		Handler:      app.routes(),
		IdleTimeout:  time.Minute,
		ReadTimeout:  10 * time.Second,
		WriteTimeout: 30 * time.Second,
	}

	logger.Printf("starting %s server on %s", cfg.env, srv.Addr)
	err = srv.ListenAndServe()
	logger.Fatal(err)
}

func openDB(cfg config) (*sql.DB, error) {
	db, err := sql.Open("postgres", cfg.db.dsn)
	if err != nil {
		return nil, err
	}

	db.SetMaxOpenConns(cfg.db.maxOpenConns)
	db.SetMaxIdleConns(cfg.db.maxIdleConns)
	duration, err := time.ParseDuration(cfg.db.maxIdleTime)
	if err != nil {
		return nil, err
	}

	db.SetConnMaxIdleTime(duration)

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	err = db.PingContext(ctx)
	if err != nil {
		return nil, err
	}

	return db, nil
}

Migrations

Install migrate

Download current relese
Create migration (see SQL scripts following slide)
Execute migrations
Note that migrations files can come from remote sources
Discuss: Migrate on startup? 🔗

curl -L https://github.com/golang-migrate/migrate/releases/download/v4.15.1/migrate.linux-arm64.tar.gz | tar xvz
sudo mv migrate $GOPATH/bin/migrate
migrate -version

migrate create -seq -ext=.sql -dir=./migrations create_heroes_table
migrate create -seq -ext=.sql -dir=./migrations add_heroes_check_constraints

migrate -path=./migrations -database=$HEROES_DB_DSN up

SQL Scripts

# UP
CREATE TABLE IF NOT EXISTS heroes (
    id bigserial PRIMARY KEY,  
    first_seen timestamp(0) NOT NULL DEFAULT NOW(),
    name text NOT NULL,
    can_fly boolean NOT NULL DEFAULT false,
    realName text NULL,
    abilities text[] NOT NULL,
    version integer NOT NULL DEFAULT 1
);

###
ALTER TABLE heroes ADD CONSTRAINT abilities_length_check CHECK (array_length(abilities, 1) BETWEEN 1 AND 5);


# DOWN
DROP TABLE IF EXISTS heroes;

###
ALTER TABLE heroes DROP CONSTRAINT abilities_length_check;

Work with Migrations

See tables in Postgres
Fix error

SELECT * FROM schema_migrations;
\d heroes

migrate -path=./migrations -database=$HEROES_DB_DSN force 1

CRUD

touch internal/data/models.go

Updated internal/data/heroes.go

type HeroModel struct {
	DB *sql.DB
}

func (m HeroModel) Insert(hero *Hero) error {
	return nil
}

func (m HeroModel) Get(id int64) (*Hero, error) {
	return nil, nil
}

func (m HeroModel) Update(hero *Hero) error {
	return nil
}

func (m HeroModel) Delete(id int64) error {
	return nil
}

models.go

package data

import (
	"database/sql"
	"errors"
)

// Error returned when looking up a hero that doesn't exist in our database.
var (
	ErrRecordNotFound = errors.New("record not found")
)

type Models struct {
	Heroes HeroModel
}

func NewModels(db *sql.DB) Models {
	return Models{
		Heroes: HeroModel{DB: db},
	}
}

Updated main.go

type application struct {
	config config
	logger *log.Logger
	models data.Models
}

func main() {
	var cfg config

	flag.IntVar(&cfg.port, "port", 4000, "API server port")
	flag.StringVar(&cfg.env, "env", "development", "Environment (development|staging|production)")
	flag.StringVar(&cfg.db.dsn, "db-dsn", os.Getenv("HEROES_DB_DSN"), "PostgreSQL DSN")
	flag.IntVar(&cfg.db.maxOpenConns, "db-max-open-conns", 25, "PostgreSQL max open connections")
	flag.IntVar(&cfg.db.maxIdleConns, "db-max-idle-conns", 25, "PostgreSQL max idle connections")
	flag.StringVar(&cfg.db.maxIdleTime, "db-max-idle-time", "15m", "PostgreSQL max connection idle time")
	flag.Parse()

	logger := log.New(os.Stdout, "", log.Ldate|log.Ltime)

	db, err := openDB(cfg)
	if err != nil {
		logger.Fatal(err)
	}

	defer db.Close()
	logger.Printf("database connection pool established")

	app := &application{
		config: cfg,
		logger: logger,
		models: data.NewModels(db),
	}

	srv := &http.Server{
		Addr:         fmt.Sprintf(":%d", cfg.port),
		Handler:      app.routes(),
		IdleTimeout:  time.Minute,
		ReadTimeout:  10 * time.Second,
		WriteTimeout: 30 * time.Second,
	}

	logger.Printf("starting %s server on %s", cfg.env, srv.Addr)
	err = srv.ListenAndServe()
	logger.Fatal(err)
}

func openDB(cfg config) (*sql.DB, error) {
	db, err := sql.Open("postgres", cfg.db.dsn)
	if err != nil {
		return nil, err
	}

	db.SetMaxOpenConns(cfg.db.maxOpenConns)
	db.SetMaxIdleConns(cfg.db.maxIdleConns)
	duration, err := time.ParseDuration(cfg.db.maxIdleTime)
	if err != nil {
		return nil, err
	}

	db.SetConnMaxIdleTime(duration)

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	err = db.PingContext(ctx)
	if err != nil {
		return nil, err
	}

	return db, nil
}

Updated internal/data/heroes.go

func (m HeroModel) Insert(hero *Hero) error {
	query := `
             INSERT INTO heroes (first_seen, name, can_fly, realname, abilities) 
             VALUES ($1, $2, $3, $4, $5)
             RETURNING id, version`
	args := []interface{}{hero.FirstSeen, hero.Name, hero.CanFly, hero.RealName, pq.Array(hero.Abilities)}
	return m.DB.QueryRow(query, args...).Scan(&hero.ID, &hero.Version)
}

Updated heroes.go

func (app *application) createHeroHandler(w http.ResponseWriter, r *http.Request) {
	var input struct {
		Name      string    `json:"name"`
		FirstSeen time.Time `json:"firstSeen"`
		CanFly    bool      `json:"canFly"`
		RealName  string    `json:"realName,omitempty"`
		Abilities []string  `json:"abilities"`
	}

	err := app.readJSON(w, r, &input)
	if err != nil {
		app.badRequestResponse(w, r, err)
		return
	}

	hero := &data.Hero{
		Name:      input.Name,
		FirstSeen: input.FirstSeen,
		CanFly:    input.CanFly,
		RealName:  input.RealName,
		Abilities: input.Abilities,
	}

	v := validator.New()

	if data.ValidateHero(v, hero); !v.Valid() {
		app.failedValidationResponse(w, r, v.Errors)
		return
	}

	err = app.models.Heroes.Insert(hero)
	if err != nil {
		app.serverErrorResponse(w, r, err)
		return
	}

	headers := make(http.Header)
	headers.Set("Location", fmt.Sprintf("/v1/heroes/%d", hero.ID))

	err = app.writeJSON(w, http.StatusCreated, hero, headers)
	if err != nil {
		app.serverErrorResponse(w, r, err)
	}
}

Generate Demo Data

go get github.com/brianvoe/gofakeit

Updated routes.go

func (app *application) routes() *httprouter.Router {
	router := httprouter.New()

	router.NotFound = http.HandlerFunc(app.notFoundResponse)
	router.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)

	router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)
	router.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
	router.HandlerFunc(http.MethodPost, "/v1/generate", app.generateDemoDataHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)

	// Return the httprouter instance.
	return router
}

Updated heroes.go

func (app *application) generateDemoDataHandler(w http.ResponseWriter, r *http.Request) {
	for i := 0; i < 20; i++ {
		hero := &data.Hero{
			Name:      gofakeit.Name(),
			FirstSeen: gofakeit.DateRange(time.Date(1900, time.Month(1), 1, 0, 0, 0, 0, time.UTC), time.Now()),
			CanFly:    gofakeit.Bool(),
			RealName:  gofakeit.Name(),
			Abilities: []string{"foo", "bar"},
		}

		err := app.models.Heroes.Insert(hero)
		if err != nil {
			app.serverErrorResponse(w, r, err)
			return
		}
	}

	headers := make(http.Header)
	err := app.writeJSON(w, http.StatusCreated, "created", headers)
	if err != nil {
		app.serverErrorResponse(w, r, err)
	}
}

Updated internal/data/heroes.go

func (m HeroModel) Get(id int64) (*Hero, error) {
	if id < 1 {
		return nil, ErrRecordNotFound
	}

	query := `
        SELECT id, first_seen, name, can_fly, realname, abilities, version
        FROM heroes
        WHERE id = $1`

	var hero Hero

	err := m.DB.QueryRow(query, id).Scan(
		&hero.ID,
		&hero.FirstSeen,
		&hero.Name,
		&hero.CanFly,
		&hero.RealName,
		pq.Array(&hero.Abilities),
		&hero.Version,
	)

	if err != nil {
		switch {
		case errors.Is(err, sql.ErrNoRows):
			return nil, ErrRecordNotFound
		default:
			return nil, err
		}
	}

	return &hero, nil
}

Updated heroes.go

func (app *application) showHeroHandler(w http.ResponseWriter, r *http.Request) {
	id, err := app.readIDParam(r)
	if err != nil {
		http.NotFound(w, r)
		return
	}

	hero, err := app.models.Heroes.Get(id)
	if err != nil {
		switch {
		case errors.Is(err, data.ErrRecordNotFound):
			app.notFoundResponse(w, r)
		default:
			app.serverErrorResponse(w, r, err)
		}
		return
	}

	err = app.writeJSON(w, http.StatusOK, hero, nil)
	if err != nil {
		app.serverErrorResponse(w, r, err)
	}
}

Updated internal/data/heroes.go

func (m HeroModel) Update(hero *Hero) error {
	query := `
        UPDATE heroes
        SET first_seen = $1, name = $2, can_fly = $3, realname = $4, abilities = $5, version = version + 1
        WHERE id = $6
        RETURNING version`

	args := []interface{}{
		hero.FirstSeen,
		hero.Name,
		hero.CanFly,
		hero.RealName,
		pq.Array(hero.Abilities),
		hero.ID,
	}

	return m.DB.QueryRow(query, args...).Scan(&hero.Version)
}

Updated heroes.go

func (app *application) updateHeroHandler(w http.ResponseWriter, r *http.Request) {
    id, err := app.readIDParam(r)
    if err != nil {
        app.notFoundResponse(w, r)
        return
    }

    hero, err := app.models.Heroes.Get(id)
    if err != nil {
        switch {
        case errors.Is(err, data.ErrRecordNotFound):
            app.notFoundResponse(w, r)
        default:
            app.serverErrorResponse(w, r, err)
        }
        return
    }

	var input struct {
		Name      string    `json:"name"`
		FirstSeen time.Time `json:"firstSeen"`
		CanFly    bool      `json:"canFly"`
		RealName  string    `json:"realName,omitempty"`
		Abilities []string  `json:"abilities"`
	}

    err = app.readJSON(w, r, &input)
    if err != nil {
        app.badRequestResponse(w, r, err)
        return
    }

    hero.FirstSeen = input.FirstSeen
    hero.Name = input.Name
    hero.CanFly = input.CanFly
    hero.RealName = input.RealName
    hero.Abilities = input.Abilities

    v := validator.New()

    if data.ValidateHero(v, hero); !v.Valid() {
        app.failedValidationResponse(w, r, v.Errors)
        return
    }

    err = app.models.Heroes.Update(hero)
    if err != nil {
        app.serverErrorResponse(w, r, err)
        return
    }

    err = app.writeJSON(w, http.StatusOK, hero, nil)
    if err != nil {
        app.serverErrorResponse(w, r, err)
    }
}

Updated routes.go

func (app *application) routes() *httprouter.Router {
	router := httprouter.New()

	router.NotFound = http.HandlerFunc(app.notFoundResponse)
	router.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)

	router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)
	router.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
	router.HandlerFunc(http.MethodPut, "/v1/heroes/:id", app.updateHeroHandler)
	router.HandlerFunc(http.MethodPost, "/v1/generate", app.generateDemoDataHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)

	// Return the httprouter instance.
	return router
}

Updated internal/data/heroes.go

func (m HeroModel) Delete(id int64) error {
	if id < 1 {
		return ErrRecordNotFound
	}

	query := `
        DELETE FROM heroes
        WHERE id = $1`

	result, err := m.DB.Exec(query, id)
	if err != nil {
		return err
	}

	rowsAffected, err := result.RowsAffected()
	if err != nil {
		return err
	}

	if rowsAffected == 0 {
		return ErrRecordNotFound
	}

	return nil
}

Updated heroes.go

func (app *application) deleteHeroHandler(w http.ResponseWriter, r *http.Request) {
    id, err := app.readIDParam(r)
    if err != nil {
        app.notFoundResponse(w, r)
        return
    }

    err = app.models.Heroes.Delete(id)
    if err != nil {
        switch {
        case errors.Is(err, data.ErrRecordNotFound):
            app.notFoundResponse(w, r)
        default:
            app.serverErrorResponse(w, r, err)
        }
        return
    }

    err = app.writeJSON(w, http.StatusOK, "successfully deleted", nil)
    if err != nil {
        app.serverErrorResponse(w, r, err)
    }
}

Updated routes.go

func (app *application) routes() *httprouter.Router {
	router := httprouter.New()

	router.NotFound = http.HandlerFunc(app.notFoundResponse)
	router.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)

	router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)
	router.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
	router.HandlerFunc(http.MethodPut, "/v1/heroes/:id", app.updateHeroHandler)
	router.HandlerFunc(http.MethodDelete, "/v1/heroes/:id", app.deleteHeroHandler)
	router.HandlerFunc(http.MethodPost, "/v1/generate", app.generateDemoDataHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)

	// Return the httprouter instance.
	return router
}

Querying Data

touch internal/data/filters.go

Updated helpers.go

func (app *application) readString(qs url.Values, key string, defaultValue string) string {
	s := qs.Get(key)
	if s == "" {
		return defaultValue
	}

	return s
}

func (app *application) readCSV(qs url.Values, key string, defaultValue []string) []string {
	csv := qs.Get(key)
	if csv == "" {
		return defaultValue
	}

	return strings.Split(csv, ",")
}

func (app *application) readInt(qs url.Values, key string, defaultValue int, v *validator.Validator) int {
	s := qs.Get(key)
	if s == "" {
		return defaultValue
	}

	i, err := strconv.Atoi(s)
	if err != nil {
		v.AddError(key, "must be an integer value")
		return defaultValue
	}

	return i
}

filters.go

package data

import "heroes.rainerstropek.com/internal/validator"

type Filters struct {
	Page         int
	PageSize     int
	Sort         string
	SortSafelist []string
}

func ValidateFilters(v *validator.Validator, f Filters) {
	v.Check(f.Page > 0, "page", "must be greater than zero")
	v.Check(f.Page <= 10_000_000, "page", "must be a maximum of 10 million")
	v.Check(f.PageSize > 0, "page_size", "must be greater than zero")
	v.Check(f.PageSize <= 100, "page_size", "must be a maximum of 100")
	v.Check(validator.In(f.Sort, f.SortSafelist...), "sort", "invalid sort value")
}

func (f Filters) limit() int {
	return f.PageSize
}

func (f Filters) offset() int {
	return (f.Page - 1) * f.PageSize
}

Updated heroes.go

func (app *application) listHeroesHandler(w http.ResponseWriter, r *http.Request) {
	var input struct {
		Name      string
		Abilities []string
		data.Filters
	}

	v := validator.New()
	qs := r.URL.Query()
	input.Name = app.readString(qs, "name", "")
	input.Name = fmt.Sprintf("%%%s%%", input.Name)
	input.Abilities = app.readCSV(qs, "abilities", []string{})
	input.Filters.Page = app.readInt(qs, "page", 1, v)
	input.Filters.PageSize = app.readInt(qs, "page_size", 20, v)
	input.Filters.Sort = app.readString(qs, "sort", "id")
	input.Filters.SortSafelist = []string{"id", "name", "realname"}

	if data.ValidateFilters(v, input.Filters); !v.Valid() {
		app.failedValidationResponse(w, r, v.Errors)
		return
	}

	fmt.Fprintf(w, "%+v\n", input)
}

Updated routes.go

func (app *application) routes() *httprouter.Router {
	router := httprouter.New()

	router.NotFound = http.HandlerFunc(app.notFoundResponse)
	router.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)

	router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes", app.listHeroesHandler)
	router.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
	router.HandlerFunc(http.MethodPut, "/v1/heroes/:id", app.updateHeroHandler)
	router.HandlerFunc(http.MethodDelete, "/v1/heroes/:id", app.deleteHeroHandler)
	router.HandlerFunc(http.MethodPost, "/v1/generate", app.generateDemoDataHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)

	// Return the httprouter instance.
	return router
}

Updated internal/data/heroes.go

func (m HeroModel) GetAll(name string, abilities []string, filters Filters) ([]*Hero, error) {
	query := `
        SELECT id, first_seen, name, can_fly, realname, abilities, version
        FROM heroes
        ORDER BY id`

	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()

	rows, err := m.DB.QueryContext(ctx, query)
	if err != nil {
		return nil, err
	}

	defer rows.Close()

	heroes := []*Hero{}

	for rows.Next() {
		var hero Hero

		err := rows.Scan(
			&hero.ID,
			&hero.FirstSeen,
			&hero.Name,
			&hero.CanFly,
			&hero.RealName,
			pq.Array(&hero.Abilities),
			&hero.Version,
		)
		if err != nil {
			return nil, err
		}

		heroes = append(heroes, &hero)
	}

	if err = rows.Err(); err != nil {
		return nil, err
	}

	return heroes, nil
}

Updated heroes.go

func (app *application) listHeroesHandler(w http.ResponseWriter, r *http.Request) {
	var input struct {
		Name      string
		Abilities []string
		data.Filters
	}

	v := validator.New()
	qs := r.URL.Query()
	input.Name = app.readString(qs, "name", "")
	input.Name = fmt.Sprintf("%%%s%%", input.Name)
	input.Abilities = app.readCSV(qs, "abilities", []string{})
	input.Filters.Page = app.readInt(qs, "page", 1, v)
	input.Filters.PageSize = app.readInt(qs, "page_size", 20, v)
	input.Filters.Sort = app.readString(qs, "sort", "id")
	input.Filters.SortSafelist = []string{"id", "name", "realname"}

	if data.ValidateFilters(v, input.Filters); !v.Valid() {
		app.failedValidationResponse(w, r, v.Errors)
		return
	}

	heroes, err := app.models.Heroes.GetAll(input.Name, input.Abilities, input.Filters)
	if err != nil {
		app.serverErrorResponse(w, r, err)
		return
	}

	err = app.writeJSON(w, http.StatusOK, heroes, nil)
	if err != nil {
		app.serverErrorResponse(w, r, err)
	}
}

Updated models.go

type Models struct {
	Heroes interface {
		Insert(hero *Hero) error
		Get(id int64) (*Hero, error)
		Update(hero *Hero) error
		Delete(id int64) error
		GetAll(name string, abilities []string, filters Filters) ([]*Hero, error)
	}
}

Updated internal/data/heroes.go

func (m HeroModel) GetAll(name string, abilities []string, filters Filters) ([]*Hero, error) {
	query := `
        SELECT id, first_seen, name, can_fly, realname, abilities, version
        FROM heroes
        WHERE (LOWER(name) LIKE LOWER($1) OR $1 = '') 
        AND (abilities @> $2 OR $2 = '{}')     
        ORDER BY id`

	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()

	rows, err := m.DB.QueryContext(ctx, query, name, pq.Array(abilities))
	if err != nil {
		return nil, err
	}

	defer rows.Close()

	heroes := []*Hero{}

	for rows.Next() {
		var hero Hero

		err := rows.Scan(
			&hero.ID,
			&hero.FirstSeen,
			&hero.Name,
			&hero.CanFly,
			&hero.RealName,
			pq.Array(&hero.Abilities),
			&hero.Version,
		)
		if err != nil {
			return nil, err
		}

		heroes = append(heroes, &hero)
	}

	if err = rows.Err(); err != nil {
		return nil, err
	}

	return heroes, nil
}

Updated internal/data/heroes.go

func (m HeroModel) GetAll(name string, abilities []string, filters Filters) ([]*Hero, error) {
	query := fmt.Sprintf(`
        SELECT id, first_seen, name, can_fly, realname, abilities, version
        FROM heroes
        WHERE (LOWER(name) LIKE LOWER($1) OR $1 = '') 
        AND (abilities @> $2 OR $2 = '{}')     
        ORDER BY %s ASC, id ASC`, filters.Sort)

	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()

	rows, err := m.DB.QueryContext(ctx, query, name, pq.Array(abilities))
	if err != nil {
		return nil, err
	}

	defer rows.Close()

	heroes := []*Hero{}

	for rows.Next() {
		var hero Hero

		err := rows.Scan(
			&hero.ID,
			&hero.FirstSeen,
			&hero.Name,
			&hero.CanFly,
			&hero.RealName,
			pq.Array(&hero.Abilities),
			&hero.Version,
		)
		if err != nil {
			return nil, err
		}

		heroes = append(heroes, &hero)
	}

	if err = rows.Err(); err != nil {
		return nil, err
	}

	return heroes, nil
}

Updated internal/data/heroes.go

func (m HeroModel) GetAll(name string, abilities []string, filters Filters) ([]*Hero, error) {
	query := fmt.Sprintf(`
        SELECT id, first_seen, name, can_fly, realname, abilities, version
        FROM heroes
        WHERE (LOWER(name) LIKE LOWER($1) OR $1 = '') 
        AND (abilities @> $2 OR $2 = '{}')     
        ORDER BY %s ASC, id ASC
        LIMIT $3 OFFSET $4`, filters.Sort)

	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()

	args := []interface{}{name, pq.Array(abilities), filters.limit(), filters.offset()}
	rows, err := m.DB.QueryContext(ctx, query, args...)
	if err != nil {
		return nil, err
	}

	defer rows.Close()

	heroes := []*Hero{}

	for rows.Next() {
		var hero Hero

		err := rows.Scan(
			&hero.ID,
			&hero.FirstSeen,
			&hero.Name,
			&hero.CanFly,
			&hero.RealName,
			pq.Array(&hero.Abilities),
			&hero.Version,
		)
		if err != nil {
			return nil, err
		}

		heroes = append(heroes, &hero)
	}

	if err = rows.Err(); err != nil {
		return nil, err
	}

	return heroes, nil
}

Structured Logging,
Additional Error Handling

Logging, Error Handling

Many options 🔗
- Here: zerolog
- go get -u github.com/rs/zerolog/log
- Lots of config options (see zerolog docs)
Return HTTP 500 on panics
- touch cmd/api/middleware.go

Updated main.go

package main

import (
	"context"
	"database/sql"

	"flag"
	"fmt"
	"log"
	"net/http"
	"os"
	"time"

	_ "github.com/lib/pq"
	"github.com/rs/zerolog"
	"heroes.rainerstropek.com/internal/data"
)

const version = "1.0.0"

type config struct {
	port int
	env  string
	db   struct {
		dsn          string
		maxOpenConns int
		maxIdleConns int
		maxIdleTime  string
	}
}

type application struct {
	config config
	logger *zerolog.Logger
	models data.Models
}

func main() {
	var cfg config

	flag.IntVar(&cfg.port, "port", 4000, "API server port")
	flag.StringVar(&cfg.env, "env", "development", "Environment (development|staging|production)")
	flag.StringVar(&cfg.db.dsn, "db-dsn", os.Getenv("HEROES_DB_DSN"), "PostgreSQL DSN")
	flag.IntVar(&cfg.db.maxOpenConns, "db-max-open-conns", 25, "PostgreSQL max open connections")
	flag.IntVar(&cfg.db.maxIdleConns, "db-max-idle-conns", 25, "PostgreSQL max idle connections")
	flag.StringVar(&cfg.db.maxIdleTime, "db-max-idle-time", "15m", "PostgreSQL max connection idle time")
	flag.Parse()

	//logger := log.New(os.Stdout, "", log.Ldate|log.Ltime)
	logger := zerolog.New(os.Stdout).With().Timestamp().Logger()

	db, err := openDB(cfg)
	if err != nil {
		logger.Fatal().Err(err)
	}

	defer db.Close()
	logger.Printf("database connection pool established")

	app := &application{
		config: cfg,
		logger: &logger,
		models: data.NewModels(db),
	}

	srv := &http.Server{
		Addr:         fmt.Sprintf(":%d", cfg.port),
		Handler:      app.routes(),
		ErrorLog:     log.New(app.logger, "", 0),
		IdleTimeout:  time.Minute,
		ReadTimeout:  10 * time.Second,
		WriteTimeout: 30 * time.Second,
	}

	app.logger.Info().Str("addr", srv.Addr).Str("env", app.config.env).Msg("Starting server")
	err = srv.ListenAndServe()
	logger.Fatal().Err(err)
}

func openDB(cfg config) (*sql.DB, error) {
	db, err := sql.Open("postgres", cfg.db.dsn)
	if err != nil {
		return nil, err
	}

	db.SetMaxOpenConns(cfg.db.maxOpenConns)
	db.SetMaxIdleConns(cfg.db.maxIdleConns)
	duration, err := time.ParseDuration(cfg.db.maxIdleTime)
	if err != nil {
		return nil, err
	}

	db.SetConnMaxIdleTime(duration)

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	err = db.PingContext(ctx)
	if err != nil {
		return nil, err
	}

	return db, nil
}

Updated errors.go

func (app *application) logError(r *http.Request, err error) {
	app.logger.Error().Err(err)
}

middleware.go

package main

import (
	"fmt"
	"net/http"
)

func (app *application) recoverPanic(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		defer func() {
			if err := recover(); err != nil {
				w.Header().Set("Connection", "close")
				app.serverErrorResponse(w, r, fmt.Errorf("%s", err))
			}
		}()

		next.ServeHTTP(w, r)
	})
}

Update routes.go

func (app *application) routes() http.Handler {
	router := httprouter.New()

	router.NotFound = http.HandlerFunc(app.notFoundResponse)
	router.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)

	router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes", app.listHeroesHandler)
	router.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
	router.HandlerFunc(http.MethodPut, "/v1/heroes/:id", app.updateHeroHandler)
	router.HandlerFunc(http.MethodDelete, "/v1/heroes/:id", app.deleteHeroHandler)
	router.HandlerFunc(http.MethodPost, "/v1/generate", app.generateDemoDataHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)

	return app.recoverPanic(router)
}

Graceful Shutdown

touch cmd/api/server.go

server.go

package main

import (
	"context"
	"errors"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func (app *application) serve() error {
	srv := &http.Server{
		Addr:         fmt.Sprintf(":%d", app.config.port),
		Handler:      app.routes(),
		ErrorLog:     log.New(app.logger, "", 0),
		IdleTimeout:  time.Minute,
		ReadTimeout:  10 * time.Second,
		WriteTimeout: 30 * time.Second,
	}

	shutdownError := make(chan error)

	go func() {
		quit := make(chan os.Signal, 1)
		signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
		s := <-quit

		app.logger.Info().Str("signal", s.String()).Msg("shutting down server")

		// Give active requests a chance to finish
		// See also https://github.com/golang/go/issues/33191
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()

		shutdownError <- srv.Shutdown(ctx)
	}()

	app.logger.Info().Str("addr", srv.Addr).Str("env", app.config.env).Msg("starting server")

	err := srv.ListenAndServe()
	if !errors.Is(err, http.ErrServerClosed) {
		return err
	}

	err = <-shutdownError
	if err != nil {
		return err
	}

	app.logger.Info().Str("addr", srv.Addr).Msg("stopped server")
	return nil
}

Updated main.go

func main() {
	var cfg config

	flag.IntVar(&cfg.port, "port", 4000, "API server port")
	flag.StringVar(&cfg.env, "env", "development", "Environment (development|staging|production)")
	flag.StringVar(&cfg.db.dsn, "db-dsn", os.Getenv("HEROES_DB_DSN"), "PostgreSQL DSN")
	flag.IntVar(&cfg.db.maxOpenConns, "db-max-open-conns", 25, "PostgreSQL max open connections")
	flag.IntVar(&cfg.db.maxIdleConns, "db-max-idle-conns", 25, "PostgreSQL max idle connections")
	flag.StringVar(&cfg.db.maxIdleTime, "db-max-idle-time", "15m", "PostgreSQL max connection idle time")
	flag.Parse()

	//logger := log.New(os.Stdout, "", log.Ldate|log.Ltime)
	logger := zerolog.New(os.Stdout).With().Timestamp().Logger()

	db, err := openDB(cfg)
	if err != nil {
		logger.Fatal().Err(err)
	}

	defer db.Close()
	logger.Printf("database connection pool established")

	app := &application{
		config: cfg,
		logger: &logger,
		models: data.NewModels(db),
	}

	err = app.serve()
	logger.Fatal().Err(err)
}

CORS

Updated middleware.go

func (app *application) enableCORS(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Access-Control-Allow-Origin", "*")
		w.Header().Set("Access-Control-Allow-Methods", "*")
		w.Header().Set("Access-Control-Allow-Headers", "*")

		next.ServeHTTP(w, r)
	})
}

Updated routes.go

func (app *application) routes() http.Handler {
	router := httprouter.New()

	router.NotFound = http.HandlerFunc(app.notFoundResponse)
	router.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)

	router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes", app.listHeroesHandler)
	router.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
	router.HandlerFunc(http.MethodPut, "/v1/heroes/:id", app.updateHeroHandler)
	router.HandlerFunc(http.MethodDelete, "/v1/heroes/:id", app.deleteHeroHandler)
	router.HandlerFunc(http.MethodPost, "/v1/generate", app.generateDemoDataHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)

	return app.recoverPanic(app.enableCORS(router))
}

www/index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <title>CORS Test</title>
</head>
<body>
    
</body>
<script lang="JavaScript" >
    fetch('http://localhost:4000/v1/healthcheck')
        .then(response => response.json())
        .then(data => console.log(data));
</script>
</html>

Serve and test in browser

npm install -g serve
cd www
serve .

Make adding middlewares easier

package main

import (
	"net/http"

	"github.com/julienschmidt/httprouter"
	"github.com/justinas/alice"
)

func (app *application) routes() http.Handler {
	router := httprouter.New()

	router.NotFound = http.HandlerFunc(app.notFoundResponse)
	router.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)

	router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes", app.listHeroesHandler)
	router.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
	router.HandlerFunc(http.MethodPut, "/v1/heroes/:id", app.updateHeroHandler)
	router.HandlerFunc(http.MethodDelete, "/v1/heroes/:id", app.deleteHeroHandler)
	router.HandlerFunc(http.MethodPost, "/v1/generate", app.generateDemoDataHandler)
	router.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)

	c := alice.New()
	c.Append(app.recoverPanic)
	c.Append(app.enableCORS)
	chain := c.Then(router)

	return chain
}

Updated routes.go

go get "github.com/justinas/alice"

Testing

Test types

Traditional unit tests
- Example 🔗
Tests for handler functions
End-to-end Tests

validator_test.go

package validator

import (
	"testing"

	"github.com/stretchr/testify/assert"
)

func TestEmptyValidatorIsValid(t *testing.T) {
	v := New()
	assert.True(t, v.Valid())
}

func TestCheck(t *testing.T) {
	v := New()
	v.Check(false, "test", "testmessage")
	assert.False(t, v.Valid())
	assert.Equal(t, "testmessage", v.Errors["test"])
}

func TestIn(t *testing.T) {
	assert.True(t, In("a", []string{"a", "b"}...))
	assert.False(t, In("c", []string{"a", "b"}...))
}

func TestMatches(t *testing.T) {
	assert.True(t, Matches("john.doe@somewhere.com", EmailRX))
	assert.False(t, Matches("john.doe@", EmailRX))
}

func TestUnique(t *testing.T) {
	assert.True(t, Unique([]string{"a", "b"}))
	assert.False(t, Unique([]string{"a", "a"}))
}

healthcheck_test.go

package main

import (
	"encoding/json"
	"net/http"
	"net/http/httptest"
	"testing"

	"github.com/stretchr/testify/assert"
)

func TestHealthcheck(t *testing.T) {
	// Setup mock configuration for "Test" environment
	app := &application{
		config: config{
			env: "Test",
		},
	}

	// Test healthcheck handler function in isolation
	rr := httptest.NewRecorder()
	r, err := http.NewRequest(http.MethodGet, "/v1/healthcheck", nil)
	if err != nil {
		t.Fatal(err)
	}

	app.healthcheckHandler(rr, r)

	// Get and verify result
	rs := rr.Result()
	if rs.StatusCode != http.StatusOK {
		t.Errorf("want %d; got %d", http.StatusOK, rs.StatusCode)
	}

	healthResult := make(map[string]string)
	err = json.NewDecoder(rs.Body).Decode(&healthResult)
	if err != nil {
		t.Fatal(err)
		return
	}

	assert.Equal(t, "Test", healthResult["environment"])
}

Updated healthcheck_test.go

func TestHealthcheckEndToEnd(t *testing.T) {
	// Setup mock configuration for "Test" environment
	app := &application{
		config: config{
			env: "Test",
		},
	}

	// Run HTTPS server on random port
	ts := httptest.NewTLSServer(app.routes())
	defer ts.Close()

	rs, err := ts.Client().Get(ts.URL + "/v1/healthcheck")
	if err != nil {
		t.Fatal(err)
	}

	if rs.StatusCode != http.StatusOK {
		t.Errorf("want %d; got %d", http.StatusOK, rs.StatusCode)
	}

	healthResult := make(map[string]string)
	err = json.NewDecoder(rs.Body).Decode(&healthResult)
	if err != nil {
		t.Fatal(err)
		return
	}

	assert.Equal(t, "Test", healthResult["environment"])
}

Auth

Goal: Authentication with AAD

Project structure
- mkdir internal/middleware
  touch internal/middleware/jwt.go
Validate JWT
- Good middleware from Auth0 🔗
- go get github.com/auth0/go-jwt-middleware/v2
- V2 is currently beta!
Prerequisites
- Access to AAD
- Register an application in AAD
- Use e.g. Postman to get an access token for your API

Updated main.go

package main

import (
	"context"
	"database/sql"

	"flag"
	"os"
	"time"

	_ "github.com/lib/pq"
	"github.com/rs/zerolog"
	"heroes.rainerstropek.com/internal/data"
)

const version = "1.0.0"

type config struct {
	port int
	env  string
	db   struct {
		dsn          string
		maxOpenConns int
		maxIdleConns int
		maxIdleTime  string
	}
	azure struct {
		tenantId string
	}
}

type application struct {
	config config
	logger *zerolog.Logger
	models data.Models
}

func main() {
	var cfg config

	flag.IntVar(&cfg.port, "port", 4000, "API server port")
	flag.StringVar(&cfg.env, "env", "development", "Environment (development|staging|production)")
	flag.StringVar(&cfg.db.dsn, "db-dsn", os.Getenv("HEROES_DB_DSN"), "PostgreSQL DSN")
	flag.IntVar(&cfg.db.maxOpenConns, "db-max-open-conns", 25, "PostgreSQL max open connections")
	flag.IntVar(&cfg.db.maxIdleConns, "db-max-idle-conns", 25, "PostgreSQL max idle connections")
	flag.StringVar(&cfg.db.maxIdleTime, "db-max-idle-time", "15m", "PostgreSQL max connection idle time")
	flag.StringVar(&cfg.azure.tenantId, "azure-tenant", os.Getenv("AZURE_TENANT"), "AAD Tenant")
	flag.Parse()

	//logger := log.New(os.Stdout, "", log.Ldate|log.Ltime)
	logger := zerolog.New(os.Stdout).With().Timestamp().Logger()

	db, err := openDB(cfg)
	if err != nil {
		logger.Fatal().Err(err)
	}

	defer db.Close()
	logger.Printf("database connection pool established")

	app := &application{
		config: cfg,
		logger: &logger,
		models: data.NewModels(db),
	}

	err = app.serve()
	logger.Fatal().Err(err)
}

func openDB(cfg config) (*sql.DB, error) {
	db, err := sql.Open("postgres", cfg.db.dsn)
	if err != nil {
		return nil, err
	}

	db.SetMaxOpenConns(cfg.db.maxOpenConns)
	db.SetMaxIdleConns(cfg.db.maxIdleConns)
	duration, err := time.ParseDuration(cfg.db.maxIdleTime)
	if err != nil {
		return nil, err
	}

	db.SetConnMaxIdleTime(duration)

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	err = db.PingContext(ctx)
	if err != nil {
		return nil, err
	}

	return db, nil
}

jwt.go

package middleware

import (
	"context"
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"net/url"
	"time"

	jwtmiddleware "github.com/auth0/go-jwt-middleware/v2"
	"github.com/auth0/go-jwt-middleware/v2/jwks"
	v "github.com/auth0/go-jwt-middleware/v2/validator"
)

type CustomClaimsExample struct {
	Name       string `json:"name"`
	FamilyName string `json:"family_name"`
}

// Validate does nothing for this example.
func (c *CustomClaimsExample) Validate(ctx context.Context) error {
	return nil
}

func NewJwtMiddleware(azureTenantId string, requiredScopes []string) *jwtmiddleware.JWTMiddleware {
	issuerURL, err := url.Parse(fmt.Sprintf("https://login.microsoftonline.com/%s/", azureTenantId))
	if err != nil {
		log.Fatalf("failed to parse the issuer url: %v", err)
	}

	provider := jwks.NewCachingProvider(issuerURL, 5*time.Minute)
	customClaims := &CustomClaimsExample{}
	jwtValidator, _ := v.New(
		provider.KeyFunc,
		"RS256",
		fmt.Sprintf("https://sts.windows.net/%s/", azureTenantId),
		requiredScopes,
		v.WithCustomClaims(customClaims))
	return jwtmiddleware.New(jwtValidator.ValidateToken)
}

func ClaimsHandler(w http.ResponseWriter, r *http.Request) {
	claims := r.Context().Value(jwtmiddleware.ContextKey{}).(*v.ValidatedClaims)

	payload, err := json.Marshal(claims)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	w.Header().Set("Content-Type", "application/json")
	w.Write(payload)
}

Updated routes.go

package main

import (
	"net/http"

	"heroes.rainerstropek.com/internal/middleware"

	"github.com/julienschmidt/httprouter"
	"github.com/justinas/alice"
)

func (app *application) routes() http.Handler {
	router := httprouter.New()
	router.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)
	router.HandlerFunc(http.MethodGet, "/v1/healthcheck", app.healthcheckHandler)

	jwtMiddleware := middleware.NewJwtMiddleware(
		app.config.azure.tenantId,
		[]string{"api://4fac0887-b94f-4ea9-a8d3-06c7bca2a7bd"}) // Make scope configurable if you need to

	protectedrouter := httprouter.New()
	protectedrouter.NotFound = http.HandlerFunc(app.notFoundResponse)
	protectedrouter.MethodNotAllowed = http.HandlerFunc(app.methodNotAllowedResponse)
	protectedrouter.HandlerFunc(http.MethodGet, "/v1/heroes", app.listHeroesHandler)
	protectedrouter.HandlerFunc(http.MethodPost, "/v1/heroes", app.createHeroHandler)
	protectedrouter.HandlerFunc(http.MethodPut, "/v1/heroes/:id", app.updateHeroHandler)
	protectedrouter.HandlerFunc(http.MethodDelete, "/v1/heroes/:id", app.deleteHeroHandler)
	protectedrouter.HandlerFunc(http.MethodPost, "/v1/generate", app.generateDemoDataHandler)
	protectedrouter.HandlerFunc(http.MethodGet, "/v1/heroes/:id", app.showHeroHandler)
	protectedrouter.HandlerFunc(http.MethodGet, "/v1/claims", middleware.ClaimsHandler)
	router.NotFound = jwtMiddleware.CheckJWT(protectedrouter)

	c := alice.New()
	c.Append(app.recoverPanic)
	c.Append(app.enableCORS)
	chain := c.Then(router)

	return chain
}

Updated requests.go

@host=http://localhost:4000
@token=eyJ0eXAiOi...

###
GET {{host}}/v1/healthcheck

###
POST {{host}}/v1/heroes
Authorization: Bearer {{token}}

{
    "name": "Superman",
    "firstSeen": "1935-01-01T00:00:00Z",
    "canFly": true,
    "realName": "Clark Kent",
    "abilities": [ "super strong", "can disguise with glasses" ]
}

###
GET {{host}}/v1/heroes/1
Authorization: Bearer {{token}}

###
# Test error handling
GET {{host}}/v1/somethingThatDoesNotExist
Authorization: Bearer {{token}}

###
# Test error handling
POST {{host}}/v1/healthcheck
Authorization: Bearer {{token}}

###
POST {{host}}/v1/generate
Authorization: Bearer {{token}}

###
PUT {{host}}/v1/heroes/1
Authorization: Bearer {{token}}

{
    "name": "Homelander",
    "firstSeen": "2020-01-01T00:00:00Z",
    "canFly": true,
    "abilities": [ "super strong" ]
}

###
DELETE {{host}}/v1/heroes/1
Authorization: Bearer {{token}}

###
GET {{host}}/v1/heroes?name=Or&abilities=foo&page=1&page_size=3&sort=name
Authorization: Bearer {{token}}

###
GET {{host}}/v1/heroes?page=2&page_size=3&sort=name
Authorization: Bearer {{token}}

###
GET {{host}}/v1/claims
Authorization: Bearer {{token}}

Web APIs in Go

Introduction

Rainer Stropek

Recommended Reading

Getting Started

Setup Environment

Setup folders

requests.http

Basic Web Server

main.go

healthcheck.go

Notes

Notes

http.Handler

API Routes

Switch Router

routes.go

Updated main.go

heroes.go

Test

helpers.go

Updated heroes.go

Sending JSON

Updated healthcheck.go

Generating JSON

Updated healthcheck.go

Updated helpers.go

Updated healthcheck.go

internal/data/heroes.go

Updated heroes.go

Configure JSON

Updated internal/data/heroes.go

Error Messages

errors.go

Updated healthcheck.go

Updated heroes.go

Updated routes.go

When to panic()?

Parsing JSON

Parsing JSON

Updated heroes.go

Updated helpers.go

Updated heroes.go

Validating Input

Validating

validator.go

Updated heroes.go

Updated internal/data/heroes.go

Updated heroes.go

Setting up the Database

Setup Postgres

Setup Postgres

What about ORMs?

Updated main.go

Migrations

Install migrate

SQL Scripts

Work with Migrations

CRUD

Updated internal/data/heroes.go

models.go

Updated main.go

Updated internal/data/heroes.go

Updated heroes.go

Generate Demo Data

Updated routes.go

Updated heroes.go

Updated internal/data/heroes.go

Updated heroes.go

Updated internal/data/heroes.go

Updated heroes.go

Updated routes.go

Updated internal/data/heroes.go

Updated heroes.go

Updated routes.go

Querying Data

Updated helpers.go

filters.go

Updated heroes.go

Updated routes.go

Structured Logging,
Additional Error Handling