Developing a HTTP API

SHODO

Hello! I'm Nathan.

What about you?

The next four hours

Welcome to the "Developing a HTTP API" training

This presentation is in English. Many terms do not have a proper translation in French.

 

This is my early stages as Go trainer. If you have any question at any time, feel free to ask.

Slides

https://slides.com/nathancastelein/go-course-api/fullscreen

 

Code

https://github.com/nathancastelein/go-course-api

Schedule

Morning session

  • 9h: Session start
  • 10h: 5 minutes break
  • 11h: 10 minutes break
  • 12h: 5 minutes break
  • 13h: Session ends

Afternoon session

  • 14h: Session start
  • 15h: 5 minutes break
  • 16h: 10 minutes break
  • 17h: 5 minutes break
  • 18h: Session ends

What do you expect

from this training?

Content

  • Developing an API, the why
  • Our projet: Charlie's Shelter, a place for animals
  • Using a SQL database in Go
  • Writing a HTTP API

A four-hours training means ...

  • Something like 40% of theory, 60% of practice
  • A lot of resources will be provided to go further outside of the training session
  • Some topics are excluded from this training session
  • Sometimes, it's my point of view
  • I'm available for any question following the session: nathan.castelein@shodo-lille.io

Prerequisites

  • Go v1.21
  • Visual Studio Code
  • Git
  • (optional) Docker for PostgreSQL

Developing an API, the why

Developing an API, the why

Expose your services to your customers through a well-known protocol: HTTP.

Go is a solid candidate for writing micro-services. Developing performant APIs becomes a key point to propose this kind of architecture.

Charlie's Shelter, a place for animals

Charlie's Shelter

You're helping a friend in their dream: a shelter for old animals!
They need a bit of help to create their website.

You start by creating a database to store all information, and a HTTP API to give access to the stored information.

Charlie's Shelter

This project is based on the Swagger Petstore Sample.

In this project, we will learn:

  • How to work with a database
  • How to create a HTTP API
  • How to work with OpenAPI 3

Questions?

Thanks Lexica Aperture for generating wonderful images.

Using a SQL database in Go

Simple SQL schema

CREATE TABLE category (
    id SERIAL PRIMARY KEY,
    name TEXT UNIQUE NOT NULL    
);

CREATE TABLE pet (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    category INT REFERENCES category(id)
);

INSERT INTO category(name) VALUES ('Dog'), ('Cat'), ('Hamster'), ('Rabbit');

INSERT INTO pet(name, category) VALUES 
    ('Rafale', 1), 
    ('Rio', 1), 
    ('Mia', 2), 
    ('Laverne', 3);

Start the database with Docker

$ cd database
$ docker run -d --name gocourse-postgres \
    -v ./init.sql:/docker-entrypoint-initdb.d/init.sql \
    -e POSTGRES_PASSWORD=test \
    -e POSTGRES_USER=gocourse \
    -e POSTGRES_DB=gocourse \
    -p 5432:5432 \
    postgres:latest
a6fb78c839629d9cf71a2cf7fca9d6bfbbdd3233da2201d7170447d99a75a527

$ docker exec -it gocourse-postgres psql -U gocourse 
psql (16.1 (Debian 16.1-1.pgdg120+1))
Type "help" for help.

gocourse=# select * from pet;
 id |  name   | category 
----+---------+----------
  1 | Rafale  |        1
  2 | Rio     |        1
  3 | Mia     |        2
  4 | Laverne |        3
(4 rows)

Run a PostgreSQL database with the schema using Docker

Don't have Docker?

You received by mail a PostgreSQL DSN (data source name).

You can use it in the project by exporting it as an ENV variable:

$ export PG_DSN='...'

The database/sql package

Connect to the database: the standard database/sql package

Go provides a standard SQL package to work with a database.

First of all, let's see how we can connect to the database.

 

Useful link:

https://pkg.go.dev/database/sql

Write the connection to the database

Write the Connect function, to create a connection to the database.

file/to/open.go
$ go test .

File to open:

Test your code:

database/connect/connect.go
postgres://gocourse:test@localhost:5432/gocourse?sslmode=disable

Connection string to your database:

Connect to a database: using a driver

The sql package must be used in conjunction with a database driver.

The database driver implements the database interface to properly discuss with a given type of database.

As we are using PostgreSQL, we will use lib/pq, the famous driver for PostgreSQL: https://github.com/lib/pq

 

List of drivers: https://go.dev/wiki/SQLDrivers

Connect to the database: using a driver

package database

import (
	"database/sql"

	_ "github.com/lib/pq"
)

func ConnectSolution() (*sql.DB, error) {
	db, err := sql.Open("postgres", "postgres://gocourse:test@localhost:5432/gocourse?sslmode=disable")
	if err != nil {
		return nil, err
	}

	return db, db.Ping()
}

Blank identifier

Blank identifier _ can be used to import a package.

If you import a package and not using it in your code, it does not compile.

Blank identifier allows the import, and it will only call the init() function of the package.

In our case, lib/pq has an init function which registers the driver for database/sql.

The init function

// from lib/pq

func init() {
	sql.Register("postgres", &Driver{})
}

Init function is a simple function that will be executed before anything else.
You can't manage errors properly with the init function.

https://github.com/lib/pq/blob/master/conn.go

Using database/sql package

Now that we have a connection to the database, we will be able to perform queries on the database.

 

database/sql is a low-level package, so we don't have a lot of helpers to make things easier.

 

Let's have our first SELECT!

Exec vs. Query

Exec

https://pkg.go.dev/database/sql#DB.Exec

 

Exec executes a query without returning any rows.

DELETE, UPDATE, INSERT

 

Returns the number of affected rows.

Query

https://pkg.go.dev/database/sql#DB.Query

 

Query executes a query that returns rows.

SELECT

 

Returns the data.

A simple query

func SimpleQuery() (bool, error) {
	query := `SELECT 1 = 1;`

	db, err := database.ConnectSolution()
	if err != nil {
		return false, err
	}

	rows, err := db.Query(query)
	if err != nil {
		return false, err
	}

	for rows.Next() {
		var result bool
		if err := rows.Scan(&result); err != nil {
			return false, err
		}

		return result, nil
	}

	return false, errors.New("no data")
}

Writing your first SELECT

Write the SelectAllPetNames function.

 

Query: SELECT pet.name FROM pet;

file/to/open.go
$ go test -run ^TestSelectAllPetNames$

File to open:

Test your code:

database/sql/select.go

The first SELECT

func SelectAllPetNames() ([]string, error) {
	query := `SELECT pet.name FROM pet;`

	db, err := database.ConnectSolution()
	if err != nil {
		return nil, err
	}

	rows, err := db.Query(query)
	if err != nil {
		return nil, err
	}

	petNames := make([]string, 0)
	for rows.Next() {
		var petName string
		if err := rows.Scan(&petName); err != nil {
			return nil, err
		}

		petNames = append(petNames, petName)
	}

	return petNames, nil
}

Select pets and their category

package shelter

type Pet struct {
	Id       int
	Name     string
	Category string
}

Package shelter describes one struct to manage pets.

Let's select from database using those struct.

Complex SELECT

Write the SelectAllPets function.

file/to/open.go
$ go test -run ^TestSelectAllPets$

File to open:

Test your code:

database/sql/select.go
...
query := `SELECT 
		pet.id, 
		pet.name, 
		category.name 
		FROM pet 
		INNER JOIN category 
		ON pet.category = category.id;`
...

Complex SELECT

func SelectAllPetsSolution() ([]shelter.Pet, error) {
	query := `SELECT pet.id, pet.name, category.name FROM pet INNER JOIN category ON pet.category = category.id;`

	db, err := database.ConnectSolution()
	if err != nil {
		return nil, err
	}

	rows, err := db.Query(query)
	if err != nil {
		return nil, err
	}

	pets := make([]shelter.Pet, 0)
	for rows.Next() {
		pet := shelter.Pet{}
		if err := rows.Scan(&pet.Id, &pet.Name, &pet.Category); err != nil {
			return nil, err
		}

		pets = append(pets, pet)
	}

	return pets, nil
}

Passing parameters to a query

Both Exec and Query methods have a variadic parameter, to provide values to the query, while avoiding SQL injection.

Passing parameters to a query

func QueryWithArgs() {
	...
    query := `SELECT * FROM pet WHERE name = $1 OR name = $2;`
    
    rows, err := db.Query(query, "Rio", "Rafale")
    ...
}

Passing parameters to a query or an exec: the UPDATE

Write the UpdatePetName function to update the pet name for a given id.

file/to/open.go
$ go test -run ^TestUpdatePetName$

File to open:

Test your code:

database/sql/update.go
...
query := `UPDATE pet 
	SET name = $1 
    WHERE id = $2;`
...

Exec: the UPDATE example

func UpdatePetName(id int, newName string) error {
	query := `UPDATE pet SET name = $1 WHERE id = $2`

	db, err := connect.SQL()
	if err != nil {
		return err
	}

	result, err := db.Exec(query, newName, id)
	if err != nil {
		return err
	}

	rowsAffected, err := result.RowsAffected()
	if err != nil {
		return err
	}

	if rowsAffected != 1 {
		return errors.New("update failed, no rows affected")
	}

	return nil
}

A bit unfriendly, isn't it?

Using database/sql is a good way to start for simple usage, but it quickly becomes unfriendly.

If you want to go further:

https://go.dev/wiki/SQLInterface

 

Hopefully, the community provides packages to manage database discussions... with ORMs!

SQL ORMs

Go well-know ORMs

ORM Github # of stars
SQLx https://github.com/jmoiron/sqlx 15k
Gorm https://github.com/go-gorm/gorm 35k
ENT https://github.com/ent/ent 15k
SQLBoiler https://github.com/volatiletech/sqlboiler 6.5k
SQLc https://github.com/sqlc-dev/sqlc 10k

Go has many ORMs available on Github, but some of them have more success than others:

SQLx

sqlx: an extension to Go database/sql

SQLx is an extension to the database/sql package.

It provides helpers to easily unwrap data from SQL to your structures.

 

It provides quite the same API (Query, Exec), and some new functions:

  • Select, to select multiple rows
  • Get, to select a single row

SQLx

/*
	CREATE TABLE person (
      first_name text,
      last_name text,
      email text
	);
*/

type Person struct {
    FirstName string `db:"first_name"`
    LastName  string `db:"last_name"`
    Email     string `db:"email"`
}

The best way to use SQLx is to create a struct to welcome your data from the database. The struct has fields that correspond to the output of your SELECT.

Go struct tags

Struct tags are small pieces of metadata attached to fields of a struct that provide instructions to other Go code that works with the struct.

Struct tags are key-value pairs.

The most common one is the json tag.

type Person struct {
	Name string `db:"name" json:"name" example:"whatyouwant"`
}

SELECT with SQLx

func SelectWithSQLx() error {
	// Select multiple rows
	people := []Person{}
    if err := db.Select(&people, "SELECT * FROM person ORDER BY first_name ASC"); err != nil {
    	return err
    }
    
    fmt.Println("I found people: ", people)
    
    // Select a single row
    jason := Person{}
    if err := db.Get(&jason, "SELECT * FROM person WHERE first_name=$1", "Jason"); err != nil {
    	return err
    }
    
    fmt.Println("Hello you", jason)
}

SELECT with SQLx

Write the SelectAllPets function.

file/to/open.go
$ go test -run ^TestSelectAllPets$

File to open:

Test your code:

database/sqlx/select.go

SELECT with SQLx

type Pet struct {
	Id       int    `db:"id"`
	Name     string `db:"name"`
	Category string `db:"category"`
}

Select all pets with SQLx

func SelectAllPets() ([]shelter.Pet, error) {
	query := `SELECT pet.id as id, pet.name as name, category.name as category FROM pet INNER JOIN category ON pet.category = category.id;`

	db, err := connect.ConnectSQLx()
	if err != nil {
		return nil, err
	}

	pets := make([]Pet, 0)
	err = db.Select(&pets, query)
	if err != nil {
		return nil, err
	}

	shelterPets := make([]shelter.Pet, len(pets))
	for i, pet := range pets {
		shelterPets[i] = shelter.Pet{
			Id:       pet.Id,
			Name:     pet.Name,
			Category: pet.Category,
		}
	}

	return shelterPets, nil
}

Select one pet with SQLx

func SelectOnePet(id int) (*shelter.Pet, error) {
	query := `SELECT pet.id as id, pet.name as name, category.name as category FROM pet INNER JOIN category ON pet.category = category.id WHERE pet.id = $1;`

	db, err := connect.ConnectSQLx()
	if err != nil {
		return nil, err
	}

	pet := Pet{}
	err = db.Get(&pet, query, id)
	if err != nil {
		return nil, err
	}

	return &shelter.Pet{
		Id:       pet.Id,
		Name:     pet.Name,
		Category: pet.Category,
	}, nil
}

ORMs

ORMs: Code-first vs. Schema-first

Code-first

 

You write your structs/objects, you add some meta-data, then the database schema is created from your code.

Schema-first

 

You write your database schema, then your code (structs/objects) is generated from your schema.

Code-first ORMs

Gorm

Full-featured ORM. You write your structs, all links and constraints between them. Then you apply this on your database, and your schema is created according to your codebase.

Some of the features:

  • Associations
  • Hooks
  • Automigration
  • Transactions
  • ...

Gorm structure declaration

type User struct {
  ID        uint           `gorm:"primaryKey"`
  CreatedAt time.Time
  UpdatedAt time.Time
  DeletedAt gorm.DeletedAt `gorm:"index"`
  Name string
}

Declare a struct, and add tags to declare your schema.

Tags: https://gorm.io/docs/models.html#Fields-Tags

Gorm tags

Gorm usage

// Get first matched record
db.Where("name = ?", "Nathan").First(&user)
// SELECT * FROM users WHERE name = 'Nathan' ORDER BY id LIMIT 1;

// Get all matched records
db.Where("name <> ?", "Nathan").Find(&users)
// SELECT * FROM users WHERE name <> 'Nathan';

// Struct
db.Where(&User{Name: "Nathan", Age: 33}).First(&user)
// SELECT * FROM users WHERE name = "Nathan" AND age = 33 ORDER BY id LIMIT 1;


db.First(&user)

user.Name = "Roger"
user.Age = 33
db.Save(&user)
// UPDATE users SET name='Roger', age=33, updated_at = '2013-11-17 21:34:10' WHERE id=111;

Gorm limits

type User struct {
  gorm.Model
  Firstname string  `gorm:"default:Nathan"`
  Lastname  string
  CompanyID int
  Company   Company `gorm:"constraint:OnUpdate:CASCADE,OnDelete:SET NULL;"`
  FullName  string  `gorm:"->;type:GENERATED ALWAYS AS (concat(firstname,' ',lastname));default:(-);"`
}

Is the purpose of your Go code to dictate your SQL schema?

Gorm limits

Gorm is a magic box. It can be really powerful, but its code-first approach creates a huge dependency to the tool, and can easily result with poor database performances.

 

Moreover Gorm is kind of a black box, performing some requests you're not thinking of, sometimes not useful.

ENT

ENT example

// User schema.
type User struct {
    ent.Schema
}

// Fields of the user.
func (User) Fields() []ent.Field {
    return []ent.Field{
        field.Int("age"),
        field.String("name"),
        field.String("username").
            Unique(),
        field.Time("created_at").
            Default(time.Now),
    }
}

ENT features

func QueryUser(ctx context.Context, client *ent.Client) (*ent.User, error) {
    u, err := client.User.
        Query().
        Where(user.Name("Nathan")).
        // `Only` fails if no user found,
        // or more than 1 user returned.
        Only(ctx)
    if err != nil {
        return nil, fmt.Errorf("failed querying user: %w", err)
    }
    log.Println("user returned: ", u)
    return u, nil
}

Based on your description, ENT will generate your SQL schema, then your Go structures. You will then be able to use the generated code.

Schema-first ORM

SQLc: generate code from SQL requests

sqlc generates type-safe code from SQL. Here's how it works:

  • You write queries in SQL.
  • You run sqlc to generate code with type-safe interfaces to those queries.
  • You write application code that calls the generated code.

 

Works for Go, Kotlin, Python and Typescript. Mainly use database/sql package.

SQLc: https://play.sqlc.dev/

SQLBoiler: generate code from SQL schema

SQLBoiler comes with an approach to read the database schema, then generate code from it:

  • Write your schema (use tools like SQL migrate to manage migrations)
  • Launch SQLboiler tool
  • Use generated code to query the database

Why I recommend SQLBoiler?

It allows you to focus on your database schema first.

It follows OVHcloud database team way of working.

It's powerful and will provide good help to discuss with the database.

WILFRIED

APPROVES

Install and use SQLBoiler

$ go install github.com/volatiletech/sqlboiler/v4@latest

$ go install github.com/volatiletech/sqlboiler/v4/drivers/sqlboiler-psql@latest

$ sqlboiler -c sqlboiler.toml psql
# sqlboiler.toml configuration file
[psql]
dbname = "gocourse"
host = "localhost"
port = 5432
user = "gocourse"
pass = "test"
sslmode = "disable"

SQLBoiler generation

SQLBoiler generates a lot of files, for different tables and types.

 

It provides helper to query the database: https://github.com/volatiletech/sqlboiler?tab=readme-ov-file#query-building

SQLBoiler, some examples

// SELECT COUNT(*) FROM pilots;
count, err := models.Pilots().Count(ctx, db)

// SELECT * FROM "pilots" LIMIT 5;
pilots, err := models.Pilots(qm.Limit(5)).All(ctx, db)

// SELECT * FROM "pilots" WHERE "name"=$1;
pilots, err := models.Pilots(qm.Where("name=?", "Porco")).All(ctx, db)
// type safe version of above
pilots, err := models.Pilots(models.PilotWhere.Name.EQ("Porco")).All(ctx, db)

// SELECT * FROM "pilots" WHERE "name"=$1 LIMIT 1;
pilot, err := models.Pilots(models.PilotWhere.Name.EQ("Porco")).One(ctx, db)

// SELECT * FROM "pilots" WHERE id = $1;
pilot, err := models.FindPilot(ctx, db, 42)

models is the name of the generated package by SQLBoiler with structs and methods.

SELECT with SQLBoiler

Write the SelectAllPets and SelectOnePet functions.

file/to/open.go
$ go test -run ^TestSelectAllPets$

$ go test -run ^TestSelectOnePet$

File to open:

Test your code:

database/sqlboiler/select.go

Select with SQLBoiler

func SelectAllPets() ([]shelter.Pet, error) {
	ctx := context.Background()
	db, err := connect.Connect()
	if err != nil {
		return nil, err
	}

	pets, err := models.Pets().All(ctx, db)
	if err != nil {
		return nil, err
	}

	shelterPets := make([]shelter.Pet, len(pets))
	for i, pet := range pets {
		category, err := pet.PetCategory().One(ctx, db)
		if err != nil {
			return nil, err
		}
		shelterPets[i] = shelter.Pet{
			Id:       pet.ID,
			Name:     pet.Name,
			Category: category.Name,
		}
	}

	return shelterPets, nil
}

Select one pet

func SelectOnePet(id int) (shelter.Pet, error) {
	ctx := context.Background()
	db, err := connect.SQL()
	if err != nil {
		return shelter.Pet{}, err
	}

	pet, err := models.FindPet(ctx, db, id)
	if err != nil {
		return shelter.Pet{}, err
	}

	category, err := pet.PetCategory().One(ctx, db)
	if err != nil {
		return shelter.Pet{}, err
	}
	return shelter.Pet{
		Id:       pet.ID,
		Name:     pet.Name,
		Category: category.Name,
	}, nil
}

SQLBoiler update

func UpdatePetName(id int, newName string) error {
    ctx := context.Background()
	db, err := connect.SQL()
	if err != nil {
		return err
	}

	databasePet, err := models.FindPet(ctx, db, id)
	if err != nil {
		return err
	}

	databasePet.Name = newName
	_, err = databasePet.Update(ctx, db, boil.Infer())
	return err
}

SQLBoiler insert

func InsertNewPet(pet shelter.Pet) (shelter.Pet, error) {
	ctx := context.Background()
	db, err := connect.SQL()
	if err != nil {
		return shelter.Pet{}, err
	}

	category, err := models.Categories(models.CategoryWhere.Name.EQ(pet.Category)).One(ctx, db)
	if err != nil {
		return shelter.Pet{}, err
	}

	databasePet := models.Pet{
		Name:     pet.Name,
		Category: null.NewInt(category.ID, true),
	}

	if err := databasePet.Insert(ctx, db, boil.Infer()); err != nil {
		return shelter.Pet{}, err
	}

	return shelter.Pet{
		Id:       databasePet.ID,
		Name:     databasePet.Name,
		Category: category.Name,
	}, nil
}

SQL and null values

Go database/sql package provides NullX types to handle nullable values.

If your field is nullable in your database, then consider using the proper Null type to pass or get the field.

 

https://pkg.go.dev/database/sql#NullString

Choose the right ORM

Code-first

 

Make things easier for a PoC. Focus on your code, not on your database.

 

Hard to maintain on large codebase, pollute your code.

Bad experience with Gorm.

Schema-first

 

Focus on your database first, then on your code. You can keep your code clean from database-related information.

 

Take more time to get results.

Choose the right ORM

Take time to test a bit the different solutions. You can start with SQLx, then migrate to a more complex ORM if needed.

 

If you do so, I'd suggest to use SQLBoiler.

Writing a HTTP API

From the basics: net/http package

We now have a functional method to select data from the database. Let's use this method in a HTTP API.

 

Let's have a simple start, by using the standard net/http package.

We will see how we can encode data to JSON!

Write GET /pet

Write the ListPetsHandler function.

Expected output:

file/to/open.go
$ go run main.go

File to open:

Test your code:

api/nethttp/main.go

Then open

http://localhost:8080/swaggerui/

And execute GET /pet

[
    {
        "id": 2,
        "name": "Rio",
        "category": "Dog"
    },
    {
        "id": 3,
        "name": "Mia",
        "category": "Cat"
    }
    ...
]

Encode a struct in JSON

type Pet struct {
	Id       int    `json:"id"`
	Name     string `json:"name"`
	Category string `json:"category"`
}

You can add json tags to give the name of the field on the JSON output.

What happen if we lowercase the field name?

Return JSON

func ListPetsHandler(w http.ResponseWriter, r *http.Request) {
	pets, err := sqlboiler.SelectAllPets()
	if err != nil {
		fmt.Fprint(w, "an error occured")
		w.WriteHeader(http.StatusInternalServerError)
		return
	}

	jsonPets := make([]Pet, len(pets))
	for i, pet := range pets {
		jsonPets[i] = Pet{
			Id:       pet.Id,
			Name:     pet.Name,
			Category: pet.Category,
		}
	}

	jsonPetsBytes, err := json.MarshalIndent(jsonPets, "", "    ")
	if err != nil {
		fmt.Fprint(w, "an error occured")
		w.WriteHeader(http.StatusInternalServerError)
		return
	}

	w.WriteHeader(http.StatusOK)
	fmt.Fprint(w, string(jsonPetsBytes))
}

Insert a new pet

For this exercise, you will need to read the body of the request, and unmarshal it!

File to open:

api/nethttp/main.go
file/to/open.go
$ go run main.go

Test your code:

Then open

http://localhost:8080/swaggerui/

And execute POST /pet

Insert a new pet

type NewPet struct {
	Name     string `json:"name"`
	Category string `json:"category"`
}

func AddNewPetHandler(w http.ResponseWriter, r *http.Request) {
	body, err := io.ReadAll(r.Body)

	newPet := &NewPet{}
	err := json.Unmarshal(body, newPet)

	pet, err := sqlboiler.InsertNewPet(shelter.Pet{
		Name:     newPet.Name,
		Category: newPet.Category,
	})

	jsonPetsBytes, err := json.MarshalIndent(Pet{
		Name:     pet.Name,
		Category: pet.Category,
		Id:       pet.Id,
	}, "", "    ")

	w.WriteHeader(http.StatusOK)
	fmt.Fprint(w, string(jsonPetsBytes))
}

Routing to the proper handler

	http.HandleFunc("/pet", func(w http.ResponseWriter, r *http.Request) {
		if r.Method == http.MethodGet {
			ListPetsHandler(w, r)
			return
		}
		if r.Method == http.MethodPost {
			AddNewPetHandler(w, r)
			return
		}
	})

Some modifications are required to handle properly GET and POST!

net/http, a powerful standard package

net/http is a powerful package, providing you features to build your HTTP API.

But if you start to work on a huge project, you will notice that you will often write the same methods again and again for:

  • Encode and write results
  • Manage authentication and middlewares
  • ...

This is why I suggest to use web frameworks for your projects!

Web frameworks: work easier thanks to the community

Things can be easier with a web framework!

Web frameworks will provide helpers for:

  • HTTP routing
  • Binding input and output JSON, XML, form payload, ...
  • Error handling
  • Middlewares
  • ...

Most famous web frameworks

Over the years, multiple web frameworks became famous for Go.

Web Framework Github # of stars
Gin https://github.com/gin-gonic/gin 75k
Echo https://github.com/labstack/echo 28k
Chi https://github.com/go-chi/chi 17k
Iris https://github.com/kataras/iris 25k

The best one?

All of them, or none of them. No huge differences between them, there is no wrong choice.

 

All of them are compatible with the OpenAPI code generation tool.

 

In this session, we will use Echo!

Echo, a simple example

// Handler
func hello(c echo.Context) error {
  return c.String(http.StatusOK, "Hello, World!")
}

func main() {
  // Echo instance
  e := echo.New()

  // Routes
  e.GET("/", hello)

  // Start server
  e.Logger.Fatal(e.Start(":8080"))
}

Each web framework defines a way to declare handlers, and to register routes.

ListPetsHandler

Write an Echo handler to return all the pets, using sqlboiler.SelectAllPets function.

 

ATTENTION: don't forget to register your route in main function!

file/to/open.go
$ go run main.go

File to open:

Test your code:

api/echo/simple/main.go

Then open

http://localhost:8080/swaggerui/

And execute GET /pet

ListPetsHandler

func ListPetsHandler(c echo.Context) error {
	pets, err := sqlboiler.SelectAllPets()
	if err != nil {
		return c.String(http.StatusInternalServerError, "internal server error")
	}

	jsonPets := make([]Pet, len(pets))
	for i, pet := range pets {
		jsonPets[i] = Pet{
			Id:       pet.Id,
			Name:     pet.Name,
			Category: pet.Category,
		}
	}

	return c.JSON(http.StatusOK, jsonPets)
}

FindPetHandler

Write an Echo handler to return one pet by its id, using sqlboiler.SelectOnePet function.

file/to/open.go
$ go run main.go

File to open:

Test your code:

api/echo/simple/main.go

Then open

http://localhost:8080/swaggerui/

And execute GET /pet/{petId}

FindPetHandler

func FindPetHandler(c echo.Context) error {
	// Get id from URL path
	id := c.Param("id")
	idInt, err := strconv.Atoi(id)
	if err != nil {
		return c.String(http.StatusBadRequest, "id must be an int")
	}

	pet, err := sqlboiler.SelectOnePet(idInt)
	if err != nil {
		return c.String(http.StatusNotFound, "pet not found")
	}

	return c.JSON(http.StatusOK, &Pet{
		Id:       pet.Id,
		Name:     pet.Name,
		Category: pet.Category,
	})
}

Working with

OpenAPI 3

An API is a contract.

When you design an API, you write a contract with your customers.

You propose a set of endpoints and associated methods to use your services in a certain way.

This contract is a building block of your API.

A contract? Meet OpenAPI 3

The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to HTTP APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection.

 

An OpenAPI definition can then be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

 

https://swagger.io/specification/ 

Working with OpenAPI 3

Working with OpenAPI 3

In Go, you have two choices to work with OpenAPI 3...

  • Schema-first: Write your OpenAPI 3 file, then generate code and HTTP handlers from the schema
  • Code-first: Annotate your code with information related to your schema, then generate the OpenAPI 3 schema from your code

Schema-first approach

$ go install github.com/deepmap/oapi-codegen/v2/cmd/oapi-codegen@latest

$ oapi-codegen -package petshelter openapi.yaml > petshelter.gen.go

Thanks to this tool: https://github.com/deepmap/oapi-codegen

It generates structures for inputs and outputs, http handlers, and client boilerplate. It provides a lot of options for configuration.

 

Other tools:

Code-first approach

There are no viable approach for OpenAPI 3. But there are alternative working with OpenAPI 2: https://github.com/swaggo/swag

// ListAccounts godoc
// @Summary      List accounts
// @Description  get accounts
// @Tags         accounts
// @Accept       json
// @Produce      json
// @Param        q    query     string  false  "name search by q"  Format(email)
// @Success      200  {array}   model.Account
// @Failure      400  {object}  httputil.HTTPError
// @Failure      404  {object}  httputil.HTTPError
// @Failure      500  {object}  httputil.HTTPError
// @Router       /accounts [get]
func (c *Controller) ListAccounts(ctx *gin.Context) {
  q := ctx.Request.URL.Query().Get("q")
  accounts, err := model.AccountsAll(q)
  if err != nil {
    httputil.NewError(ctx, http.StatusNotFound, err)
    return
  }
  ctx.JSON(http.StatusOK, accounts)
}

Schema-first or code-first?

As I did for SQL ORMs, I'd definitely suggest to use schema-first approach.

You focus on your API contract, then you generate boilerplates and write your handlers.

 

OpenAPI 3 is a powerful specification, and a lot of tools can be helpful to work with your API. So I'd suggest to work on a proper OpenAPI 3 definition and generate what you need (boilerplates, tests, SDK, etc.)

Echo & OpenAPI 3

First of all, let's generate boilerplates using oapi-codegen

$ cd openapi

$ oapi-codegen -package openapi ./openapi.yaml > petshelter.gen.go

It generates a petshelter.gen.go file

oapi-codegen result

...
// ServerInterface represents all server handlers.
type ServerInterface interface {
	// Lists Pets
	// (GET /pet)
	ListPets(ctx echo.Context) error
	// Add a new pet to the shelter
	// (POST /pet)
	AddPet(ctx echo.Context) error
	// Finds Pets by categories
	// (GET /pet/findByCategories)
	FindPetsByCategories(ctx echo.Context, params FindPetsByCategoriesParams) error
	// Deletes a pet
	// (DELETE /pet/{petId})
	DeletePet(ctx echo.Context, petId int) error
	// Find pet by ID
	// (GET /pet/{petId})
	GetPetById(ctx echo.Context, petId int) error
	// Rename a pet
	// (POST /pet/{petId}/rename)
	RenamePetById(ctx echo.Context, petId int) error
}
...

Interfaces

An interface type is defined as a set of method signatures.

A value of interface type can hold any value that implements those methods.

 

Our goal now is to implement those methods!

ListPets

List all pets using sqlboiler.SelectAllPets function.

 

The newOpenAPIPet function is provided to create an openapi.Pet from a shelter.Pet.

File to open:

api/echo/codegen/server.go
file/to/open.go
$ go run *.go

Test your code:

Then open

http://localhost:8080/swaggerui/

And execute GET /pet

ListPets

func (s *Server) ListPets(ctx echo.Context) error {
	pets, err := sqlboiler.SelectAllPets()
	if err != nil {
		return ctx.String(http.StatusInternalServerError, "internal server error")
	}

	jsonPets := make([]openapi.Pet, len(pets))
	for i, pet := range pets {
		jsonPets[i] = newOpenAPIPet(pet)
	}

	return ctx.JSON(http.StatusOK, jsonPets)
}

FindPetsByCategories

List pets filtered by categories using sqlboiler.SelectByCategories.

File to open:

api/echo/codegen/server.go
file/to/open.go
$ go run *.go

Test your code:

Then open

http://localhost:8080/swaggerui/

And execute GET /pet/findByCategories

FindPetsByCategories

func (s *Server) FindPetsByCategories(ctx echo.Context, params openapi.FindPetsByCategoriesParams) error {
	pets, err := sqlboiler.SelectByCategories(params.Categories...)
	if err != nil {
		return ctx.String(http.StatusInternalServerError, "internal server error")
	}

	jsonPets := make([]openapi.Pet, len(pets))
	for i, pet := range pets {
		jsonPets[i] = newOpenAPIPet(pet)
	}

	return ctx.JSON(http.StatusOK, jsonPets)
}

GetPetById

Get one pet using sqlboiler.SelectOnePet function.

File to open:

api/echo/codegen/server.go
file/to/open.go
$ go run *.go

Test your code:

Then open

http://localhost:8080/swaggerui/

And execute GET /pet/1

GetPetById

func (s *Server) GetPetById(ctx echo.Context, petId int) error {
	pet, err := sqlboiler.SelectOnePet(petId)
	if err != nil {
		return ctx.String(http.StatusNotFound, "pet not found")
	}

	return ctx.JSON(http.StatusOK, newOpenAPIPet(*pet))
}

AddPet

Add a new pet using sqlboiler.InsertNewPet function.

File to open:

api/echo/codegen/server.go
file/to/open.go
$ go run *.go

Test your code:

Then open

http://localhost:8080/swaggerui/

And execute POST /pet

AddPet

func (s *Server) AddPet(ctx echo.Context) error {
	addPet := openapi.AddPetJSONRequestBody{}
	if err := ctx.Bind(&addPet); err != nil {
		return ctx.String(http.StatusBadRequest, "fail to read body")
	}

	pet, err := sqlboiler.InsertNewPet(&shelter.Pet{
		Name:     addPet.Name,
		Category: addPet.Category,
	})
	if err != nil {
		return ctx.String(http.StatusInternalServerError, "fail to insert pet")
	}

	return ctx.JSON(http.StatusOK, newOpenAPIPet(*pet))
}

DeletePet

Delete one pet using sqlboiler.DeleteOnePet function.

File to open:

api/echo/codegen/server.go
file/to/open.go
$ go run *.go

Test your code:

Then open

http://localhost:8080/swaggerui/

And execute DELETE /pet/4

DeletePet

func (s *Server) DeletePet(ctx echo.Context, petId int64) error {
	err := sqlboiler.DeleteOnePet(int(petId))
	if err != nil {
		return ctx.String(http.StatusBadRequest, "deletion failed")
	}
	return ctx.NoContent(http.StatusOK)
}

RenamePetById

Rename a pet using sqlboiler.UpdatePet function.

File to open:

api/echo/codegen/server.go
file/to/open.go
$ go run *.go

Test your code:

Then open

http://localhost:8080/swaggerui/

And execute POST /pet/{petId}/rename

RenamePetById

func (s *Server) RenamePetById(ctx echo.Context, petId int) error {
	updatePet := openapi.RenamePetByIdJSONRequestBody{}
	if err := ctx.Bind(&updatePet); err != nil {
		return ctx.String(http.StatusBadRequest, "fail to read body")
	}

	if err := sqlboiler.UpdatePetName(petId, updatePet.Name); err != nil {
		return ctx.String(http.StatusInternalServerError, "fail to update pet")
	}

	pet, err := sqlboiler.SelectOnePet(petId)
	if err != nil {
		return ctx.String(http.StatusInternalServerError, "fail to select pet")
	}

	return ctx.JSON(http.StatusOK, newOpenAPIPet(pet))
}

Middleware

Middleware

A middleware in a web framework is a piece of code that will be called on each request.

In a middleware, you will be able to perform actions before and/or after each request, stop the request, etc.

Common usages:

  • Logs
  • Authentication
  • Timeout
  • Prometheus
  • ...

Middleware

Let's write a logger middleware which print the time elapsed for a request to be performed.

 

Expected output:

file/to/open.go
$ go run *.go

File to open:

Test your code:

api/echo/codegen/middleware.go

Then open

http://localhost:8080/swaggerui/

And execute GET /pet and check output in the console

time elapsed: 32.685875ms

Middleware

func LogTime(next echo.HandlerFunc) echo.HandlerFunc {
	return func(c echo.Context) error {
		start := time.Now()

		if err := next(c); err != nil {
			c.Error(err)
		}

		elapsed := time.Since(start)

		c.Logger().Printf("time elapsed: %s", elapsed)
		return nil
	}
}

What's next?

Complete usage of the standard library:

https://www.jetbrains.com/guide/go/tutorials/rest_api_series/stdlib/

A simple tutorial with Go and Gin: https://go.dev/doc/tutorial/web-service-gin

Learn Echo's features: https://echo.labstack.com/docs

Dive into OpenAPI 3: https://swagger.io/specification/

 

Ask me if you have any questions!

Thanks! Questions?

https://bento.me/nathan-castelein

Developing a HTTP API in Go - v1

By Nathan Castelein

Developing a HTTP API in Go - v1

  • 36

More from Nathan Castelein