Building gRPC powered REST APIs in Go

Tsvetan Dimitrov

I'm hungryyyy....Dm

API

  • Architecture - MVC

  • Data exchange format - JSON

  • Transport protocol - HTTP 1.1

  • Some spices - MVC frameworks

Agenda

  • HTTP/JSON definition

  • Data Exchange

    • Protocol Buffers

  • Transport

    • gRPC and HTTP/2

  • Some spices

    • gRPC-gateway

  • Architecture

    • Onion vs Layered

    • go-kit web toolkit

  • Demo Application

HTTP/JSON

What do we mean?

  • A RESTful interface with JSON payloads

  • The standard for service communication

  • Easy to read and write

  • Native Javascript support

Disadvantages of JSON

  • Hard to write

    • no trailing commas

    • no comments

  • Inefficient on the wire

  • Expensive marshaling

  • Poorly typed

  • No single source of truth for the API (Swagger?)

Protocol Buffers

Definition

  • Serialization of structured data for use in communication protocols, data storage and more

  • Language agnostic (generators for different languages)

syntax = "proto3";

package example;


message Student {
    string student_id = 1;
    string first_name = 2;
    string last_name = 3;
    string address = 4;
    bool is_joined = 5;
}

PB Language Version

Package of generated code

Field Tag

Field Type

Field Name

Message Type

Definition / 1

  • gRPC Remote Procedure Calls

  • RPC Framework

  • Open sourced version of Stubby RPC used at Google

RMI

CORBA

Definition / 2

  • Abstractions and best practices on how to design RPCs

  • Extension points to plug custom implementations and modifications

  • Supports 10+ programming languages

Features / 1

  • Serialization with Protocol Buffers 

service SearchService {
	rpc Search (SearchRequest) returns (SearchResponse);
}

  • HTTP / 2

Features / 2

  • Supports bi-directional streaming 

service SearchService {
	rpc Search (stream SearchRequest) returns (stream SearchResponse);
}

The problem with novel tech

  • Existing services/clients

  • Company unwillingness to invest in new tech

  • Public API expectation is HTTP/JSON

gRPC-gateway

Definition

  • Protoc plugin

  • Translates HTTP/JSON requests to gRPC and back

  • Simple annotation in .proto files

  • Provides Swagger/OpenAPI generator

  • Widely used within the gRPC community

service CustomerService {
    rpc RegisterCustomer (RegisterCustomerRequest) returns (RegisterCustomerResponse) {
    	option (google.api.http) = {
    		post: "/customer"
    		body: "*"
        };
    	option (grpc.gateway.protoc_gen_swagger.options.openapiv2_operation) = {
    		summary: "Register a new Customer"
    		description: "Register a new Customer"
    		tags: "customer"
       	};
    }
    rpc GetCustomerByID (GetCustomerByIDRequest) returns (GetCustomerByIDResponse) {
    	option (google.api.http) = {
        	get: "/customer/{customer_id}"
        };
        option (grpc.gateway.protoc_gen_swagger.options.openapiv2_operation) = {
            summary: "Get an existing customer"
            description: "Get an existing customer"
            tags: "customer"
        };
    }
}

HTTP Request Metadata

Swagger Metadata

example.ProfileService.Update
	user_id: "123"
	email: "foo@example.com"
    ...
PUT /v1/user/123/profile
...

{"email": "foo@example.com", ...}

product_service.proto

generates proxy

generates stub

Reverse Proxy

Your gRPC service

API Client

grpc-gateway

protoc

gRPC

REST API (JSON over HTTP)

Onion vs Layered Architecture

Layered Architecture

Dependency Inversion Principle

Abstractions should not depend upon details. Details should depend upon abstractions.

Onion Architecture

Go-kit

Overview

  • Microservices Toolkit

  • High Separation of Concerns

  • "Few opinions, lightly held"

  • Transports, Endpoints, Services

  • Extensible via Middleware

Features

  • Built-in HTTP and gRPC transports

  • Built-in Middlewares:

    • Auth

    • Logs

    • Metrics

    • Rate Limiting

    • Service Discovery

    • Distributed Tracing

Go-kit Onion Architecture

Reservation System Demo Application

Overview

  • Booking system

  • gRPC powered REST API

  • CRUD operations

  • Database: SQLITE

Entity Relationship Diagram

Talk is cheap, show me the code?!?!!

Summary

  • A new way to develop APIs

  • Used a transport agnostic architecture

  • Defined RPC communication

  • Introduced Protocol Buffers

  • Introduced gRPC

  • Implemented a hybrid approach between HTTP/JSON and gRPC

Questions?