Let The Tools Do The Work

Marwan Sulaiman

The New York Times

Go NYC July 18th, 2019

Agenda

  • Examine net/http
  • Explore current tools in the open source
  • protoc-gen-go
  • protoc-gen-twirp
  • gqlgen
  • protoc-gen-twirpql

 

Assumptions

  • If you write Go, you probably write servers
  • If you write servers, you work at a company
  • That company probably runs multiple servers
  • Multiple Servers probably means multiple teams.

What does that look like? 

Challenges?

- Communication

- Documentation

- Plumbing

How do we write servers in Go?

net/http

What net/http brings

  • Simplicity

  • Production readiness

  • Pluggability (i.e. gorilla/mux)

What net/http does not bring (and it shouldn't)

  • Documentation

  • Plumbing

  • Interactiveness

Documentation

How do your clients discover your API?

Look at the code

OpenAPI (Swagger)

{
  "swagger": "2.0",
  "info": {
    "title": "Search API",
    "version": "1.0",
    "contact": {
      "name": "Search Team",
      "email": "searchteam@product.com"
    }
  },
  "host": "search.product.com",
  "schemes": [
    "http",
    "https"
  ],
  "consumes": [
    "application/json"
  ],
  "produces": [
    "application/json"
  ],
  "paths": {
    "/search": {
      "get": {
        "summary": "search allows you to search by a keyword",
        "operationId": "Search",
        "responses": {
          "200": {
            "description": "A successful response.",
            "schema": {
              "$ref": "#/definitions/SearchResult"
            }
          },
          "400": {
            "description": "missing query"
          }
        },
        "parameters": [
          {
            "name": "q",
            "in": "query",
            "required": true,
            "type": "string"
          }
        ],
        "tags": [
          "Service"
        ]
      }
    }
  },
  "definitions": {
    "SearchResult": {
      "type": "object",
      "properties": {
        "Sites": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "Images": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "Videos": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    }
  }
}

What OpenAPI Offers

  • Contract only ✅

  • Language agonistic ✅

  • Type information ✅

  • Friendly UI ✅

What OpenAPI does not offer (and it shouldn't)

  • Automation ❌

  • Plumbing 😕

  • Interactiveness 🤷🏻‍♂️

How do we fix automation?

  • Write the contract once

  • Generate everything else from it

  • Ensure contracts never get outdated

Protocol Buffers

  • Interface Definition Language (IDL)

  • You start with the documentation

  • Uses plugins to generate code

service Index {
    rpc Search(SearchRequest) returns (SearchResponse);
}

message SearchRequest {
    string query = 1;
}

message SearchResponse {
    repeated string URLS = 1;
    repeated string images = 2;
    repeated string videos = 3;
}

~ go get github.com/golang/protobuf/protoc-gen-go
~ protoc --go_out=. service.proto
type SearchRequest struct {
    Query string `json:"query"`
}

type SearchResponse struct {
    URLS   []string `json:"URLS"`
    Images []string `json:"images"`
    Videos []string `json:"videos"`
}

~ go get github.com/twitchtv/twirp/protoc-gen-twirp
~ protoc --go_out=. --twirp_out=. service.proto
type Index interface {
    Search(context.Context, *SearchRequest) (*SearchResponse, error)
}


func NewIndexServer(Index) http.Handler {}

func NewIndexJSONClient(addr string) Index {}

func NewIndexProtobufClient(addr string) Index {}


~ go get github.com/elliots/protoc-gen-twirp_swagger
~ protoc --twirp_swagger_out=. service.proto
{
  "swagger": "2.0",
  "info": {
    "title": "service.proto",
    "version": "version not set"
  },
  "schemes": [
    "http",
    "https"
  ],
  "consumes": [
    "application/json"
  ],
  "produces": [
    "application/json"
  ],
  "paths": {
    "/twirp/company.indexteam.Index/Search": {
      "post": {
        "operationId": "Search",
        "responses": {
          "200": {
            "description": "A successful response.",
            "schema": {
              "$ref": "#/definitions/indexteamSearchResponse"
            }
          }
        },
        "parameters": [
          {
            "name": "body",
            "in": "body",
            "required": true,
            "schema": {
              "$ref": "#/definitions/indexteamSearchRequest"
            }
          }
        ],
        "tags": [
          "Index"
        ]
      }
    }
  },
  "definitions": {
    "indexteamSearchRequest": {
      "type": "object",
      "properties": {
        "query": {
          "type": "string"
        }
      }
    },
    "indexteamSearchResponse": {
      "type": "object",
      "properties": {
        "URLS": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "images": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "videos": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    }
  }
}

What was accomplished so far

  • Automation ✅

  • Plumbing ✅

  • Interactiveness ❌

Interactiveness

cURL

~ curl -H "Content-Type: application/json" -d '{"query": "Go"}' \
    'localhost:8080/twirp/company.indexteam.Index/Search'

Generated Client

package main

import (
	"companyindex"
	"context"
	"fmt"
	"net/http"
)

func main() {
	idx := companyindex.NewIndexProtobufClient(
            "http://localhost:8080",
            http.DefaultClient,
        )
	resp, err := idx.Search(context.Background(), &companyindex.SearchRequest{
		Query: "Go",
	})
	if err != nil {
		panic(err)
	}
	fmt.Println(resp)
}

Graph(i)QL

  • Assuming you have a GraphQL layer

  • Graph(i)QL turns your API into an IDE

TwirpQL

What TwirpQL brings

  • Generates a full GraphQL server

  • Provides a Graph(i)QL endpoint

Given a Protocol Buffer file:

How it works

  • Translates .proto file into a valid GraphQL schema.

  • Binds Schema Types to Generated Go Types

  • Creates a Handler that uses gqlgen under the hood

~ go get marwan.io/protoc-gen-twirpql
~ protoc --twirpql_out=. service.proto
func Handler(service Index) http.Handler {}

func Playground(title, endpoint string) http.Handler {}

Generate everything!

  • Swagger Documentation

  • Clients in Go/Ruby/Java and more

  • REST Server

  • GraphQL Server

  • Graph(i)QL UI

🧚🏻‍♂️ Demo 🧚🏻‍♂️

Conclusion

  • IDLs help you maintain one SST

  • Let the tools do the plumbing

Thank you

https://twirpql.dev

Let The Tools Do The Work

By marwansameer

Made with Slides.com

Let The Tools Do The Work

  • 866

More from marwansameer