Let The Tools Do The Work
Marwan Sulaiman
The New York Times
Go NYC July 18th, 2019
Agenda
- Examine net/http
- Explore current tools in the open source
- protoc-gen-go
- protoc-gen-twirp
- gqlgen
- protoc-gen-twirpql
Assumptions
- If you write Go, you probably write servers
- If you write servers, you work at a company
- That company probably runs multiple servers
- Multiple Servers probably means multiple teams.
What does that look like?
Challenges?
- Communication
- Documentation
- Plumbing
How do we write servers in Go?
net/http
What net/http brings
-
Simplicity
-
Production readiness
-
Pluggability (i.e. gorilla/mux)
What net/http does not bring (and it shouldn't)
-
Documentation
-
Plumbing
-
Interactiveness
Documentation
How do your clients discover your API?
Look at the code
OpenAPI (Swagger)
{
"swagger": "2.0",
"info": {
"title": "Search API",
"version": "1.0",
"contact": {
"name": "Search Team",
"email": "searchteam@product.com"
}
},
"host": "search.product.com",
"schemes": [
"http",
"https"
],
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"paths": {
"/search": {
"get": {
"summary": "search allows you to search by a keyword",
"operationId": "Search",
"responses": {
"200": {
"description": "A successful response.",
"schema": {
"$ref": "#/definitions/SearchResult"
}
},
"400": {
"description": "missing query"
}
},
"parameters": [
{
"name": "q",
"in": "query",
"required": true,
"type": "string"
}
],
"tags": [
"Service"
]
}
}
},
"definitions": {
"SearchResult": {
"type": "object",
"properties": {
"Sites": {
"type": "array",
"items": {
"type": "string"
}
},
"Images": {
"type": "array",
"items": {
"type": "string"
}
},
"Videos": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
}
What OpenAPI Offers
-
Contract only ✅
-
Language agonistic ✅
-
Type information ✅
-
Friendly UI ✅
What OpenAPI does not offer (and it shouldn't)
-
Automation ❌
-
Plumbing 😕
-
Interactiveness 🤷🏻♂️
How do we fix automation?
-
Write the contract once
-
Generate everything else from it
-
Ensure contracts never get outdated
Protocol Buffers
-
Interface Definition Language (IDL)
-
You start with the documentation
-
Uses plugins to generate code
service Index {
rpc Search(SearchRequest) returns (SearchResponse);
}
message SearchRequest {
string query = 1;
}
message SearchResponse {
repeated string URLS = 1;
repeated string images = 2;
repeated string videos = 3;
}
~ go get github.com/golang/protobuf/protoc-gen-go
~ protoc --go_out=. service.prototype SearchRequest struct {
Query string `json:"query"`
}
type SearchResponse struct {
URLS []string `json:"URLS"`
Images []string `json:"images"`
Videos []string `json:"videos"`
}
~ go get github.com/twitchtv/twirp/protoc-gen-twirp
~ protoc --go_out=. --twirp_out=. service.prototype Index interface {
Search(context.Context, *SearchRequest) (*SearchResponse, error)
}
func NewIndexServer(Index) http.Handler {}
func NewIndexJSONClient(addr string) Index {}
func NewIndexProtobufClient(addr string) Index {}
~ go get github.com/elliots/protoc-gen-twirp_swagger
~ protoc --twirp_swagger_out=. service.proto{
"swagger": "2.0",
"info": {
"title": "service.proto",
"version": "version not set"
},
"schemes": [
"http",
"https"
],
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"paths": {
"/twirp/company.indexteam.Index/Search": {
"post": {
"operationId": "Search",
"responses": {
"200": {
"description": "A successful response.",
"schema": {
"$ref": "#/definitions/indexteamSearchResponse"
}
}
},
"parameters": [
{
"name": "body",
"in": "body",
"required": true,
"schema": {
"$ref": "#/definitions/indexteamSearchRequest"
}
}
],
"tags": [
"Index"
]
}
}
},
"definitions": {
"indexteamSearchRequest": {
"type": "object",
"properties": {
"query": {
"type": "string"
}
}
},
"indexteamSearchResponse": {
"type": "object",
"properties": {
"URLS": {
"type": "array",
"items": {
"type": "string"
}
},
"images": {
"type": "array",
"items": {
"type": "string"
}
},
"videos": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
}
What was accomplished so far
-
Automation ✅
-
Plumbing ✅
-
Interactiveness ❌
Interactiveness
cURL
~ curl -H "Content-Type: application/json" -d '{"query": "Go"}' \
'localhost:8080/twirp/company.indexteam.Index/Search'Generated Client
package main
import (
"companyindex"
"context"
"fmt"
"net/http"
)
func main() {
idx := companyindex.NewIndexProtobufClient(
"http://localhost:8080",
http.DefaultClient,
)
resp, err := idx.Search(context.Background(), &companyindex.SearchRequest{
Query: "Go",
})
if err != nil {
panic(err)
}
fmt.Println(resp)
}
Graph(i)QL
-
Assuming you have a GraphQL layer
-
Graph(i)QL turns your API into an IDE
TwirpQL
What TwirpQL brings
-
Generates a full GraphQL server
-
Provides a Graph(i)QL endpoint
Given a Protocol Buffer file:
How it works
-
Translates .proto file into a valid GraphQL schema.
-
Binds Schema Types to Generated Go Types
-
Creates a Handler that uses gqlgen under the hood
~ go get marwan.io/protoc-gen-twirpql
~ protoc --twirpql_out=. service.protofunc Handler(service Index) http.Handler {}
func Playground(title, endpoint string) http.Handler {}Generate everything!
-
Swagger Documentation
-
Clients in Go/Ruby/Java and more
-
REST Server
-
GraphQL Server
-
Graph(i)QL UI
🧚🏻♂️ Demo 🧚🏻♂️
Conclusion
-
IDLs help you maintain one SST
-
Let the tools do the plumbing