The Enterprise is ready for gRPC
Robust features for the enterprise
For the best experience (and most up-to-date)
Almost 30 years in the sector
Mostly as Software Engineer
Web - 3D - Middleware - Mobile - Big Data
More recent as Architect
Data - SRE - Infrastructure
Community
Apache Beam contributor
OpenTelemetry Collector contributor
Collibra
Principal Systems Architect
Alex
Van Boxel
Alex
Van Boxel

Ceci n'est pas un gRPC talk

A high performance, open source universal RPC framework

Focus
Robust Binary Foundation
Domain Specific Language
Reuse and Integrations
Robust Binary Foundation
Base 128 Varints
0x96150 =Base 128 Varints
0x96150 =Base 128 Varints
continuation bit
0x96150 =Base 128 Varints
0x96150 =Base 128 Varints
0x96150 =Base 128 Varints
0x96150 =0x960x01TLV
message Test1 {
int32 a = 1;
}TLV
message Test1 {
int32 a = 1;
}tag
wire-type
value
TLV - Tag
message Test1 {
int32 a = 1;
}tag
wire-type
value
- tag ties to the schema
- 4 bit tags are optimal
- very large tags space
TLV - Wire Type (hint length)
message Test1 {
int32 a = 1;
}tag
wire-type
value
- length hint, or type says how to interpreter the value
- six wire types: LEN, VARINT, I32, I64, ...
TLV - Value
message Test1 {
int32 a = 1;
}tag
wire-type
value
0x960x010x08=
150TLV - Value (LEN wire type)
message Test1 {
int32 a = 1;
Test2 b = 2;
}tag
wire-type
value
0x08length
Unknown Fields
message Identity {
string first_name = 1;
string last_name = 2;
}message Identity {
string first_name = 1;
string last_name = 2;
}message Identity {
string first_name = 1;
string last_name = 2;
}producer
consumer
filter (on last_name)
Unknown Fields
message Identity {
string first_name = 1;
string last_name = 2;
}message Identity {
string first_name = 1;
string last_name = 2;
string social_security_id = 3;
SocialMedia social_media = 4;
}message Identity {
string first_name = 1;
string last_name = 2;
}producer
consumer
filter (on last_name)
Unknown Fields
message Identity {
string first_name = 1;
string last_name = 2;
string social_security_id = 3;
SocialMedia social_media = 4;
}message Identity {
string first_name = 1;
string last_name = 2;
string social_security_id = 3;
SocialMedia social_media = 4;
}message Identity {
string first_name = 1;
string last_name = 2;
}producer
consumer
filter (on last_name)
Unknown Fields
message Identity {
string first_name = 1;
string last_name = 2;
string social_security_id = 3;
SocialMedia social_media = 4;
}message Identity {
string first_name = 1;
string last_name = 2;
string social_security_id = 3;
SocialMedia social_media = 4;
}message Identity {
string first_name = 1;
string last_name = 2;
}producer
consumer
filter (on last_name)
FINAL
Descriptor
message Test1 {
int32 a = 1;
}google.protobuf.DescriptorProto0x42 0xAB 0xFF 0x7C
0xD5 0x3E 0x11 0x00
Descriptor - Source Embedding
public final class TestOuterClass {
static {
java.lang.String[] descriptorData = {
"\n8com/redacted/v1alpha1" +
"/file.proto\022&com.example.file.v1alpha1\032\037google/protobuf/time" +
"stamp.proto\1P\001b\006proto3"
};
}
}Server Reflection
syntax = "proto3";
package grpc.reflection.v1;
service ServerReflection {
// The reflection service is structured as a bidirectional stream, ensuring
// all related requests go to a single server.
rpc ServerReflectionInfo(stream ServerReflectionRequest)
returns (stream ServerReflectionResponse);
}
Test/Demo it out with an public reflection server: https://grpcb.in/
Nice UI (bruno): https://docs.usebruno.com/send-requests/grpc/grpc-request
Popular CLI: https://grpcurl.com/
Robust Binary Foundation
TLV
-
Parsable without schema (no registry required)
-
Enables adding and removing fields without braking the contract
Unknown Fields
-
Schemas can rollout in different order without impact
(Server) Reflection
-
Descriptors enable standardized reflection across languages
-
Standard ways of discovering remove services (debugging)
Binary vs JSON
$$$
Domain Specific Language
No Tool - Schema/PM First
// Service that can be used to push spans between one ... [redacted for slide]
service TraceService {
rpc Export(ExportTraceServiceRequest) returns (ExportTraceServiceResponse) {}
}
message ExportTraceServiceRequest {
// An array of ResourceSpans.
// For data coming from a single resource this array will typically contain one
// ... [redacted for slide]
repeated opentelemetry.proto.trace.v1.ResourceSpans resource_spans = 1;
}
message ExportTraceServiceResponse {
// The details of a partially successful export request.
//
// If the request is only partially accepted ... [redacted for slide]
ExportTracePartialSuccess partial_success = 1;
}
message ExportTracePartialSuccess {
// The number of rejected spans. [redacted for slide]
string error_message = 2;
}YAML/JSON is not PM friendly
openapi: 3.0.1
info:
title: OpenTelemetry OTLP Ingress, exposed through a tenant-environment host for ingesting telemetry from the front-end or edge.
version: 1.0.0
servers:
- url: /otlp
paths:
/v1/traces:
post:
description: Service that can be used to push spans between one Application instrumented with OpenTelemetry and a collector, or between a collector and a central collector (in this case spans are sent/received to/from multiple Applications).
operationId: ingestTracesForTenantEnvironment
responses:
"200":
description: The details of a export request.
summary: Ingest traces for a tenant environment from front-end or edge.
tags:
- OpenTelemetry
tags:
- description: OpenTelemetry provides a single, open source standard and a set of technologies to capture and export metrics, traces, and logs from your cloud-native applications and infrastructure.
name: OpenTelemetryDiff friendly

File Structure
example/domain/service/v1
service.proto
service FooService {
rpc Bar(BarRequest) returns (BarResponse) {}
}
message BarRequest {...}
message BarResponse {...}
baz.proto
message Baz {...}
waldo.proto
message Waldo {...}
example/domain
import
Common Messages
example/domain/service/v1
example/domain/common/v1
service.proto
service FooService {
rpc Bar(BarRequest) returns (BarResponse) {}
}
message BarRequest {...}
message BarResponse {...}
baz.proto
message Baz {...}
waldo.proto
message Waldo {...}
quux.proto
message Quux {...}
corge.proto
message Corge {...}
example/domain
import
import
Versioning - Pre Release
example/domain/service/v1
service.proto
baz.proto
waldo.proto
example/domain
example/domain/service/v1alpha1
service.proto
baz.proto
plugh.proto
NEVER import
Versioning - Braking
example/domain/service/v1
service.proto
baz.proto
waldo.proto
example/domain
example/domain/service/v2
service.proto
baz.proto
garply.proto
NEVER import
Versioning

Evolution by Tag
message Metric {
string name = 1;
string unit = 3;
IntGauge int_gauge = 4;
Gauge gauge = 5;
IntSum int_sum = 6;
Sum sum = 7;
}Primitives
message Metric {
string name = 1;
string unit = 3;
IntGauge int_gauge = 4;
Gauge gauge = 5;
IntSum int_sum = 6;
Sum sum = 7;
}Messages, Enums, ...
message Metric {
string name = 1;
string unit = 3;
IntGauge int_gauge = 4;
Gauge gauge = 5;
IntSum int_sum = 6;
Sum sum = 7;
}Inline Documentation
message Metric {
string name = 1;
// unit in which the metric value is reported. Follows the format
// described by http://unitsofmeasure.org/ucum.html.
string unit = 3;
IntGauge int_gauge = 4;
Gauge gauge = 5;
IntSum int_sum = 6;
Sum sum = 7;
}Deprecation
message Metric {
string name = 1;
// This field will be removed in ~3 months, on July 1, 2021.
IntGauge int_gauge = 4 [deprecated = true]
Gauge gauge = 5;
// This field will be removed in ~3 months, on July 1, 2021.
IntSum int_sum = 6 [deprecated = true]
Sum sum = 7;message Metric {
reserved 4, 6;
string name = 1;
Gauge gauge = 5;
Sum sum = 7;Semantic Rename
message ResourceLogs {
opentelemetry.proto.resource.v1.Resource resource = 1;
repeated InstrumentationLibraryLogs instrumentation_library_logs = 2;
}message ResourceLogs {
opentelemetry.proto.resource.v1.Resource resource = 1;
repeated ScopeLogs scope_logs = 2;
// This field is deprecated and will be removed after grace period expires
repeated InstrumentationLibraryLogs instrumentation_library_logs = 1000 [deprecated = true];
}API Guides
Google API Design Guide
Uber gRPC api guidelines
buf.build validator
Domain Specific Language
No tools required
-
Straightforward DSL enables technical program managers to drive the API
-
Any diffing UI will do (GitLab/GitHub)
Versioning Strategy
-
Common accepted versioning (package based) help define a versioning strategy
Evolution
-
Tagging (TLV) makes it easy to add and deprecated without breaking
-
Renaming field without braking binary compatibility
Reuse and Integration
Unified RPC and Messaging
example/domain/service/v1
example/domain/common/v1
service.proto
service FooService {
rpc Bar(BarRequest) returns (BarResponse) {}
}
message BarRequest {...}
message BarResponse {...}
baz.proto
message Baz {...}
waldo.proto
message Waldo {...}
quux.proto
message Quux {...}
corge.proto
message Corge {...}
example/domain
Unified RPC and Messaging
example/domain/service/v1
example/domain/common/v1
service.proto
service FooService {
rpc Bar(BarRequest) returns (BarResponse) {}
}
message BarRequest {...}
message BarResponse {...}
baz.proto
message Baz {...}
waldo.proto
message Waldo {...}
quux.proto
message Quux {...}
corge.proto
message Corge {...}
example/domain
Unified RPC and Messaging
example/domain/service/v1
example/domain/common/v1
service.proto
service FooService {
rpc Bar(BarRequest) returns (BarResponse) {}
}
message BarRequest {...}
message BarResponse {...}
baz.proto
message Baz {...}
waldo.proto
message Waldo {...}
quux.proto
message Quux {...}
corge.proto
message Corge {...}
example/domain
thud.proto
message Thud {...}
ProtoJSON
Protobuf supports a canonical encoding in JSON, making it easier to share data with systems that do not support the standard protobuf binary wire format.
ProtoJSON

gRPC/JSON - reusable options

gRPC/JSON - service scoped

gRPC/JSON - method scoped

gRPC/JSON support
The Monolith is Back
The Monolith is Back
Modular Monolith
Fused gRPC








Keep you options open








Reuse and Integration
Unified RPC and Messaging
-
Same tooling, same Data Objects
-
Less context for teams
ProtoJSON - gRPC/JSON
-
Integrate with external parties
Fused gRPC
-
Keep the monolith, but keep options open
-
Guides you to modularisation
-
Easier to test

The Enterprise is ready for gRPC
By Alex Van Boxel
The Enterprise is ready for gRPC
- 37