Protocol buffers

Protocol Buffers are a way of encoding structured data in an efficient yet extensible format.

Google designed and used since 2001. (proto1)
Open-sourced in 2008. (proto2)
A language neutral, platform neutral, extensible way of serializing structured data.

JSON

expensive & massive

unclear type
readable

self-contained

portable

light weight & faster

strong type

unreadble

self-describe

need parser

PRoto buffers

vs

benchmarks

Speed

Size

[+] Setup

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

install Homebrew

brew install automake
brew install libtool
brew install protobuf

install protobuf

[+] Steps

Step 1 (SCHEMA)
Define structured data format by creating a .proto file.
Step 2 (IMPLEMENTATIONS)
Generated interface code from a .proto file for serializing/ deserializing the protobuf.
Step 3
Use the interface from the generated code to exchange your message.

[+] schema

The definition of protobuf to serialize/deserialize the data.
Strong type
Nested & Importing
type & name-value pairs
Each type has one or more unique number for binary mapping.

(.proto)

syntax = "proto3";

package Ecommerce;

message Product {
    bool available = 1;
    string name = 2;   
    string desc = 3;   
    float price = 4;   
    repeated Item item = 5;
}

message Item {
    string title = 1;
    int32 number = 2;
}

[-] .proto

syntax = "proto3";

package Ecommerce;

message Product {
    bool available = 1;
    string name = 2;   
    string desc = 3;   
    float price = 4;   
    repeated Item item = 5;
}

message Item {
    string title = 1;
    int32 number = 2;
}

version type

message collection

field's number

field's name

field's type

message type (object)

[+] Implementations

(INTERFACE)

protoc -I=$SRC_DIR --$FORMAT=$DST_DIR $SRC_DIR/xxx.proto
/*
    $SRC_DIR=source directory,
    $DST_DIR=target directory,
    $FORMAT=output language (cpp_out, java_out...)
*/

The static interface of protobuf implementation.
Support: Python, C#, C++, Java, Go, PHP, Ruby...

syntax = "proto3";

package Ecommerce;

message Product {
    bool available = 1;
    string name = 2;   
    string desc = 3;   
    float price = 4;   
    repeated Item item = 5;
}

message Item {
    string title = 1;
    int32 number = 2;
}

// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: test.proto

#include "test.pb.h"
#include <algorithm>
#include <google/protobuf/stubs/common.h>
#include <google/protobuf/stubs/port.h>
#include ...
// @@protoc_insertion_point(includes)
namespace Ecommerce {
class ProductDefaultTypeInternal {
 public:
  ::google::protobuf::internal::...
      _instance;
} _Product_default_instance_;
...

// Generated by the protocol buffer compiler.  DO NOT EDIT!
// source: test.proto

#ifndef PROTOBUF_test_2eproto__INCLUDED
#define PROTOBUF_test_2eproto__INCLUDED
#include <google/protobuf/stubs/common.h>
#include <google/protobuf/io/coded_stream.h>
#include ...
// @@protoc_insertion_point(includes)

namespace Ecommerce {
class Item;
class ...
}  // namespace Ecommerce
...

test.proto

test.pb.cc

test.pb.h

[-] INTERFACE

[+] PROS & CONS

Multiple language support
Self-describing
Simpler & Smaller & Faster
RPC support
Structure validation

Unreadable
Lack self-contained
Smaller community

👿

😇

protobuf.js

pure JavaScript implementation with TypeScript support for node.js and the browser.

better performance, better documentation

node.js

npm install -g protobufjs

[+] Setup

install protobufJS

npm install protobufjs [--save --save-prefix=~]
var protobuf = require("protobufjs");

browsers

// Development
<script src="//cdn.rawgit.com/dcodeIO/protobuf.js/6.X.X/dist/protobuf.js"></script>
// Production
<script src="//cdn.rawgit.com/dcodeIO/protobuf.js/6.X.X/dist/protobuf.min.js"></script>

[+] Implementations

(INTERFACE)

# ComoonJS
pbjs -t static-module -w commonjs -o $DST_DIR/xxx-common.js $SRC_DIR/xxx.proto
# ES6
pbjs -t static-module -w es6 -o $DST_DIR/xxx-es6.js $SRC_DIR/xxx.proto
# json
pbjs -t json -o $DST_DIR/xxx.json $SRC_DIR/xxx.proto
# json-module
pbjs -t json-module -o $DST_DIR/xxx-json.js $SRC_DIR/xxx.proto

/*
    $SRC_DIR=source directory,
    $DST_DIR=target directory,
    $FORMAT=output language (cpp_out, java_out...)
*/

[+] usage

import ProtoBuf from 'protobufjs';
import proto from './schema/protos/xxx.proto';

ProtoBuf.load(proto, function(err, root) {
    if (err)
        throw err;
    const Product = root.lookup('Ecommerce.Product');
        
    const result = Product.decode(data);
    console.log(result);
});

import { Ecommerce } from './interface/test-es6';

const result = Ecommerce.Product.decode(data);
console.log(result);

Reflection

static code

[-] compare

Source	Library	Advantages	Tradeoffs
.proto (Reflection)	full	Easily editable Interoperability with other libraries No compile step	Some parsing and possibly network overhead
JSON (Reflection)	light	Easily editable No parsing overhead Single bundle (no network overhead)	protobuf.js specific Has a compile step
static (Static code)	minimal	Works where eval access is restricted Fully documented Small footprint for small protos	Can be hard to edit No reflection Has a compile step

[+] flow

interface

message

encode

decode

binary

.proto

interface

message

protoc-gen-doc

Documentation generator plugin for the Protocol Buffers.

Support proto2 and proto3.
Support multiple form of document. (HTML, JSON, DocBook and Markdown)
Simple.
(Based on comments)

[+] Getting started

docker pull pseudomuto/protoc-gen-doc

install docker &

docker run --rm \
-v $DST_DIR:/out \
-v $SRC_DIR:/protos \
pseudomuto/protoc-gen-doc --doc_opt=$FORMAT,$FILE
/*
    $SRC_DIR=source directory,
    $DST_DIR=target directory,
    $FORMAT=output form (Markdown, HTML, JSON, DocBook)
    $FILE=the output file (xxx.md, xxx.html...)
*/

usage

POGOProtos

A central repository for all proto files of PokémonGO.

syntax = "proto3";
package POGOProtos.Map.Pokemon;

import "POGOProtos/Enums/PokemonId.proto";
import "POGOProtos/Data/PokemonDisplay.proto";

message NearbyPokemon {
  .POGOProtos.Enums.PokemonId pokemon_id = 1;
  float distance_in_meters = 2;
  fixed64 encounter_id = 3;
  string fort_id = 4;
  string fort_image_url = 5;
  .POGOProtos.Data.PokemonDisplay pokemon_display = 6;
}

what next?

an open source remote procedure call (RPC) system

Protocol Buffers

OVERVIEW

PROTOCOL BUFFERS

PROTOBUF.JS

PROTOC-GEN-DOC

WHAT ​NEXT?

Protocol Buffers are a way of encoding structured data in an efficient yet extensible format.

JSON

expensive & massive

unclear type readable

self-contained

portable

light weight & faster

strong type

unreadble

self-describe

need parser

PRoto buffers

vs

Speed

Size

[+] Setup

[+] Steps

[+] schema

[+] Implementations

(INTERFACE)

[-] INTERFACE

[+] PROS & CONS

pure JavaScript implementation with TypeScript support for node.js and the browser.

better performance, better documentation

node.js

[+] Setup

browsers

[+] Implementations

(INTERFACE)

[+] usage

Reflection​

static code

[-] compare

[+] flow

Documentation generator plugin for the Protocol Buffers.

[+] Getting started

install docker &

usage

references

Protocol Buffers

More from Chien Chieh Wang

WHAT NEXT?

unclear type
readable

Reflection