Julián Duque
Developer and Educator
Life Beyond JSON...
Exploring binary encoding in Node.js
Why?
Our applications always manipulate and communicate using data; an encoding mechanism is needed to guarantee that both systems can read and write that data in a compatible and optimal way.
That data needs to travel on a channel using a protocol; the web we know is mostly text-based, but, for certain use cases, we can use binary encoding algorithms to guarantee compatibility, schema evolution, and performance.
🗣🎶 📦 🎶💃
JSON
JavaScript Object Notation
- It is string based
- Human readable!
- Supported by multiple languages and platforms
- It is the currently the most common encoding (along with XML and CSV)
MsgPack
- It's a binary encoding algorithm
- Similar to JSON but smaller (faster in some cases)
- Not human-readable (It's binary)
- It's Schema-less (Doesn't need a definition file)
Source: https://msgpack.org/
MsgPack
- npm install @msgpack/msgpack
- Use encode and decode
- Profit!
Caveat:
- Node.js JSON encoding/decoding is way much faster
Protobuffers
- Created by Google
- It's binary
- It's super fast and small
- Schema-based so it relies on a definition file
- Widely used in gRPC
Protobuffers
- npm install protobufjs
- Need to define a .proto file (or JSON definition is also supported)
Avro
- It's a project by the Apache Foundation
- Very popular in the Java Ecosystem
- It's super fast and small 🚀
- Schema-based, they can be written in JSON or with an IDL
Avro
- npm install avsc
- Create a schema using JSON definition or IDL
- Load schema with avro.type.forSchema()
- Use type.fromBuffer() and type.toBuffer()
Benchmarks 🚀
Conclusions
Use them when it makes sense.
- Do you need to decrease the size of your payload to reduce bandwidth?
- Are you processing huge amounts of data in a real-time manner? (e.g.: Apache Kafka)
- Are you writing integration services that need to be highly performant?
- Do you want to experiment with RPC?
Resources
- Designing Data-Intensive Applications by Martin Kleppmann (Chapter 4)
- Schema evolution in Avro, Protocol Buffers and Thrift by Martin Kleppmann https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
And... that's it?
And... that's it?
Join us next week for more! I'll be talking about RPC Servers using Binary Encoding! 🎉
Thank you!
Life Beyond JSON
By Julián Duque
Life Beyond JSON
Exploring binary encoding protocols in Node.js
