Life Beyond JSON...

Exploring binary encoding in Node.js

Julián Duque

Developer and Educator

https://twitch.tv/julianduque

Why?

Our applications always manipulate and communicate using data; an encoding mechanism is needed to guarantee that both systems can read and write that data in a compatible and optimal way.

That data needs to travel on a channel using a protocol; the web we know is mostly text-based, but, for certain use cases, we can use binary encoding algorithms to guarantee compatibility, schema evolution, and performance.

JSON

JavaScript Object Notation

• It is string based
• Human readable!
• Supported by multiple languages and platforms
• It is the currently the most common encoding (along with XML and CSV)

MsgPack

• It's a binary encoding algorithm
• Similar to JSON but smaller (faster in some cases)
• Not human-readable (It's binary)
• It's Schema-less (Doesn't need a definition file)

Source: https://msgpack.org/

MsgPack

• npm install @msgpack/msgpack
• Use encode and decode
• Profit!

Caveat:

• Node.js JSON encoding/decoding is way much faster

Protobuffers

• Created by Google 
• It's binary
• It's super fast and small
• Schema-based so it relies on a definition file
• Widely used in gRPC

Protobuffers

• npm install protobufjs
• Need to define a .proto file (or JSON definition is also supported)

Avro

• It's a project by the Apache Foundation
• Very popular in the Java Ecosystem
• It's super fast and small 🚀
• Schema-based, they can be written in JSON or with an IDL

Avro

• npm install avsc
• Create a schema using JSON definition or IDL
• Load schema with avro.type.forSchema()
• Use type.fromBuffer() and type.toBuffer()

Benchmarks 🚀

Conclusions

Use them when it makes sense.

• Do you need to decrease the size of your payload to reduce bandwidth?
• Are you processing huge amounts of data in a real-time manner? (e.g.: Apache Kafka)
• Are you writing integration services that need to be highly performant?
• Do you want to experiment with RPC?

Resources

• Designing Data-Intensive Applications by Martin Kleppmann (Chapter 4)
• Schema evolution in Avro, Protocol Buffers and Thrift by Martin Kleppmann https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

And... that's it?

And... that's it?

Join us next week for more! I'll be talking about RPC Servers using Binary Encoding! 🎉

Thank you!