streaming the web

--binary json and streaming in javascript


Slides available at bit.ly/hkosc2014-stream
@javascripthk HKOSC 2014

What we are doing here today

For Web Technologist

- Why streaming

-What you will do to a stream --Encoding/Compression

-Binary JSON -- Smile/UBJSON

Platform-specific --NodeJs / HTML5

-Other stuff in the world --web socket, SPDY

VINCENT LAU

Javascript, Java

@vincent_lcy
code4.hk - Code for Hong Kong
@javascripthk - Javascript@HK Meetup


Why Streaming


From user perspective...

You get it **NOW**

You get **what you want**







Developer perspectives

~unix philosophy
~fx programming
~natural publisher-consumer model


force you to separate your concerns

This is the Unix philosophy:

  • Write programs that do one thing and do it well.
  • Write programs to work together.
  • Write programs to handle text streams, because that is a universal interface.
--Doug McIlroy




non-blocking

Memory

Transform on the fly



we have been doing that in node and javascript!

 var server = http.createServer(function (req, res) {
											var stream = fs.createReadStream(__dirname + '/data.txt');
											stream.pipe(res);
										});

Stream in NOdejs


readable, writable, transform, duplex, and "classic"


.pipe()





Streaming is an abstraction

and

Abstraction leaks




HTTP persistent connection?

Encoding Header?

Compression header?

"Bulk write" inside streams? stream.cork in node v0.11

A revision



Ref: http://servicetechmag.com/

Layers of Overhead


A-->JSON-->B
JSON is text!

Binary to textual Encoding 

JSON representation -> Minimal!

HTTP

TCP








My Experience



Streaming Large file from Dropbox to ElasticSearch


Support?
Client level -issue #100



What will you do on a stream?

Encoding
Compression
Encryption


//** online algorithm **//

Encoding - You HAVE CHOICES



Base64

base85

basE91

YEnc

UUencode

Binary JSON

BinaRY jSON - AGAIN You have Choices



SMILE
BSON
UBJSON
BJSON


Make Sense?



Streaming large binary data with JSON

Why not separate the stream and JSON?
Another overhead

using JSON sometimes provides a more RESTful design

Reuse existing architecture

I hear what you say..


Javascript is unfriendly to binary data




PAST



simulated by treating the raw data as a string and using the charCodeAt() method to read the bytes from the data buffer




ref: Mozilla 

Things are getting better



NodeJs - Buffer

in ES6 - 0b and 0o

TypedArraycan i use?

HTML5

https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm

function handler() { 
	if(this.readyState == this.LOADING) {
		var theStream = this.response;
		var streamURL = URL.createObjectURL(theStream);
		document.getElementById("myVideoTag").src = streamURL;
	}
}

var client = new XMLHttpRequest();
client.onreadystatechange = handler;
client.setRequestHeader('customHeader', 'value');
client.setRequestHeader('customHeader2', 'value2');
client.open("GET", "myvideo.h264");
client.responseType = "stream";
client.send();

Call for CODING



javascript parser for SMILE
- C binding for node
- HTML5 / Websocket for browser

https://github.com/Sannis/node-ubjson

Even better FUTURE


Related but not limited to streaming


Libraries - Socket.IO ?
Binaryjs

Websocket

SPDY

WebSocket

WebSocket Protocol is an independent TCP-based protocol. Its only relationship to HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.[1]

Socket.IO do this

  • WebSocket
  • Adobe® Flash® Socket
  • AJAX long polling
  • AJAX multipart streaming
  • Forever Iframe
  • JSONP Polling

SPDY


Eliminate HTTP's overhead, avoid multiple connections for concurrency

~Transport / Session Layer
reuse HTTP's semantics

Multiplexed requests

Prioritized requests
 Compressed headers
 Server pushed streams


Other nice STUFF


https://github.com/substack/stream-handbook

http://felixge.s3.amazonaws.com/11/nodejs-streams.pdf


https://github.com/substack/oppressor


Made with Slides.com