streaming the web

--binary json and streaming in javascript

Slides available at bit.ly/hkosc2014-stream

@javascripthk

HKOSC 2014

What we are doing here today

For Web Technologist

- Why streaming

-What you will do to a stream --Encoding/Compression

-Binary JSON -- Smile/UBJSON

Platform-specific --NodeJs / HTML5

-Other stuff in the world --web socket, SPDY

VINCENT LAU

Javascript, Java

@vincent_lcy

code4.hk - Code for Hong Kong

@javascripthk - Javascript@HK Meetup

blog kleineblase.postach.io

Why Streaming

From user perspective...

You get it **NOW**

You get **what you want**

Developer perspectives

~unix philosophy

~fx programming

~natural publisher-consumer model

force you to separate your concerns

This is the Unix philosophy:

Write programs that do one thing and do it well.

Write programs to work together.

Write programs to handle text streams, because that is a universal interface.

--Doug McIlroy

non-blocking

Memory

Transform on the fly

we have been doing that in node and javascript!

 var server = http.createServer(function (req, res) {
											var stream = fs.createReadStream(__dirname + '/data.txt');
											stream.pipe(res);
										});

Stream in NOdejs

readable, writable, transform, duplex, and "classic"

.pipe()

transform vs duplex?

Streaming is an abstraction

and

Abstraction leaks
--joel sposky's The Law of Leaky Abstractions

HTTP persistent connection?

Encoding Header?

Compression header?

"Bulk write" inside streams? stream.cork in node v0.11

A revision

Ref: http://servicetechmag.com/

Layers of Overhead

A-->JSON-->B

JSON is text!

Binary to textual Encoding

JSON representation -> Minimal!

HTTP

TCP

My Experience

Streaming Large file from Dropbox to ElasticSearch

Support?

API level (Protocol)

Client level -issue #100

What will you do on a stream?

Encoding

Compression

Encryption

//** online algorithm **//

Encoding - You HAVE CHOICES

Base64

base85

basE91

YEnc

UUencode

Binary JSON

BinaRY jSON - AGAIN You have Choices

SMILE

BSON

UBJSON

BJSON

Make Sense?

Streaming large binary data with JSON

Why not separate the stream and JSON?
Another overhead

using JSON sometimes provides a more RESTful design

Reuse existing architecture

I hear what you say..

Javascript is unfriendly to binary data

PAST

simulated by treating the raw data as a string and using the charCodeAt() method to read the bytes from the data buffer

ref: Mozilla

Things are getting better

NodeJs - Buffer

in ES6 - 0b and 0o

TypedArray can i use?

HTML5

https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm


function handler() { 
	if(this.readyState == this.LOADING) {
		var theStream = this.response;
		var streamURL = URL.createObjectURL(theStream);
		document.getElementById("myVideoTag").src = streamURL;
	}
}

var client = new XMLHttpRequest();
client.onreadystatechange = handler;
client.setRequestHeader('customHeader', 'value');
client.setRequestHeader('customHeader2', 'value2');
client.open("GET", "myvideo.h264");
client.responseType = "stream";
client.send();

Call for CODING

javascript parser for SMILE

- C binding for node

- HTML5 / Websocket for browser

https://github.com/Sannis/node-ubjson

Even better FUTURE

Related but not limited to streaming

Libraries - Socket.IO ?

Binaryjs

Websocket

SPDY

WebSocket

WebSocket Protocol is an independent TCP-based protocol. Its only relationship to HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.[1]

Socket.IO do this

WebSocket
Adobe® Flash® Socket
AJAX long polling
AJAX multipart streaming
Forever Iframe
JSONP Polling

SPDY

Eliminate HTTP's overhead, avoid multiple connections for concurrency

~Transport / Session Layer

reuse HTTP's semantics

Multiplexed requests

Prioritized requests
Compressed headers
Server pushed streams

in Chrome/Firefox

Other nice STUFF

https://github.com/substack/stream-handbook

http://felixge.s3.amazonaws.com/11/nodejs-streams.pdf

https://github.com/substack/oppressor

Streaming the web

By Chun Yin Vincent Lau

Streaming the web

2,964