Streaming in JavaScript

Miłosz Chmura

https://bitbucket.org/mieszko4/, https://github.com/mieszko4, @chmuraOrg

 

Definition

A stream is an abstract interface implemented by various objects in Node.js.

All streams are instances of EventEmitter.

 

 

Approach of programming.

 

Types

  • writable
  • readable
  • duplex

Streams in Node.js

Examples of Streams

 

  • process.stdin (readable)
  • process.stdout (writeable)
  • HTTP request on the server (readable) 
  • HTTP response on the server (writable)
  • fs (readable, writable)
  • TCP sockets (duplex)

My interview task

 

  • We have some customer records in a text file one customer per line, JSON-encoded. 
  • We want to invite any customer within 100km of our Dublin office (GPS coordinates 53.3381985, -6.2592576) to some food and drinks on us.
  • Write a program that will read the full list of customers
  • Sort by user id (ascending)
  • Output the names and user ids of matching customers (within 100km)

Solution draft

 

  1. read file -> string
  2. split LF -> array[string]
  3. parse -> array[object]
  4. validate -> array[object]
  5. calculate distance and filter items -> array[object]
  6. sort -> array[object]
  7. pick name and id -> array[string]
  8. join LF -> string

Simple approach

 

  • Read entire data to string
  • Split string to get an array
  • Loop each item, validate, filter, etc. and push it into new array
  • Sort array
  • Loop each item in order to display it

BUT

Better to treat this as a stream of data

() -> split -> parse -> validate -> filter -> toString -> join -> ()

Functional approach

 

Demo

 

https://gist.github.com/mieszko4/19707bae7e127019953635ab3aa40c0d

Streaming approach

 

Demo

 

https://gist.github.com/mieszko4/5595b5c18a7ca70a62c4cb2d35aa87e0

Playing with streams

 

  • add throttle for testing
    • @demo
  • parallelize compute intensive task
    • @filtering

Benefits of Streams

 

  • live processing - no need to wait for entire input data - it is processed as soon as it can
  • potential for performance boost when paralleling tasks 
  • infinite amount of input data - no memory is needed to store input data

 

Things to check out

 

Thanx! Questions?

Made with Slides.com