gulping streams

Fabrício Matté

twitter.com/Ult_Combo

github.com/UltCombo

Streams

What?

A sequence of data elements made available over time.

An array of asynchronous values.

Where?

  • Audio/video streams
  • File system (read/write)
  • Browser fetching resources
  • CLI
$ curl --data domain=test http://api.otsus.vm/install | format-json-stream > test.json

Why?

Process data chunks as soon as they are made available.

Better perf, smaller memory footprint.

Efficient I/O

Connect streams through pipes.

Separation of concerns, reusable code.

Composability

Efficiency

http://blog.chromium.org/2015/03/new-javascript-techniques-for-rapid.html

Streams !== Promises

Has a single resolution (fulfillment/rejection) value

Promise

Consumes/processes/emits streaming chunks of data

Stream

Stream types

Produces data

Readable

Writable

Consumes data

Is both readable and writable

Duplex

Stream types

Is a duplex stream, where the output is in some way calculated from the input.

Written data goes through the stream, being transformed into new readable data.

Transform (through)

Is a simple transform stream implementation which passes the input to the output without any modification.

PassThrough

.pipe()

Connects a readable stream to a writable stream.

Handles backpressure and pause/resume auto-magically.

Readable

HLS

Trans form

Writable

Video feed

Codec

.pipe()

.pipe()

Stream flow overview

Video playback

RTP

File system

File copy

fs.createReadStream('path')

fs.createWriteStream('path')

.pipe()

HTTP req stream

FTP GET stream

Composability!

A platform to run JavaScript everywhere.

Control file system, network and everything in the host machine.

Develop CLI programs using JavaScript.

A fork of Node.js.

Updated JavaScript engine (V8) and other base libraries.

Open governance model.

Improved streams.

var Readable = require('stream').Readable;

var randomStream = new Readable({
  read: function() {
    var n = Math.floor(Math.random() * 100);
    this.push(n < 95 ? n + '\n' : null);
  }
});

var Readable = require('stream').Readable;
var util = require('util');
util.inherits(RandomNumbers, Readable);

function RandomNumbers(opt) {
  Readable.call(this, opt);
}

RandomNumbers.prototype._read = function() {
  var n = Math.floor(Math.random() * 100);
  this.push(n < 95 ? n + '\n' : null);
};

var randomStream = new RandomNumbers();

The package manager for JavaScript.

Over 140,000 public packages.

*npm is always lowercase

gulp

  • Vinyl: virtual file format
  • Streams: vinyl file streams
  • Strict plugin guidelines
  • Orchestrator: task system
    (series and parallel tasks execution is now possible with gulp 4.0!)

*gulp is always lowercase

Vinyl adapter

Returns a readable stream which produces vinyl files.

  • `gulp.src()` (vinyl-fs):  Vinyl adapter for the file system
  • gulp-ruby-sass v1.0.0+: replaces `gulp.src()`, compiles files with Sass and produces CSS vinyl files

Examples:

gulp plugins

Are transform streams that operate on vinyl file streams.

  • gulp-babel
  • gulp-uglify
  • gulp-concat
  • gulp-buster (shameless self-promotion)
  • Another 1,400+ plugins

Examples:

gulp.src('js/*.js')
    .pipe(gulp.dest('dist/js'));

Copy files

Is equivalent to:

var readableStream = gulp.src('js/*.js');
var writableStream = gulp.dest('dest/js');

readableStream.pipe(writableStream);
gulp.src('js/*.js')              // readable stream
    .pipe(babel())               // transform stream
    .pipe(uglify())              // transform stream
    .pipe(concat('all.js'))      // transform stream
    .pipe(gulp.dest('dist/js')); // writable stream

Build

No intermediary disk writes! 

gulp.src('js/*.js')              // readable stream
    .pipe(babel())               // transform stream
    .pipe(through2.obj(function(file, enc, cb) {
        file.contents = Buffer.concat([new Buffer('// Gulping Streams\n'), file.contents]);
        cb(null, file);
    }))
    .pipe(gulp.dest('dest/js')); // writable stream

Creating transforms

Web Platform

WHATWG Streams.

Experimental implementation in Chromium/Chrome Canary.

Demo.

Thanks!

Fabrício Matté

twitter.com/Ult_Combo

github.com/UltCombo

https://slides.com/fabriciomatte/gulping-streams

References

  • http://nodejs.org/api/stream.html
  • https://github.com/iojs/io.js/blob/master/doc/api/stream.markdown
  • https://streams.spec.whatwg.org/
  • https://docs.google.com/document/d/1ZlEYZ-06Sbqds98EA0MgXP2JYxnpGH4j4c_x27Fjp6c/edit
  • https://nodejs.org/
  • https://iojs.org/
  • https://www.npmjs.com/
  • https://docs.npmjs.com/misc/faq#is-it-npm-or-npm-or-npm
  • https://medium.com/@contrahacks/gulp-3828e8126466
  • https://github.com/gulpjs/gulp/blob/master/docs/FAQ.md#is-it-gulp-or-gulp
  • http://gulpjs.com/
  • https://www.npmjs.com/search?q=gulpplugin

Gulping streams

By Fabrício Matté

Made with Slides.com

Gulping streams

#streams #gulpjs

  • 1,705

More from Fabrício Matté