The streaming build system


Consulting, training, products, professional services

Open source is at


Your application is not a big truck - it's a series of tubes

We should have some ways of connecting programs like garden hose--screw in another segment when it becomes necessary to massage data in another way. This is the way of IO also.
- Doug McIlroy

Streams come to us from the earliest days of unix and have proven themselves over the decades as a dependable way to compose large systems out of small components that do one thing well.

You can then plug the output of one stream to the input of another and use libraries that operate abstractly on streams to institute higher-level flow control.
- substack

Understanding flow control is what makes you a programmer

Read a comprehensive overview of streams by substack

Learning new flow control techniques will make you more efficient

Why streams?

Picture a build system in your head.

(It should take in files, modify them, and output the new ones)

You pictured this

You didn't picture this


What's so bad?

  • Plugins do multiple things
    • Want a banner? Use the javascript minifier
  • Plugins do things that don't need to be plugins
    • Need to run your tests? Use a plugin
  • Grunt config format is a mess that tries to do everything
    • Not idiomatic with "the node way"
  • Headache of temp files/folders due to bad flow control

Your build system should empower not impede

It should only manipulate files - let other libraries handle the rest.

Sample Gruntfile

 module.exports = function(grunt) {

    pkg: grunt.file.readJSON('package.json'),
    concat: {
      options: {
        separator: ';'
      dist: {
        src: ['src/**/*.js'],
        dest: 'dist/<%= %>.js'
    uglify: {
      options: {
        banner: '/*! <%= %> <%="dd-mm-yyyy") %> */\n'
      dist: {
        files: {
          'dist/<%= %>.min.js': ['<%= concat.dist.dest %>']
    qunit: {
      files: ['test/**/*.html']
    jshint: {
      files: ['gruntfile.js', 'src/**/*.js', 'test/**/*.js'],
      options: {
        // options here to override JSHint defaults
        globals: {
          jQuery: true,
          console: true,
          module: true,
          document: true
    watch: {
      files: ['<%= jshint.files %>'],
      tasks: ['jshint', 'qunit']


  grunt.registerTask('test', ['jshint', 'qunit']);

  grunt.registerTask('default', ['jshint', 'qunit', 'concat', 'uglify']);

  1. Runs tests
  2. Lints code
  3. Concats javascript
  4. Minifies it
  5. Runs again if files are changed


Sample Gulpfile

var gulp = require('gulp');
var pkg = require('./package.json');
var concat = require('gulp-concat');
var minify = require('gulp-minify');
var jshint = require('gulp-jshint');
var spawn = require('child_process').spawn;

var scriptFiles = './src/**/*.js';

gulp.task('compile', function(){
  // concat all scripts, minify, and output

gulp.task('test', function(){
  // lint our scripts
  // run our tests
  spawn('npm', ['test'], {stdio: 'inherit'});
gulp.task('default', function(){'test', 'compile');, function(){'test', 'compile');
  1. Runs tests
  2. Lints code
  3. Concats javascript
  4. Minifies it
  5. Runs again if files are changed

What's the difference?

  • With Gulp your build file is code, not config
  • You use standard libraries to do things
  • Plugins are simple and do one thing - most are a ~20 line function
  • Tasks are executed with maximum concurrency
  • I/O works the way you picture it

Gulp does nothing but provide some streams and a basic task system

Gulp has only 5 functions you need to learn

gulp.task(name, fn)

It registers the function with a name.

You can optionally specify some dependencies if other tasks need to run first.

Runs all tasks with maximum concurrency, fn)

Runs a function when a file that matches the glob changes

Included in core for simplicity


This returns a readable stream.

Takes a file system glob (like grunt) and starts emitting files that match.

This is piped to other streams


This returns a writable stream

File objects piped to this are saved to the file system


You are now a Gulp expert

For guides on how to make your own plugins check the README




By Eric Schoffstall


Streaming build systems

  • 163,134