Journey from procedural to reactive JavaScript with stops

Dr. Gleb Bahmutov PhD

Kensho Boston / NYC

Personal site glebbahmutov.com
Blog glebbahmutov.com/blog
Twitter @bahmutov

Boston Code Camp 24 #boscc

Coding in JS

source: Improved Minesweeper

JavaScript == table

source: Don't Make Javascript Equality Look Worse Than It Is

JavaScript < table

source: Don't Make Javascript Equality Look Worse Than It Is

We use JS because it is everywhere

browser
server
mobile
robots

cyclon.js

source: careers.stackoverflow.com

JS package growth

source: modulecounts.com

Changing how JS developers program will have a huge impact on the software quality in the world

Kensho - instant financial analysis

Harvard Sq, Cambridge MA

WTC New York

Kensho - instant financial analytics

Problem: code complexity is quickly growing

number of engineers

lifetime of the company

We need JavaScript to

be as clear as writing an English sentence (or clearer)
produce expected result
be simple to test
assemble complex logic from very simple parts

Use FP to implement Reactive Programming patterns

reactivex.io/tutorials.html

js books

FP in JS

small pure functions
compose complex logic from simple steps
semi-performant async with promises
simple(r) data with immutable structures

reactive programming

every user action is stream of events
every server action is stream of events
streams can be processed using standard operations

2 major parts

Dev team (otherwise smart people)

Everyone has their own path to JS

Everyone is different

100% jQuery
30% FP in JS
25% OO in JS

! junior vs senior

50% jQuery
90% CSS
5% OO in JS

20% jQuery
30% CSS
75% OO in JS
90% FP in JS

In theory

anyone

functional reactive programmer

In practice

anyone

functional reactive programmer

...

procedural
object-oriented
functional
abandon return values
immutable data
lazy evaluation
lazy async values
promises
event emitters
event emitters returning promises
chained event emitters
reactive streams

Here is my journey through the JS land

Show me the code!

Problem: given an array of numbers, multiply each number by a constant and print the result.

var numbers = [3, 1, 7];
var constant = 2;
// 6 2 14

more than 20 implementations at

github.com/bahmutov/javascript-journey

procedural / imperative JS

var numbers = [3, 1, 7];
var constant = 2;
var k = 0;
for(k = 0; k < numbers.length; k += 1) {
  console.log(numbers[k] * constant);
}
// 6 2 14

"for" loop is a pain to debug and use
mixing data manipulation with printing
- code is hard to reuse

Programming without for loops

Start splitting code into procedures

var numbers = [3, 1, 7];
var constant = 2;
function mul(a, b) {
  return a * b;
}
function print(x) {
  console.log(x);
}
function processNumber(n) {
  print(mul(n, constant));
}
for(k = 0; k < numbers.length; k += 1) {
  processNumber(numbers[k]);
}
// 6 2 14

reusable simple functions

complex logic

iteration

Our goal

make the code boring

Boring code does not surprise you

// using "this" WILL surprise you
numbers.forEach(function (x) {
  this.doSomething(x);
  // ERROR!
});

var self = this;
numbers.forEach(function (x) {
  self.doSomething(x);
});

Boring code does not surprise you

// using "this" in your objects
// WILL surprise someone else
numbers.forEach(console.log);

numbers.forEach(console.log.bind(console));

// works under Node
// in Chrome
// Uncaught TypeError: Illegal invocation

Boring code does not surprise you

// for loops WILL surprise you
for(k = 0; k < numbers.length; k += 1) {
  processNumber(numbers[k]);
}

common problems: off by 1 errors, iterator variable, changing array size

Boring code does not surprise you

// using non-local variables WILL surprise you
constant = 2;
...
numbers.forEach(function process(n) {
  return mul(n, constant);
});

Pure functions are really boring

function mul(a, b) {
  return a * b;
}
function print(n) {
  console.log(n);
}

output depends ONLY on the inputs
- same result every time
no surprises
pure against set of unit tests
- Test if a function is pure

Boring functions make new boring functions using fn.bind

var mulBy2 = mul.bind(null, 2);
mulBy2(5); // 10

Binding vs partial application

function mul(a, b) {
  return a * b;
}

var mul2by3 = mul.bind(null, 2, 3);
mul2by3(); // 6

var mul2By10 = mulBy2.bind(null, 10);
mul2By10('extra', 'args are', 'ignored'); // 20

Partial application

function mul(a, b) {
  return a * b;
}
var by2 = mul.bind(null, 2);
// same as
var _ = require('lodash');
var by2 = _.partial(mul, 2);

_.partial, _.partialRight, spots, obind, heroin

Boring code is nice, but I want encapsulated logic, data access, inheritance!

function NumberMultiplier() {}

NumberMultiplier.prototype.setNumbers = function (numbers) {
    this.numbers = numbers;
    return this;
};
NumberMultiplier.prototype.multiply = function (constant) {
    for (var k = 0; k < this.numbers.length; k += 1) {
      this.numbers[k] = constant * this.numbers[k];
    }
    return this;
};
NumberMultiplier.prototype.print = function () {
    console.log(this.numbers);
};

Object-Oriented JavaScript

// using NEW keyword
new NumberMultiplier()
  .setNumbers(numbers)
  .multiply(constant)
  .print();
// [ 6, 2, 14 ]

Object-Oriented JavaScript: 3 parts

function NumberMultiplier() {}

NumberMultiplier.prototype.setNumbers = function() {}
NumberMultiplier.prototype.multiply = function() {}
NumberMultiplier.prototype.print = function() {}

new NumberMultiplier();

constructor function

prototype

"new" keyword

// NumberMultiplier.prototype object
{
  setNumbers: function () { ... },
  multiply: function () { ... },
  print: function () { ... }
}
  /\
  || [[ prototype ]]
  ||
// instance object
{
  numbers: [3, 1, 7]
}

OO JavaScript is prototypical

Common Misconceptions About Inheritance in JavaScript

var NumberMultiplier = {
  setNumbers: function (numbers) {
    this.numbers = numbers; return this;
  },
  multiply: function (constant) {
    for (var k = 0; k < this.numbers.length; k += 1) {
      this.numbers[k] = constant * this.numbers[k];
    } return this;
  },
  print: function () { console.log(this.numbers); }
};

Prototype is a plain object!

javascript-journey source

var numberMultiplier = Object.create(NumberMultiplier);
numberMultiplier
  .setNumbers(numbers)
  .multiply(constant)
  .print();
// [ 6, 2, 14 ]

plain object

class NumberMultiplier {
  setNumbers(numbers) {
    this.numbers = numbers;
    return this;
  }
  multiply(constant) {
    for (var k = 0; k < this.numbers.length; k += 1) {
      this.numbers[k] = constant * this.numbers[k];
    }
    return this;
  }
  print() {
    console.log(this.numbers);
    return this;
  }
}
new NumberMultiplier()

ES6 classes: just don't

How to Fix the ES6 `class` keyword

JavaScript Arrays

typeof [1, 2] // "object"

Array.isArray([1, 2]) // true

Array.prototype
/*
  map
  forEach
  filter
  reduce
  ...
*/

Arrays: object-oriented + functional

var numbers = [3, 1, 7];
var constant = 2;
function mul(a, b) {
  return a * b;
}
function print(n) {
  console.log(n);
}
var mulBy = mul.bind(null, constant);
numbers
    .map(mulBy)
    .forEach(print);
// 6 2 14

clear "multiply then print" semantics

functional bits

Object-oriented methods

My advice

Write small pure functions
Adapt them to particular call sites
Progress slowly through the code base

flexible selective application

Observation: a lot of our code

manipulates collections of data items

// OO (built-in)
Array.prototype.map(cb)

// functional (user space)
Library.map(array, cb)

// plus
Library.groupBy(array, cb);
Library.dropWhile(array, cb);
...

2 of the top 3 most dependent upon NPM projects are FP utility libraries: underscore, lodash

var _ = require('lodash');
var byConstant = _.partial(mul, constant);
// _.forEach(array, cb);
// _.map(array, cb);

Work with arrays using Lodash

_.forEach(
  _.map(numbers, byConstant),
  print
);
// 6 2 14

_({ John: 3, Jim: 10 })
  .map(byConstant)
  .forEach(print);
// 6 20

Treat objects same as collections using lodash

object iterators

Argument order

var mulBy = ...

// OO (built-in)
Array.prototype.map(mulBy)

// lodash / underscore
_.map(array, mulBy)

Put callback first for elegance

// Ramda
R.map(mulBy, array)

var R = require('ramda');
var byConstant = R.partial(R.mul, constant);

1. compose algorithm

2. apply to data

3. profit!!!

// R.forEach = function (callback, array)
var printEach = R.partial(R.forEach, print);

// R.map: function (callback, array)
var mapByConstant = R.partial(R.map, byConstant);

composition

printEach(mapByConstant(numbers));
// 6 2 14

var R = require('ramda');
var byConstant = R.partial(R.mul, constant);

1. compose algorithm

2. apply to data

3. profit!!!

// R.forEach = function (callback, array)
var printEach = R.partial(R.forEach, print);
var algorithm = R.compose(printEach, mapByConstant);

algorithm(numbers);
// 6 2 14

// R.map: function (callback, array)
var mapByConstant = R.partial(R.map, byConstant);

// fn( data )

var R = require('ramda');
var byConstant = R.partial(R.mul, constant);

// R.forEach = function (callback, array)
var printEach = R.partial(R.forEach, print);
var algorithm = R.compose(printEach, mapByConstant);

algorithm(numbers);
// 6 2 14

// R.map: function (callback, array)
var mapByConstant = R.partial(R.map, byConstant);

"fn(data)" code is simple to reason about and test

// fn( data )

// R.mul = function (a, b) { ... }
var byConstant = R.partial(R.mul, constant);

Partial application vs curry

// in Ramda this is same as simply
var byConstant = R.mul(constant);

// for your functions
var mul = R.curry(function (a, b) {
  return a * b; 
});
mul(5)(4); // 20

var R = require('ramda');
R.pipe(
  R.map(R.multiply(constant)),
  R.forEach(print)
)(numbers);
// 6 2 14

Curry: program with very little code

Requires careful function signature design

i.e. put information most likely to be known early as first argument

User friendly API

OO

"Smart" objects

Pass data around

FP

"Dumb" objects

"Smart" pipeline

Pass code (functions) around

new NumberMultiplier()
    .add(3, 1, 7)
    .multiply(2)
    .print();

_.forEach(
  _.map(numbers, byConstant),
  print
);

Object-oriented VS functional

It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures."

- Alan J. Perlis

new NumberMultiplier()
    .add(3, 1, 7)
    .multiply(2)
    .print();

_.forEach(
  _.map(numbers, byConstant),
  print
);

What else can JavaScript do for us?

Source of complexity: mutable data

var numbers = [...];
numbers.forEach(processNumber);
// is the list "numbers" unchanged?

function processNumber(x, k, array) {
    array[1] = 10000;
    print(x);
}

Mutable data makes code hard to understand and modify

Immutable data structures

'use strict';
var immutable = require('seamless-immutable');
var byConstant = _.partial(mul, constant);
immutable(numbers)
  .map(byConstant)
  .forEach(function (x, k, array) {
    array[1] = 10000; // throws an Error
    print(x);
  });

Efficient immutable JS libraries: seamless-immutable, mori

Immutable data with Redux pattern

function mul(a, b) { return a * b; }

function multiply(state, constant) {
  var byConstant = mul.bind(null, constant);
  return state.updateIn(['numbers'], function (ns) {
    return ns.map(byConstant);
  });
}

see the Redux example in javascript-journey

var immutable = require('immutable');
var initialState = immutable.Map({ 
  numbers: immutable.List.of([3, 1, 7])
});
var newState = multiply(initialState, constant);
// newState !== initialState

Arrays: great for synchronous actions

Efficient computation with transducers

Do as little as needed with lazy evaluation

Arrays: NOT SO great for asynchronous

or infinite data!

// Does NOT work
numbers.map(function (x, done) {
  mul(x, function cb(result) {
    done(result);
  })
}).forEach(function (value) {
  print(value, function cb() {
    done():
  })
});

How to handle async callbacks?

numbers
  .map(asyncFn)
  .forEach(asyncFn);

Promises

var Q = require('q');
Q(...)
    .then(function onSuccess(result) { ... },
          function onFailure(error) { ... })
    .then(...);

A single async action with a single result or error

var Q = require('q');
function foo() {
  var defer = Q.defer();
  ... 
    defer.resolve(returnValue);
  ...
  return defer.promise;
}
foo()
    .then(bar)
    .then(baz);
// functions foo, bar, baz return a promise

Each step in the promise chain can be asynchronous and return a promise!

chaining promises

var sleepSecond = _.partial(Q.delay, 1000);

var pauseMulAndPrint = function (n) {
  return function () {
    return sleepSecond()
        .then(_.partial(byConstant, n))
        .then(print);
  };
};

Sleep, mul then print using promises

numbers.map(pauseMulAndPrint)
  .reduce(Q.when, Q())
  .done();
// ... 6 ... 2 ... 14

chaining promises

Promises handle single event very well

Extra complexity trying to handle sequence of events

var sleepSecond = _.partial(Q.delay, 1000);

var pauseMulAndPrint = function (n) {
  return function () {
    return sleepSecond()
        .then(_.partial(byConstant, n))
        .then(print);
  };
};

numbers.map(pauseMulAndPrint)
  .reduce(Q.when, Q())
  .done();
// ... 6 ... 2 ... 14

Constructing a promise for each number :(

var sleepSecond = _.partial(Q.delay, 1000);

var pauseMulAndPrint = function (n) {
  return function () {
    return sleepSecond()
        .then(_.partial(byConstantAsync, n))
        .then(print);
  };
};

Promise is like a single cartridge

Improve upon promises: need to chain async actions without boilerplate per item

"shoot the same bullet more than once"

Event Emitters

handle repeated events
decouple the event source from an action

Widely used in browser code

$(el).on('click', action)

window.on('resize', cb)

socket.on('message', handle)

We can extend the same approach to any event generator (like number sequence)

// action
var events = require('events');
var numberEmitter = new events.EventEmitter();
numberEmitter.on('number', 
  _.compose(print, byConstant)
);

Single code pipeline

var k = 0;
var ref = setInterval(function () {
  numberEmitter.emit('number', numbers[k++]);
  if (k >= numbers.length) {
    clearInterval(ref);
  }
}, 1000);
// prints 6, 2 14 with 1 second intervals

Goal: process number with 1 second pauses

Our emitter has `on`

What is the semantic meaning of `on`?
How do we connect multiple steps?

numberEmitter
  .on('number', action);

Remember Array ES5 methods?

[3, 1, 7]
  .map(mulBy2)
  .forEach(print);

I want this!

source of number events
  .map(an async callback)
  .buffer(n, async callback)
  .forEach(another async callback);

Chained emitters

var stepEmitter = {

  map: function (cb) {
    var emitter = new events.EventEmitter();




    return _.extend(emitter, stepEmitter);
  },

  forEach: function (cb) {
    var emitter = new events.EventEmitter();




    return _.extend(emitter, stepEmitter);
  }
};

Chained emitters

var stepEmitter = {

  map: function (cb) {
    var emitter = new events.EventEmitter();
    this.on('step', function (value) {
      var mappedValue = cb(value);
      emitter.emit('step', mappedValue);
    });
    return _.extend(emitter, stepEmitter);
  },

  forEach: function (cb) {
    var emitter = new events.EventEmitter();
    this.on('step', function (value) {
      cb(value);
      emitter.emit('step', value);
    });
    return _.extend(emitter, stepEmitter);
  }
};

Chained emitters

var stepEmitter = {

  map: function (cb) {
    var emitter = new events.EventEmitter();
    this.on('step', function (value) {
      var mappedValue = cb(value);
      emitter.emit('step', mappedValue);
    });
    return _.extend(emitter, stepEmitter);
  },

  forEach: function (cb) {
    var emitter = new events.EventEmitter();
    this.on('step', function (value) {
      cb(value);
      emitter.emit('step', value);
    });
    return _.extend(emitter, stepEmitter);
  }
};

.map

emits cb(x)

.forEach

emits x

source of number events
  .map(syncFn) 
  .forEach(syncFn);

Chained emitters

So far: steps are async, but each callback function is synchronous!

Async callback support via promises

var Q = require('q');
var stepEmitter = {
  map: function (cb) {
    var emitter = new events.EventEmitter();
    this.on('step', function (value) {
      Q.when(cb(value)).then(function (mappedValue) {
        emitter.emit('step', mappedValue);
      });
    });
    return _.extend(emitter, stepEmitter);
  },

  forEach: function (cb) {
    var emitter = new events.EventEmitter();
    this.on('step', function (value) {
      Q.when(cb(value)).then(function () {
        emitter.emit('step', value);
      });
    });
    return _.extend(emitter, stepEmitter);
  }
};

source of number events
  .map(asyncFn) 
  .forEach(asyncFn);

Chained emitters + promises

async for the win!

var stepEmitter = {
  // same map and forEach methods as above

  // returns a new step emitter that accumulates N items
  // emits the entire array with N items as single argument
  buffer: function (n) {
    var received = [];
    var emitter = new events.EventEmitter();
    this.on('step', function (value) {
      received.push(value);
      if (received.length === n) {
        emitter.emit('step', received);
        received = [];
      }
    });
    return _.extend(emitter, stepEmitter);
  }
};
source(numbers)
  .map(byConstant)
  .buffer(3)
  .forEach(print);
// sleeps 3 seconds then prints [6, 2, 14]

What about buffering events?

source(numbers)
  .map(byConstant) // data transform
  .buffer(3)       // sequence control
  .forEach(print); // data transform

We implemented 2 types of operations on a sequence of events

Need extra methods to control the sequence (stream) of events

.map, .filter
// vs
.buffer

Reactive programming

Everything is a source of asynchronous events.

Your code is a pipeline reacting to one or more input streams and outputting another stream of events

Reactive JS libraries

collection processing methods (map, filter, ...)
stream control methods (buffer, zip, flatMapLatest, ...)

RxJs, BaconJS, cycle

Use a reactive FP library

var Rx = require('rx');

var timeEvents = Rx.Observable
  .interval(1000)
  .timeInterval(); // stream 1

var numberEvents = Rx.Observable
  .fromArray(numbers); // stream 2

Rx.Observable.zip(timeEvents, numberEvents, 
                  function pickValue(t, n) { return n; })
  .map(byConstant)
  .subscribe(print); // stream 3

// prints 6 2 14 with 1 second intervals

javascript-journey source

User click events

var Rx = require('rx');

var clickEvents = Rx.Observable
  .fromEvent(document.querySelector('#btn'), 'click')

var numberEvents = Rx.Observable
  .fromArray(numbers);

function pickSecond(c, n) { return n; }

Rx.Observable.zip(clickEvents, numberEvents, 
                  pickSecond)
  .map(byConstant)
  .subscribe(print);
// prints a number on each button click

Marble diagram for

Rx.zip

zip(s1, s2, fn)

function searchWikipedia (term) {
    return $.ajax({
        url: 'http://en.wikipedia.org/w/api.php',
        dataType: 'jsonp',
        data: {
            action: 'opensearch',
            format: 'json',
            search: term
        }
    }).promise();
}

Autocomplete

Hard problem: need to throttle, handle out of order returned results, etc.

function searchWikipedia (term) {
    return $.ajax({
        url: 'http://en.wikipedia.org/w/api.php',
        dataType: 'jsonp',
        data: {
            action: 'opensearch',
            format: 'json',
            search: term
        }
    }).promise();
}

var keyups = Rx.Observable.fromEvent($('#input'), 'keyup')
    .map(function (e) { return e.target.value; })
    .filter(function (text) { return text.length > 2; });

Autocomplete

keyups.throttle(500)
    .distinctUntilChanged()
    .flatMapLatest(searchWikipedia)
    .subscribe(function (data) {
        // display results
    });

Rx is the machine gun,

events are the ammo

Summary

Write small pure functions

From procedural to RFP

Summary

Handle single async events using promises

From procedural to RFP

Summary

Handle multiple async event streams using Rx

From procedural to RFP

Say "Goodbye" to old friends

Tell return values Bye-bye

// imperative code
var sum = add(2, 3);
// can use sum right away!

var multiplied = [1, 2, 3].map(mul);
// can use multiplied array right away!

var _ = require('lodash');
var byConstant = _.partial(mul, constant);
_(numbers)
  .map(byConstant)
  .forEach(print);
// we have not returned any values!

Abandoning return values

Tell middle functions Bye-bye

var byConstant = _.partial(mul, constant);
_(numbers)
  .map(function (x) {
    return byConstant(x);
  })
  .forEach(function (x) {
    print(x);
  });

_(numbers)
  .map(byConstant)
  .forEach(print);
// no middle functions

Point-free is not pointless

Tell deep equality checks Bye-bye*

var result = process(numbers);
if (deepEqual(result, numbers)) {
  render();
}

var result = process(immutableNumbers);
if (result === immutableNumbers) {
  render();
}

Avoid side effects

* with immutable data

Q: Which step causes the most difficulties?

A: For me it was abandoning the return values, others had problem with promises

Abandoning return values

Q: How to nudge developers along the journey?

A: Having everyone interested train together

Overcoming functional

programming objections

Kensho progress report

object-oriented        **             2 /10

functional             *********      9 /10

immutable data         ******         6 /10

lazy evaluation        *              1 /10

promises               *********      9 /10

event emitters         *****          5 /10

reactive streams       *********      9 /10

Future is bright

All examples shown use today's ES5 standard

ES6 (EcmaScript2015) will make some pieces simpler

arrow notation
block-scoping and constants
spread operator
native Promises
binding context in callbacks

slides slides.com/bahmutov
this presentation as a blog post glebbahmutov.com/blog/journey-from-procedural-to-reactive-javascript-with-stops/
code github.com/bahmutov/javascript-journey

Dr. Gleb Bahmutov PhD Kensho

Boston Code Camp 24 #boscc

Journey from procedural to reactive JavaScript with stops

Coding in JS

JavaScript == table

JavaScript < table

We use JS because it is everywhere

browser

server

mobile

robots

JS package growth

Problem: code complexity is quickly growing

We need JavaScript to

Use FP to implement Reactive Programming patterns

FP in JS

reactive programming

Dev team (otherwise smart people)

Everyone is different

Here is my journey through the JS land

Show me the code!

procedural / imperative JS

Our goal

make the code boring

Pure functions are really boring

Boring functions make new boring functions using fn.bind

Partial application

JavaScript Arrays

My advice

Write small pure functions

Adapt them to particular call sites

Progress slowly through the code base

Observation: a lot of our code

manipulates collections of data items

2 of the top 3 most dependent upon NPM projects are FP utility libraries: underscore, lodash

Treat objects same as collections using lodash

Argument order

1. compose algorithm

2. apply to data

3. profit!!!

1. compose algorithm

2. apply to data

3. profit!!!

OO

FP

Object-oriented VS functional

What else can JavaScript do for us?

Arrays: great for synchronous actions

Arrays: NOT SO great for asynchronous

or infinite data!

How to handle async callbacks?

Promises

Sleep, mul then print using promises

Promise is like a single cartridge

Improve upon promises: need to chain async actions without boilerplate per item

Event Emitters

handle repeated events

decouple the event source from an action

Our emitter has `on`

Remember Array ES5 methods?

I want this!

Need extra methods to control the sequence (stream) of events

Reactive programming

Reactive JS libraries

Rx is the machine gun,

events are the ammo

Summary

Write small pure functions

From procedural to RFP

Summary

​Handle single async events using promises

From procedural to RFP

Summary

​Handle multiple async event streams using Rx

From procedural to RFP

Say "Goodbye" to old friends

Q: Which step causes the most difficulties?

A: For me it was abandoning the return values, others had problem with promises

Q: How to nudge developers along the journey?

A: Having everyone interested train together

Kensho progress report

Future is bright

Handle single async events using promises

Handle multiple async event streams using Rx