Transducers

Reducing complexity with reducers

Why should I care?

Transducers may be able to meaningfully improve performance within

• Parse
• Risk
• Income
• Human to human communication

Is transducer even a word?

A device that converts variations in a physical quantity, such as pressure or brightness, into an electrical signal, or vice versa.

Definition in Functional Programming

To be honest, I don't really know.  I just learned about them this weekend. BUT, I would define a transducer as:

a function which takes a reducing function -- a function that takes an accumulator and a new input and returns a new value -- and returns a new, augmented reducing function.

Ramda supports it

• map, filter, and a handful of other funcs
• Implementation is deeply abstracted
  • Also complex as hell for callers
• We don't actually leverage them at Plaid

var _curry2 = require('./internal/_curry2');
var _dispatchable = require('./internal/_dispatchable');
var _filter = require('./internal/_filter');
var _xfilter = require('./internal/_xfilter');

/**
 * filter function

 * Dispatches to the `filter` method of the second argument, if present.
 *
 * Acts as a transducer if a transformer is given in list position.
 *
module.exports = _curry2(_dispatchable('filter', _xfilter, _filter));

Filter Implementation (ramda)

module.exports = (function() {
  function XFilter(f, xf) {
    this.xf = xf;
    this.f = f;
  }
  XFilter.prototype['@@transducer/init'] = _xfBase.init;
  XFilter.prototype['@@transducer/result'] = _xfBase.result;
  XFilter.prototype['@@transducer/step'] = function(result, input) {
    return this.f(input) ? this.xf['@@transducer/step'](result, input) : result;
  };

  return _curry2(function _xfilter(f, xf) { return new XFilter(f, xf); });
}());

_xfilter Implementation (ramda)

• We take a transforming predict function (f) and a transducer (xf)
  • ​The latter is a collection that has special properties
• We return a transducer object which contains a reducer at property '@@transducer/step'
  • Which in turn invokes the reducer of transducer xf
    • ​Nesting / recursion is happening
• At the top level, we'll eventually call our top level transducers step method

What are the costs of ramda or underscore sans transducers?

Extra computations and allocations

R.pipe(
    // n operations...consume entire input collection and return new collection
    R.pluck('age'),    
    // n operations...consume entire input collection and return new collection 
    R.filter(R.gt(18)),
    // (n - i) operations...consume entire input collection and return new collection
    R.map(R.add(5))      
)

Each operation is sequentially applied to whole collections and produce intermediate data structures.

Particularly important to minimize in computation heavy services like parse

No infinite streams support

function* fibonacci(){
  var fn1 = 1;
  var fn2 = 1;
  while (true){  
    var current = fn2;
    fn2 = fn1;
    fn1 = fn1 + current;
    var reset = yield current;
    if (reset){
        fn1 = 1;
        fn2 = 1;
    }
  }
}


// Let's assume for a second that ramda supported 
// generators on some level with the take function.
// Without transducers, it would (likely) still attempt to compute
// the entire generator, which would force it to loop endlessly.
R.filter(R.gte, fibonacci)

• Tail a long file
• Iterate through a database cursor

How does Haskell enable infinite streams and efficient composition?

Lazy Evaluation of composed functions

To take(5), I don't need the entire mapped collection, I just need the first 5, transforms items

How can we recreate these features in JS?

Transducers!

function xmap(transform) {
   function transducer(innerReducer) {
       function reduce(result, _input) {
           return inner_reducer(result, transform(_input);
       } 
   }
}

// Curried version of above
const xfilter = R.curry((pred, innerReducer, result, _input) => {
    result ? !pred(_input) : inner_reducer(result, _input);
});

function concat(result, _input) {
    // An implementation of concat that adds _input to the end of result
    // and returns that new combined object.
}

const AddPositivesOnlyReducer = R.pipe(
  xmap(R.add(1))
  xfilter(R.lt(0))
)(concat);

def transduce(reducer, results, xs):
    let results = [];
    for (el in xs) { results = reducer(results, el};
    return results;


// Assume deepEqual function
deepEqual(
   tranduce(AddPositivesOnlyReducer, [], [-3, 5, 4, 0, -1, -4])),
   [6, 5]  // WHATTTT, are we sure it's not [6, 5, 1] !!!
)

const R = require('ramda');

// Import xmap, xfilter, concat, deepEqual, transduce

const pluck = R.curry((prop, innerReducer, result, _input) => {
   return xmap(obj -> obj[prop], innerReducer, result, _input)
});

const contrivedExampleReducer = R.pipe(
    pluck('age') 
    xmap(R.merge({age: 18}),
    xfilter(R.complement(R.isEmpty))
)(concat);


// The first argument to contrivedExampleReducer is 
// the initial result (an empty list)
deepEqual(contrivedExampleReducer([], [{}, {}, {}]), [])

const altContrivedExampleReducer = R.pipe(
    xfilter(R.complement(R.equals(18))
    pluck('age') 
    xmap(R.merge({age: 18}),
)(concat);

deepEqual(
    transduce(altContrivedExampleReducer, [], [{}, {}, {}])), 
    [18, 18, 18]
)

// Let's just assume that we added console.log as the first line to xmap, xfilter, pluck, etc.
// What would we see when running the transducer below?


const contrivedExampleReducer = R.pipe(
    pluck('age') 
    xmap(R.merge({age: 18}),
    xfilter(R.complement(R.isEmpty))
)(concat);

const altContrivedExampleReducer = R.pipe(
    xfilter(R.complement(R.equals(18))
    pluck('age') 
    xmap(R.merge({age: 18}),
)(concat);


> transduce(contrivedExampleReducer, [], [{}, {}, {}])),;
'filtering'
'filtering'
'filtering'


> transduce(altContrivedExampleReducer, [], [{}, {}, {}])),;
'mapping'
'plucking'
'filtering'
'mapping'
'plucking'
'filtering'
'mapping'
'plucking'
'filtering'

What about infinite streams?

function* fibonacci(){
  var fn1 = 1; fn2 = 1;
  while (true){  
    var current = fn2;
    fn2 = fn1; fn1 = fn1 + current;
    var reset = yield current;
    if (reset) fn1 = 1; fn2 = 1;
    }
  }
}

function Reduced(val) { this.value = val);

const take_while = R.curry((pred, inner_reducer, result, _input) => {
   return pred(_input) ? inner_reducer(result, _input) : new Reduced(result)
});

const transduce = (reducer, results, xs) => {
    for (let el of xs) {
        results = reducer(results, el)
        if (results.constructor.name === 'Reduced') return results.value;

    }
    return results;
};

// Let's just assume that transduce is implemented as above
// through collections via (for of xs). This means that 
// it would consume the generator returned by fibanoc
transduce(R.pipe(take_while(R.lt(20), _filter(lambda x: x > 5))(concat), [], fibonacci())

Sounds cool

to bad there is no popular library :(

Ramda

Supports transducers for certain functions.  The implementation is confusing though, and documentation is highly, highly lacking.

Transducer.js

Sweet library with cross language support.  Super simple implementation of most functions (especially as compared to ramda). Lacks the same breadth of functions.

Not so Fast!