Image processing pipeline in Javascript

Jonathan Lurie - MCIN - June 2017

  • What is it?
  • Motivations
  • Possible use cases
  • What's a pipeline?
  • Under the hood
  • More on Filters
  • Examples
  • Few metrics
  • Ressources

Jonathan Lurie - MCIN - June 2017

What is it?

  • A way to process images in the browser
  • A Javascript-only codebase
  • A ES6 bundle (npm style)
  • A headless library
  • Detached from any visualization engine
  • A project started at the BrainHack Mtl

Jonathan Lurie - MCIN - June 2017

Jonathan Lurie - MCIN - June 2017

Motivations

  • A web browser is the most accessible software
  • Make something easy to use
  • With no other dependency than pixpipe.js
  • With no required compilation or system fuss
  • Modular architecture
  • Generic enough to use different kind of data/datasource
  • Well documented for both users and contributors
  • The more on the browser, the less on the server

Jonathan Lurie - MCIN - June 2017

Possible use cases

  • Open 2D image files (png, jpg, tiff 8/16bits)
  • Open 3D image files (Minc2, NIfTI, MGH)
  • Open local (file dialog) or distant (AJAX) files
  • Perform statistics
  • Extract slices on a given axis (from 3D)
  • Create mosaics (from 3D)
  • Process the data
    • 2D convolution
    • Compute derivatives and gradients
    • Contour detection
    • Blending, scaling, pixel-wise treatment ...
  • Save your result locally
  • etc.

Jonathan Lurie - MCIN - June 2017

What's a pipeline?

à la ITK

Jonathan Lurie - MCIN - June 2017

Under the hood

Jonathan Lurie - MCIN - June 2017

Core architecture

Jonathan Lurie - MCIN - June 2017

PixpipeObject

  • Every object inherits from this one
  • Carries all the metadata logic
  • Generates a unique identifier (uuid) at instanciation

* The most generic *

Jonathan Lurie - MCIN - June 2017

PixpipeContainer

  • Inherits from PixpipeObject
  • Adds the _data attribute (of unspecified type)
  • But still cannot do anything!

data container + generic

Jonathan Lurie - MCIN - June 2017

Image2D

  • Inherits from PixpipeContainer
  • Generic container for jpeg, png, tiff
  • _data becomes a JS TypedArray, can handle:
    • Floats, ints
    • 8bits, 16bits, 32bits, 64bits data
  • Single channel, RGB, RGBA
  • Expendable to arbitrary multispectral
  • + Metadata thanks to PixpipeObject
  • Size, nb components, min/max, etc.  --> metadata

2D data container

Jonathan Lurie - MCIN - June 2017

Image3D

  • Inherits from PixpipeContainer
  • Generic container for Minc2, NIfTI, MGH
  • _data becomes a JS TypedArray, can handle:
    • Floats, ints
    • 8bits, 16bits, 32bits, 64bits data
  • Single channel or multispectral
  • Handles time series
  • + Metadata thanks to PixpipeObject
  • Size, nb components, min/max, etc.  --> metadata

3D data container

Jonathan Lurie - MCIN - June 2017

MniVolume

  • Inherits from Image3D
  • Adds some logic related to:
    • spatial coordinate system
    • handling specific header metadata

3D data container

Filter

  • Inherits from PixpipeObject
  • Can accept multiple inputs
  • Can create multiple outputs
  • Each I/O objects have an internal identifier (chosen by the dev.)
  • Input integrity checking
  • Time measurement
  • Event manager (mainly for dealing with async behaviour)
  • + Metadata thanks to PixpipeObject

data processor + generic

Jonathan Lurie - MCIN - June 2017

Jonathan Lurie - MCIN - June 2017

ImageToImageFilter

  • Inherits from Filter
  • Adds specific logic to check if all inputs have:
    • the same size
    • the same number of components per pixel

data processor + from 2D to 2D + generic

Jonathan Lurie - MCIN - June 2017

More on Filters

Jonathan Lurie - MCIN - June 2017

Example: apply a threshold

Jonathan Lurie - MCIN - June 2017

A filter is non-destructive

=

myImage2D != myOtherImage2D

Filters from the the main scope

Jonathan Lurie - MCIN - June 2017

Filters from the the main scope

(code)

// let's create an orange image of size 100x50
var myImage = new pixpipe.Image2D({
    width: 100,
    height: 50,
    color: [255, 128, 64, 255]
});

// instantiation of the thresholding filter
var thresholder = new pixpipe.SimpleThresholdFilter();

// feeding the filter with the image
thresholder.addInput( myImage );

// launching the filter
thresholder.update();

// getting the image created by the filter
var myOtherImage = thresholder.getOutput();

Jonathan Lurie - MCIN - June 2017

What about metadata?

(code)

// let's create an orange image of size 100x50
var myImage = new pixpipe.Image2D({
    width: 100,
    height: 50,
    color: [255, 128, 64, 255]
});

// instantiation of the thresholding filter
var thresholder = new pixpipe.SimpleThresholdFilter();

// define a lower threshold: 100 (default: 128)
thresholder.setMetadata("threshold", 100);

// feeding the filter with the image
thresholder.addInput( myImage );

// launching the filter
thresholder.update();

// getting the image created by the filter
var myOtherImage = thresholder.getOutput();

Jonathan Lurie - MCIN - June 2017

Different kinds of filters

  • Image processing filters [i][o][m]
  • Decoders [i][o][m]
  • Helpers [i][o][m]
  • Readers from File or URL [o][m]
  • Writers (to file or Canvas) [i][m]

for doing different kinds of things

[i] accepts input(s)

[o] gives output(s)

[m] Settings with metadata

Jonathan Lurie - MCIN - June 2017

Filter's methods available

// add an input
myFilter.addInput( myImage );

// specify metadata (usually overwrite default)
myFilter.setMetadata( "metaName", "metaValue" );

// launch the filter
myFilter.update();

// get the default output
myFilter.getOutput();

There are only 4 of them, easy to remember

Jonathan Lurie - MCIN - June 2017

More than one input or output?


// add an input with no given ID:
myFilter.addInput( myImage );

// ... is equivalent to that:
myFilter.addInput( myImage, 0 );

// but some filters require more than one input (ie. GradientImageFilter)
// (there names are given in the doc)
// Then, ID must be specified:
myFilter.addInput( myImage1, "dx" );
myFilter.addInput( myImage2, "dy" );

// The same goes for the outputs. This:
var myOutput = myFilter.getOutput();

// ... is the same as that:
var myOutput = myFilter.getOutput( 0 );

// But some filters create more than one output.
// (There names are given in the doc)
// Example, the GradientImageFilter produces 2 outputs:
var myOutput1 = myFilter.getOutput( "direction" );
var myOutput2 = myFilter.getOutput( "magnitude" );

Jonathan Lurie - MCIN - June 2017

Examples

Jonathan Lurie - MCIN - June 2017

MniVolume 3D visualization (link)

Examples

Jonathan Lurie - MCIN - June 2017

Gradient computing on neurons (link)

Examples

Jonathan Lurie - MCIN - June 2017

Few metrics

  • 1 JS source file to import
  • 8 file formats (2D/3D)
  • 23000 lines (bundled)
  • 500kB minified
  • 35+ code examples
  • 800+ lines of cookbook
  • 3 months of dev (part time)

Jonathan Lurie - MCIN - June 2017

Resources

Jonathan Lurie - MCIN - June 2017

Thanks!

any contributors?

Made with Slides.com