Image processing pipeline in Javascript
Jonathan Lurie - MCIN - June 2017
- What is it?
- Motivations
- Possible use cases
- What's a pipeline?
- Under the hood
- More on Filters
- Examples
- Few metrics
- Ressources
Jonathan Lurie - MCIN - June 2017
What is it?
- A way to process images in the browser
- A Javascript-only codebase
- A ES6 bundle (npm style)
- A headless library
- Detached from any visualization engine
- A project started at the BrainHack Mtl
Jonathan Lurie - MCIN - June 2017
Jonathan Lurie - MCIN - June 2017
Motivations
- A web browser is the most accessible software
- Make something easy to use
- With no other dependency than pixpipe.js
- With no required compilation or system fuss
- Modular architecture
- Generic enough to use different kind of data/datasource
- Well documented for both users and contributors
- The more on the browser, the less on the server
Jonathan Lurie - MCIN - June 2017
Possible use cases
- Open 2D image files (png, jpg, tiff 8/16bits)
- Open 3D image files (Minc2, NIfTI, MGH)
- Open local (file dialog) or distant (AJAX) files
- Perform statistics
- Extract slices on a given axis (from 3D)
- Create mosaics (from 3D)
- Process the data
- 2D convolution
- Compute derivatives and gradients
- Contour detection
- Blending, scaling, pixel-wise treatment ...
- Save your result locally
- etc.
Jonathan Lurie - MCIN - June 2017
What's a pipeline?
à la ITK
Jonathan Lurie - MCIN - June 2017
Under the hood
Jonathan Lurie - MCIN - June 2017
Core architecture
Jonathan Lurie - MCIN - June 2017
PixpipeObject
- Every object inherits from this one
- Carries all the metadata logic
- Generates a unique identifier (uuid) at instanciation
* The most generic *
Jonathan Lurie - MCIN - June 2017
PixpipeContainer
- Inherits from PixpipeObject
- Adds the _data attribute (of unspecified type)
- But still cannot do anything!
data container + generic
Jonathan Lurie - MCIN - June 2017
Image2D
- Inherits from PixpipeContainer
- Generic container for jpeg, png, tiff
-
_data becomes a JS TypedArray, can handle:
- Floats, ints
- 8bits, 16bits, 32bits, 64bits data
- Single channel, RGB, RGBA
- Expendable to arbitrary multispectral
- + Metadata thanks to PixpipeObject
- Size, nb components, min/max, etc. --> metadata
2D data container
Jonathan Lurie - MCIN - June 2017
Image3D
- Inherits from PixpipeContainer
- Generic container for Minc2, NIfTI, MGH
-
_data becomes a JS TypedArray, can handle:
- Floats, ints
- 8bits, 16bits, 32bits, 64bits data
- Single channel or multispectral
- Handles time series
- + Metadata thanks to PixpipeObject
- Size, nb components, min/max, etc. --> metadata
3D data container
Jonathan Lurie - MCIN - June 2017
MniVolume
- Inherits from Image3D
- Adds some logic related to:
- spatial coordinate system
- handling specific header metadata
3D data container
Filter
- Inherits from PixpipeObject
- Can accept multiple inputs
- Can create multiple outputs
- Each I/O objects have an internal identifier (chosen by the dev.)
- Input integrity checking
- Time measurement
- Event manager (mainly for dealing with async behaviour)
- + Metadata thanks to PixpipeObject
data processor + generic
Jonathan Lurie - MCIN - June 2017
Jonathan Lurie - MCIN - June 2017
ImageToImageFilter
- Inherits from Filter
- Adds specific logic to check if all inputs have:
- the same size
- the same number of components per pixel
data processor + from 2D to 2D + generic
Jonathan Lurie - MCIN - June 2017
More on Filters
Jonathan Lurie - MCIN - June 2017
Example: apply a threshold
Jonathan Lurie - MCIN - June 2017
A filter is non-destructive
=
myImage2D != myOtherImage2D
Filters from the the main scope
Jonathan Lurie - MCIN - June 2017
Filters from the the main scope
(code)
// let's create an orange image of size 100x50
var myImage = new pixpipe.Image2D({
width: 100,
height: 50,
color: [255, 128, 64, 255]
});
// instantiation of the thresholding filter
var thresholder = new pixpipe.SimpleThresholdFilter();
// feeding the filter with the image
thresholder.addInput( myImage );
// launching the filter
thresholder.update();
// getting the image created by the filter
var myOtherImage = thresholder.getOutput();
Jonathan Lurie - MCIN - June 2017
What about metadata?
(code)
// let's create an orange image of size 100x50
var myImage = new pixpipe.Image2D({
width: 100,
height: 50,
color: [255, 128, 64, 255]
});
// instantiation of the thresholding filter
var thresholder = new pixpipe.SimpleThresholdFilter();
// define a lower threshold: 100 (default: 128)
thresholder.setMetadata("threshold", 100);
// feeding the filter with the image
thresholder.addInput( myImage );
// launching the filter
thresholder.update();
// getting the image created by the filter
var myOtherImage = thresholder.getOutput();
Jonathan Lurie - MCIN - June 2017
Different kinds of filters
- Image processing filters [i][o][m]
- Decoders [i][o][m]
- Helpers [i][o][m]
- Readers from File or URL [o][m]
- Writers (to file or Canvas) [i][m]
for doing different kinds of things
[i] accepts input(s)
[o] gives output(s)
[m] Settings with metadata
Jonathan Lurie - MCIN - June 2017
Filter's methods available
// add an input
myFilter.addInput( myImage );
// specify metadata (usually overwrite default)
myFilter.setMetadata( "metaName", "metaValue" );
// launch the filter
myFilter.update();
// get the default output
myFilter.getOutput();
There are only 4 of them, easy to remember
Jonathan Lurie - MCIN - June 2017
More than one input or output?
// add an input with no given ID:
myFilter.addInput( myImage );
// ... is equivalent to that:
myFilter.addInput( myImage, 0 );
// but some filters require more than one input (ie. GradientImageFilter)
// (there names are given in the doc)
// Then, ID must be specified:
myFilter.addInput( myImage1, "dx" );
myFilter.addInput( myImage2, "dy" );
// The same goes for the outputs. This:
var myOutput = myFilter.getOutput();
// ... is the same as that:
var myOutput = myFilter.getOutput( 0 );
// But some filters create more than one output.
// (There names are given in the doc)
// Example, the GradientImageFilter produces 2 outputs:
var myOutput1 = myFilter.getOutput( "direction" );
var myOutput2 = myFilter.getOutput( "magnitude" );
Jonathan Lurie - MCIN - June 2017
Examples
Jonathan Lurie - MCIN - June 2017
MniVolume 3D visualization (link)
Examples
Jonathan Lurie - MCIN - June 2017
Gradient computing on neurons (link)
Examples
Jonathan Lurie - MCIN - June 2017
Few metrics
- 1 JS source file to import
- 8 file formats (2D/3D)
- 23000 lines (bundled)
- 500kB minified
- 35+ code examples
- 800+ lines of cookbook
- 3 months of dev (part time)
Jonathan Lurie - MCIN - June 2017
Resources
- The repo on Github
- The cookbook (with the example section)
- The doc
- Dependencies' repos
- Pako - gunziping in JS
- FileSaver.js - trigger a file download
- expr-eval - mathematical expression evaluator
- js-md5 - checksum computing
- Geotiff.js - Parse Tiff/Geotiff file in JS
Jonathan Lurie - MCIN - June 2017
Thanks!
any contributors?
Pixpipejs - intro
By jonathanlurie
Pixpipejs - intro
Introduction of PixpipeJs
- 1,314