Discover huge raster files in the Browser with geotiff.js

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Fabian Schindler

https://twitter.com/__fschindler__

https://github.com/constantinius

https://eox.at

https://slides.com/fschindler/discover-huge-raster-files-in-the-browser-with-geotiff-js

geotiff.js

Source code: https://github.com/geotiffjs/geotiff.js/

Documentation: https://geotiffjs.github.io/geotiff.js/

Pure JavaScript implementation of (Geo)TIFF
Support for browsers and node
Focusing ease of use + performance
Async approach

Since v0.4.1: rewrite

v1.0.0 is around the corner(-ish)

(Geo)TIFF

Proven standard
Well adopted
Images and "More"
- (U)Int / Float raster data
- Multiple images per file
- Metadata

(Geo) TIFF Benefits

All-in-one: Data + Metadata
Multiple resolutions of the same image (Overviews)
More than 8-bit RGB(A)
- Arbitrary number of Bands/Channels
- Arbitrary integer/float sizes

Cloud Optimized GeoTIFFs

(COG) - https://www.cogeo.org/

Conventions for cloud native TIFF files
Layout friendly for network consumption

Cloud Optimized GeoTIFFs

Recommendations
- Tiled
- Compressed
- Overview images
File structure

COG Benefit

Efficient access to subsets of large raster files

(... even via network)

TIFF layout - Header/IFD

Header

Endianness

bigtiff?

first IFD

IFD 1

size

tag 1

tag 2

...

next IFD

IFD 2

size

tag 1

tag 2

...

next IFD

IFD n

size

tag 1

tag 2

...

next IFD

TIFF layout - Tags

IFD

ID - 1	type
ID - 2	type
ID - 3	type
...

Value

Reference

Value

TIFF layout - Raster data

IFD

...
...
ID	type
...
...

Reference

Raster data

Header

IFD

IFD Data

Raster Data

TIFF file layout

Header

IFD-1

IFD-1 Data

Raster Data n

COG-TIFF file layout

...

IFD-2

IFD-2 Data

IFD-n

IFD-n Data

...

Raster Data n-1

...

Raster Data 1

...

image size

Challenges - TIFF related

Very diverse format
- Stripped/Tiled
- Compression
- Planar configuration
- Color representation
- Data type
- Internal/external Overviews
Many indirections
Potentially huge files

Challenges - Network related

Bandwidth
Concurrent requests
Memory

TIFF related

Handled by Reader/Decoder

HTTP Range requests

Allows to get a subset of the file
Can be detected with HEAD requests

GET /file.tiff HTTP/1.1
Host: example.com
Range: bytes=0-1023

Does it solve the problem?

TIFF files have many indirections
- Header → first IFD
- IFD → next IFD
- Tag → Value
- Strip/Tile table → Raster data
Many requests have to be sent

→ an additional abstraction layer needed

Handled by Source

Data Flow

Raw bytes

Tiles / rows

Data Arrays

Layers of abstraction

Source

Reader

Decoder

Reader

Select appropriate IFD
Fetch the required bytes from source
Execute the decoding of the rows/tiles
Perform transformation to RGB

low-level

Read data from a file or specific image
Subsetting/Scaling

Subsetting / Rescaling

Window/BBox
Optional resampling to desired size

RGB Transformation

Grayscale
Palette
CMYK
YCbCr
CIE L*a*b*

Output → uniform RGB

Source

Abstraction of (remote) files
simple interface

const source = {
  async fetch(offset, length) {
    // ...
    return data;  // ArrayBuffer instance
  }
};

Built-in sources

ArrayBuffer
remote (XHR/fetch/node HTTP API)
FileAPI

Remote (HTTP) sources

Use HTTP range requests

GET /file.tiff HTTP/1.1
Host: example.com
Range: bytes=0-1023

BlockedSource

Wraps any type of source
- Best for remote files
Chunks requests to blocks
Serves as a low-level cache
Configurable block and cache size

IFD

IFD Data

Raster Data

Blocked Source

Benefits

Fewer requests
- Slices may already be cached
- Consecutive blocks → single request
LRU-cache → only recently used parts are cached

Decoder

Decodes compressed data
- Deflate, LZW, JPEG, Packbits
Work splitting using thread pools

Examples

High level usage

import GeoTIFF from 'geotiff';

// ...

const tiff = await GeoTIFF.fromUrl(someUrl);
const image = await tiff.getImage();

const width = image.getWidth();
const height = image.getHeight();
const tileWidth = image.getTileWidth();
const tileHeight = image.getTileHeight();
const samplesPerPixel = image.getSamplesPerPixel();

const origin = image.getOrigin();
const resolution = image.getResolution();
const bbox = image.getBoundingBox();

Importing, loading and metadata

High level usage

const data = await image.readRasters();
const { width, height } = data;

const left = 50;
const top = 10;
const right = 150;
const bottom = 60;

const data = await image.readRasters({ 
  window: [left, top, right, bottom],
  width: 100,
  height: 100,
  resampleMethod: 'bilinear',
});

const data = await tiff.readRasters({
  bbox: [10.34, 57.28, 13.34, 60.23],
  resX: 0.1,
  resY: 0.1
});

Read raster from TIFF

Read raster from specific image

Subsetting / Scaling

Opening a file

<input type="file" id="file">
<script>
  const input = document.getElementById('file'):
  input.onchange = async function() {
    const tiff = await GeoTIFF.fromBlob(input.files[0]);
  }
</script>

Opening with external overviews

const tiff = await GeoTIFF.fromUrls(
  'LC08_L1TP_189027_20170403_20170414_01_T1_B3.TIF',
  ['LC08_L1TP_189027_20170403_20170414_01_T1_B3.TIF.ovr']
);

Custom source

const tiff = await GeoTIFF.GeoTIFF.fromSource({
  async fetch(offset, length) {
    // ...
    return data;
  }
});

Applications built with geotiff.js

TAMP

COG Explorer

https://geotiffjs.github.io/cog-explorer/#long=87.231&lat=21.664&zoom=9&scene=https://landsat-pds.s3.amazonaws.com/c1/L8/139/045/LC08_L1TP_139045_20170304_20170316_01_T1/index.html&bands=4,3,2&pipeline=sigmoidal(all,50,0.16);gamma(red,1.03);gamma(blue,0.925)

COG Explorer

Web App
Proof of concept
Visualization of TIFFs on a Map Widget
Dynamic color correction
RGB Composition from multiple sources

geotiff.io

http://app.geotiff.io/sum?url=https://s3.amazonaws.com/geotiff.io/PuertoRicoTropicalFruit.tiff

georaster-layer-for-leaflet

https://geotiff.github.io/georaster-layer-for-leaflet-example/examples/load-cog-via-script-tag.html

Leaflet.TileLayer.GL + geotiff.js

Copernicus EMS Activations viewer

https://emergency.copernicus.eu/mapping/ems/copernicus-ems-activations-viewer-concept-version

Any more?

Community

Never planned as a community project
Officially presented at foss4g 2016 (Bonn)
13 Contributors
36 closed / 30 open issues

Contributions are welcome!

Next steps

Arbitrary width integers and 16/24 bit floats
HTTP Multi-ranges (PR pending)
Extended write support
More compression algorithms
Projection in WKT format

Discover huge raster files in the Browser with geotiff.js

geotiff.js

(Geo)TIFF

(Geo) TIFF Benefits

Cloud Optimized GeoTIFFs

Cloud Optimized GeoTIFFs

COG Benefit

TIFF layout - Header/IFD

TIFF layout - Tags

TIFF layout - Raster data

TIFF file layout

COG-TIFF file layout

Challenges - TIFF related

Challenges - Network related

TIFF related

HTTP Range requests

Does it solve the problem?

Data Flow

Layers of abstraction

Reader

Subsetting / Rescaling

RGB Transformation

Source

Built-in sources

Remote (HTTP) sources

BlockedSource

Blocked Source

Benefits

Decoder

Examples

High level usage

High level usage

Opening a file

Opening with external overviews

Custom source

Applications built with geotiff.js

TAMP

COG Explorer

COG Explorer

geotiff.io

georaster-layer-for-leaflet

Leaflet.TileLayer.GL + geotiff.js

Copernicus EMS Activations viewer

Any more?

Community

Next steps

Reprojection?

tiledirectory.js

Thank you!

Discover huge raster files in the Browser with geotiff.js

More from Fabian Schindler