OSH Tutorial Part 8

Javascript Toolkit

Outline

Main features
Toolkit architecture
Technologies & frameworks
Development environment
Build the Toolkit
OSH-js by example: H264 video streaming
OSH-js by example: Map & styler
OSH-js by example: FieldSensor

Event based architecture suitable for real-time or playback
Temporal synchronization of multiple data stream
Styling using configurable stylers
Advanced support for video (H264 and MJPEG)
Generic parser for SWE data (coming soon)

Integrates with any mapping engine, currently
- Leaflet
- OpenLayers
- Cesium JS

Main features

Support for SOS & SPS
Discovery
Ajax HTTP & Websocket
Several existing modules: Orientation, Chart, Video, Map etc..

Main features

Toolkit Architecture

0. EventManager

Use as an Event bus
Contains Event enumeration
Used by main components to communicate

Toolkit Architecture

1. Data Connector

Link between data (Server) and client
No logic, gets only data
Generic
Two are currently available: HTTP & WebSocket

Toolkit Architecture

2. Data Receiver & Data Sender

Use a Data Connector to connect to the Server
Filter data depending on the needs
Generic
Support for SOS & SPS

Toolkit Architecture

2. Data Receiver - SOS

Use WebSocket Connector
Support multithreading using WebWorkers

Toolkit Architecture

2. Data Sender - SPS

Use HTTP Ajax Connector

Toolkit Architecture

3. Buffer

Synchronize playback & real-time data
Handle DataReceivers & Entities
Callback results to EventManager
Is embedded into the DataReceiverController or can be used as standalone

Toolkit Architecture

3(bis). DataReceiver/DataSender Controller

Helper class to wrap and abstract some logic
Use a Buffer instance
Callback results to EventManager

Toolkit Architecture

4. DataR/S (Receiver/Sender) vs Entity

Entity handles a set of DataR/S
Makes link between several DataR/S defined by different Views:
- 1 click onto a View representing a DataR/S can highlight another view representing a DataR/S contained in the same entity
- Allow firing group events (show/hide/highlight)

Toolkit Architecture

5. Views

Represent the data
Use the EventManager to Get/Send information
Use ViewItem:
- styler
- contextual menu
- entity id(s)
- dataReceiver/dataSender id(s)
Some views don't need viewItem:
- RangeSlider
- Discovery
- Dialog

=> because they don't represent a Data/R/S

Toolkit Architecture

6. Stylers

Allow styling data
Use function to filter data
Generic
Example: data can be displayed as
- text
- icon
- line
- map point
- ...
Are not mandatory

Toolkit Architecture

7. ContextMenu

Represent a menu
Abstract
Can have many representations
- circular
- stack
- ...
Contains menuItem
- name
- viewId
- css classes
A menuItem makes the link between View and ContextMenu (using EventManager)

Technologies & Frameworks

Core

Pure JavaScript
CSS 3
Only works on recent browsers
Framework prototypejs
- Ajax interface
- Utility methods
- Helper methods
- Extending the DOM
- Class Object
- Prototype extends the OO nature of JS
  Class.create() returns a function that acts like a Ruby class, that when called will fire its own initialize() method.

Technologies & Frameworks

External libs

Bootstrap main application (broadband + layouts)
Cesium pure JS 3D Globe
Jsonix convert XML & JSON structures
modernizr JS helper utilities
nouislider time slider bar
NVD3 re-usable chart for d3.js
OpenLayer
Leaflet
Custom tree
FFMPEG portage of FFMPEG C++ library into JS to decode H264 frames

Technologies & Frameworks

Build

NPM package manager for JavaScript
Gulp build system, automating tasks
- Gulp modules
- gulp-jshint
- gulp-minify
- gulp-sort
- cleanCSS
- ...
JSDoc build documentation
Karma for tests

Development environment

Git
Text editor or IDE (Eclipse-js, WebStorm ...)
Any local httpd server (apache-http)
Local/Remote OSH server
Recent version of Chrome/Chromium/Firefox

Build the Toolkit

Install dependencies using NPM:
Build a full version using gulp:
Build a full minified version:
Clean:
Documentation: http://opensensorhub.github.io/osh-js/Toolkit/Documentation/index.html

$ npm install

$ gulp build

$ gulp build-minify

$ gulp clean

clone git@github.com:opensensorhub/osh-js.git

OSH-js by example

Part 1: create the client "Drone Real-Time Video Draping onto Terrain"

OSH-js by example

What we want

OSH-js by example

<swes:offering>
    <sos:ObservationOffering>
        <swes:description>Archive data from Solo Nav DataStore</swes:description>
        <swes:identifier>urn:osh:solo-nav</swes:identifier>
        <swes:name>Solo Nav DataStore</swes:name>
        <swes:procedure>urn:osh:sensor:mavlink:solo:S115A5800419</swes:procedure>
        <swes:observableProperty>http://www.opengis.net/def/property/OGC/0/PlatformOrientation</swes:observableProperty>
        <swes:observableProperty>http://sensorml.com/ont/swe/property/OSH/0/GimbalOrientation</swes:observableProperty>
        <swes:observableProperty>http://www.opengis.net/def/property/OGC/0/PlatformLocation</swes:observableProperty>
        <sos:phenomenonTime>
            <gml:TimePeriod gml:id="T4">
                <gml:beginPosition>2016-08-30T18:58:46.180Z</gml:beginPosition>
                <gml:endPosition>2016-08-30T19:20:17.002Z</gml:endPosition>
            </gml:TimePeriod>
        </sos:phenomenonTime>
        ...
</swes:offering>

<swes:offering>
    <sos:ObservationOffering>
        <swes:description>Archive data from Solo Video DataStore</swes:description>
        <swes:identifier>urn:osh:solo-video</swes:identifier>
        <swes:name>Solo Video DataStore</swes:name>
        <swes:procedure>urn:osh:sensor:rtpcam:solo:S115A5800419</swes:procedure>
        <swes:observableProperty>http://sensorml.com/ont/swe/property/VideoFrame</swes:observableProperty>
        <swes:observableProperty>http://sensorml.com/ont/swe/property/Image</swes:observableProperty>
        <sos:phenomenonTime>
            <gml:TimePeriod gml:id="T5">
                <gml:beginPosition>2016-08-30T18:59:03.009Z</gml:beginPosition>
                <gml:endPosition>2016-08-30T19:20:16.402Z</gml:endPosition>
            </gml:TimePeriod>
        </sos:phenomenonTime>
        ...
</swes:offering>

Get information reading GetCapabilities

Looking for solo data

Offering

ObservableProperty

Time

OSH-js by example

DataReceiver

PlatformLocation => LatLon
GimbalOrientation => EulerOrientation
Video (H264) => VideoH264

Views

CesiumView => 3D map
DialogView => encapsulates video + chart viewers
RangeSlider => control time
Nvd3CurveChartView => Chart
FFMPEGView => Display Video using FFMPEG JS
EntityTreeView => display entities
StackMenu => for Tree

OSH-js by example

<body>
 <div id="container-1" class="mini-view"></div> <!-- Video View -->
 <div id="container-2" class="mini-view"></div> <!-- Chart View -->
 <div id="container-3" class="mini-view"></div> <!-- Cesium View -->
</body>

Define simple HTML layout
Need a container for each view

OSH-js by example

// DEFINE TIME
var startTime = "2016-08-30T19:00:40Z";
var endTime = "2016-08-30T19:22:00Z";
var replaySpeed = "1";

// LatLon DataReceiver	
var soloGPS = new OSH.DataReceiver.LatLonAlt("Solo GPS", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "urn:osh:solo-nav",
    observedProperty: "http://www.opengis.net/def/property/OGC/0/PlatformLocation",
    startTime: startTime,
    endTime: endTime,
    replaySpeed: replaySpeed,
    syncMasterTime: true
});

// Orientation DataReceiver
var soloAttitude = new OSH.DataReceiver.EulerOrientation("Solo Attitude", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "urn:osh:solo-nav",
    observedProperty: "http://www.opengis.net/def/property/OGC/0/PlatformOrientation",
    startTime: startTime,
    endTime: endTime,
    replaySpeed: replaySpeed,
    syncMasterTime: true
});

Synchronize the data

OSH-js by example

// Gimbal orientation DataReceiver
var soloGimbal = new OSH.DataReceiver.EulerOrientation("Solo Gimbal", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "urn:osh:solo-nav",
    observedProperty: "http://sensorml.com/ont/swe/property/OSH/0/GimbalOrientation",
    startTime: startTime,
    endTime: endTime,
    replaySpeed: replaySpeed,
    syncMasterTime: true
});



// Video H264 DataReceiver
var soloVideo = new OSH.DataReceiver.VideoH264("Solo Video", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "urn:osh:solo-video",
    observedProperty: "http://sensorml.com/ont/swe/property/VideoFrame",
    startTime: startTime,
    endTime: endTime,
    replaySpeed: replaySpeed,
    timeShift: -100,
    syncMasterTime: true
});

OSH-js by example

//--------------------------------------------------------------//
//-------------------------- Entities --------------------------//
//--------------------------------------------------------------//
var soloEntity = {
    id: "solo1",
    name: "3DR Solo",
    dataSources: [soloGPS, soloAttitude, soloGimbal, soloVideo]
};

Create an entity to group dataReceiver
The name is the one displayed by the EntityTreeViewer

OSH-js by example

Video View

OSH-js by example

//--------------------------------------------------------------//
//--------------------------- Views  ---------------------------//
//--------------------------------------------------------------//

    
var soloVideoView = new OSH.UI.FFMPEGView("container-1", {
    dataSourceId: soloVideo.getId(),
    entityId : soloEntity.id,
    css: "video",
    cssSelected: "video-selected",
    useWorker: true,
    width: 1280,
    height: 720
});

Create a Video View
FFMPEGView decode and render frame
Setup multithreading using WebWorker

OSH-js by example

Chart View

OSH-js by example

//--------------------------------------------------------------//
//--------------------------- Views  ---------------------------//
//--------------------------------------------------------------//

    
var altChartView = new OSH.UI.Nvd3CurveChartView("container-2",
[{
    styler: new OSH.UI.Styler.Curve({
        valuesFunc: {
            dataSourceIds: [soloGPS.getId()],
            handler: function (rec, timeStamp) {
            	if (rec.alt < 1)
            	    rec.alt *= 1e4; // *10^4 due to bug in Toulouse dataset
                return {
                    x: timeStamp,
                    y: rec.alt+mslToWgs84
                };
            }
        }
    })
}],
{
    dataSourceId: soloGPS.getId(),
    yLabel: 'Altitude (m)',
    xLabel: 'Time',
    maxPoints: 100,
    css:"chart-view",
    cssSelected: "video-selected"
});

Using NVD3 View to render altitude

We use styler to apply an altitude rectification

Define Graph View properties

OSH-js by example

//--------------------------------------------------------------//
//--------------------------- Views  ---------------------------//
//--------------------------------------------------------------//

// common point marker
var pointMarker = new OSH.UI.Styler.PointMarker({
	label: "3DR Solo",
    locationFunc : {
        dataSourceIds : [soloGPS.getId()],
        handler : function(rec) {
        	if (rec.alt < 1)
        	    rec.alt *= 1e4; // *10^4 due to bug in Toulouse dataset
            return {
                x : rec.lon,
                y : rec.lat,
                z : rec.alt+mslToWgs84-5. // model offset
            };
        }
    },
    orientationFunc : {
        dataSourceIds : [soloAttitude.getId()],
        handler : function(rec) {
            return {
                heading : rec.heading
            };
        }
    },
    icon: "./models/Drone+06B.glb"
});

Define PointMarker to style latLon + Alt data

We use styler to add MSL to LatLon data

Get heading from altitude dataReceiver

OSH-js by example

var imageDrapingStyler = new OSH.UI.Styler.ImageDraping({
    platformLocationFunc: {
        dataSourceIds: [soloGPS.getId()],
        handler: function(rec) {
            if (rec.alt < 1)
                rec.alt *= 1e4; // *10^4 due to bug in Toulouse dataset
            return {
                x: rec.lon,
                y: rec.lat,
                z: rec.alt + mslToWgs84
            };
        }
    },
    platformOrientationFunc: {
        dataSourceIds: [soloAttitude.getId()],
        handler: function(rec) {
            return {
                heading: rec.heading,
                pitch: 0, //rec.pitch,
                roll: 0, //rec.roll
            };
        }
    },
    gimbalOrientationFunc: {
        dataSourceIds: [soloGimbal.getId()],
        handler: function(rec) {
            return {
                heading: rec.heading,
                pitch: -92, //rec.pitch,
                roll: 0, //rec.roll
            };
        }
    },
    /*GoPro Mike*/
    cameraModel: {
        camProj: new Cesium.Matrix3(747.963 / 1280., 0.0, 650.66 / 1280.,
            0.0, 769.576 / 738., 373.206 / 738.,
            0.0, 0.0, 1.0),
        camDistR: new Cesium.Cartesian3(-2.644e-01, 8.4e-02, 0.0),
        camDistT: new Cesium.Cartesian2(-8.688e-04, 6.123e-04)
    },
    imageSrc: $$('#' + soloVideoView.getId() + ' canvas')[0]
})

Define ImageDrapingMarker

Location

Camera setup

EulerOrientation

GimbalOrientation

OSH-js by example

Cesium View

OSH-js by example

 var mapView = new OSH.UI.CesiumView("container-3", [
{
     name: "3DR Solo",
     entityId: soloEntity.id,
     styler: pointMarkerStyler
 }, {
     name: "Geolocated Imagery",
     entityId: soloEntity.id,
     styler: imageDrapingStyler
 }]);

Create view items with
- name
- entity Id
- Styler

ViewItem 1

ViewItem 2

OSH-js by example

var dataSourceController = new OSH.DataReceiver.DataReceiverController({
        replayFactor: 1.0
});

dataSourceController.addEntity(soloEntity);
dataSourceController.connectAll();
// or fire an event
//OSH.EventManager.fire(OSH.EventManager.EVENT.CONNECT_DATASOURCE,
// {dataSourcesId:[videoDataSource.id]});

Finally create the dataReceiverController to manage dataR/S

OSH-js by example

Add extra stuff

OSH-js by example

Decorate container using floating dialog windows
Add contextual menu
Add entity tree viewer
Add range slider

OSH-js by example

Dialog window

OSH-js by example

Dialog window can decorate any div
Extra functionnalities to get a nice render:
- dock
- drag
- close
- swap
- show/hide
- connect/disconnect any DataReceiver

OSH-js by example

Change the main HTML layout

<div id="main-container" class="main-view"></div>
<div id="dialog-main-container" class="video-main-container"></div>

OSH-js by example

Decorate FFMPEG View

// video view    
var soloVideoDialog = new OSH.UI.DialogView("dialog-main-container", {
    draggable: false,
    css: "video-dialog",
    name: "UAV Video",
    show: true,
    dockable: true,
    closeable: true,
    canDisconnect : false,
    swapId: "main-container"
});

var soloVideoView = new OSH.UI.FFMPEGView(soloVideoDialog.popContentDiv.id, {
    dataSourceId: soloVideo.getId(),
    entityId : soloEntity.id,
    css: "video",
    cssSelected: "video-selected",
    useWorker: true,
    width: 1280,
    height: 720
});

plug the dialog to a main container

set the dialog div id instead of container id

OSH-js by example

Decorate Other Views

// chart view
var altChartDialog = new OSH.UI.DialogView("dialog-main-container", {
    draggable: false,
    css: "dialog",
    name: "Solo Altitude",
    show: false,
    dockable: true,
    closeable: true,
    canDisconnect : false,
    swapId: "main-container"
});

var altChartView = new OSH.UI.Nvd3CurveChartView(altChartDialog.popContentDiv.id, ...);

// Cesium view is plugged as background 
var mapView = new OSH.UI.CesiumView("main-container",...)

plug the dialog to a main container

Cesium is set as background

OSH-js by example

Contextual Menu

OSH-js by example

Menu can be used in any View
Receive and Send events using EventManager
Contain MenuItem with
- name
- viewId
- css
- action
Can be grouped
- views use groupId

OSH-js by example

Define menu id used in the different views

Add menuId to Cesium viewItem

// menu ids
var soloTreeMenuId = "solo-tree-menu";
var soloMarkerMenuId = "solo-marker-menu";
var menuGroupId = "allmenus";

// cesium map view
var mapView = new OSH.UI.CesiumView("main-container", [{
  name: "3DR Solo",
  entityId: soloEntity.id,
  styler: pointMarker,
  contextMenuId: soloMarkerMenuId
}
[...]

OSH-js by example

Define menu items

var menuItems = [{
    name: "Show Video",
    viewId: soloVideoDialog.getId(),
    css: "fa fa-video-camera",
    action: "show"
}, {
    name: "Show Altitude Chart",
    viewId: altChartDialog.getId(),
    css: "fa fa-bar-chart",
    action: "show"
}, {
    name: "TakeOff",
    viewId: "",
    css: "fa fa-upload",
    action: "uav:takeoff"
}, {
    name: "Land",
    viewId: "",
    css: "fa fa-download",
    action: "uav:land"
}];

var markerMenu = new OSH.UI.ContextMenu.CircularMenu({
    id: soloMarkerMenuId,
    groupId: menuGroupId,
    items: menuItems
});

Can also use EventManager:

OSH.EventManager.EVENT.SHOW_VIEW

OSH-js by example

EntityTreeViewer

OSH-js by example

Add EntityTreeView
Show the whole list of entities and DataReceivers
Has menu to perform actions
inherit from OSH.UI.View

OSH-js by example

Can be plugged into Dialog

 // tree view
var entityTreeDialog = new OSH.UI.DialogView(document.body.id, {
    css: "tree-dialog",
    name: "Entities",
    show: true,
    draggable: true,
    dockable: false,
    closeable: true
});

var entityTreeView = new OSH.UI.EntityTreeView(entityTreeDialog.popContentDiv.id,
    [{
        entity : soloEntity,
        path: "Sensors/Solo",
        treeIcon : "images/drone.png",
        contextMenuId: soloTreeMenuId
    }],
    {
        css: "tree-container"
    }
);

OSH-js by example

Range Slider

OSH-js by example

Finally add a RangeSlider to change time
Send events to change dynamically DataReceiver request
Use EventManager to communicate
Support playback & Real-time (data cannot be changed in that mode)

OSH-js by example

Update HTML Layout & CSS

Since it uses EventManager to communicate, we have only to instantiate a new Object

For real-time we would set:

<div class="rangeSlider-container">
    <div id="rangeSlider" class="rangeSlider"></div>
</div>

 var rangeSlider = new OSH.UI.RangeSlider("rangeSlider",{
    startTime: "2015-02-16T07:58:00Z",
    endTime: "2015-02-16T08:09:00Z",
    refreshRate:1
});

 var rangeSlider = new OSH.UI.RangeSlider("rangeSlider");

OSH-js by example

Part 2: make your own DataReceiver

OSH-js by example

Get info from GetCapabilities

offeringID: fake-offering
startTime: 2016-08-30T19:00:40Z
endTime: 2016-08-30T19:22:00Z
observedProperty: http://www.opengis.net/def/property/OGC/0/PlatformLocation

OSH-js by example

Create a new DataReceiver

We suppose the server is sending "timeStamp,fakeLat,fakeLon,fakeAlt"

OSH.DataReceiver.MyFakeLatLonAlt = Class.create(OSH.DataReceiver.DataSource,{

    parseTimeStamp: function($super,data){
        // do something
        // this part extract the timestamp from data
        // the data could be text, binary etc..
    },
    
    parseData: function($super,data){
        // do something
        // this part extract only the data without timestamp
    }
});

OSH-js by example

Extract the timeStamp

Every data sent by the server includes a timestamp, the client has to extracted it
The server is sending "1977-04-22T06:00:00Z,2.464,5.278,30.0" as a String

parseTimeStamp: function($super,data){
    // get record from websocket
    var record = String.fromCharCode.apply(null, new Uint8Array(data));

    // split the record using "," as separator
    var tokens = record.trim().split(",");

    // return the timeStamp using Data JS Object and getTime (milliseconds)
    return new Date(tokens[0]).getTime();
}

OSH-js by example

Extract the data

The server is sending "1977-04-22T06:00:00Z,2.464,5.278,30.0" as a String
Extract 1977-04-22T06:00:00Z,2.464,5.278,30.0 and put together in a JS object

parseData: function($super,data){
    // get data from WebSocket
    var record = String.fromCharCode.apply(null, new Uint8Array(data));
    
    // split String using "," as separator
    var tokens = record.trim().split(",");

    // get lat, lon, alt
    var lat = parseFloat(tokens[1]);
    var lon = parseFloat(tokens[2]);
    var alt = parseFloat(tokens[3]);
    
    // return JS object
    return {
      lat : lat,
      lon : lon,
      alt : alt
    };
}

OSH-js by example

Instantiate your new DataReceiver

var myFakeLatLonAlt = new OSH.DataReceiver.MyFakeLatLonAlt("Fake GPS", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "fake-offering",
    observedProperty: "http://www.opengis.net/def/property/OGC/0/PlatformLocation",
    startTime: 2016-08-30T19:00:40Z,
    endTime: 2016-08-30T19:22:00Z,
    replaySpeed: 1,
    syncMasterTime: true
});

OSH-js by example

Part 3: Use the Discovery View

OSH-js by example

Select service
Select offering
Select observableProperty
Select Time (start/end)
Sync master time or not
Attach to an existing entity
The kind of view

OSH-js by example

Discovery demo

OSH-js by example

var discoveryView = new OSH.UI.DiscoveryView("",{
    services: ["http://localhost:8181/","http://sensiasoft.net:8181/"],
    css: "discovery-view",
    dataReceiverController:dataProviderController,
    swapId: "center-container",
    entities: [androidEntity],
    views: [{
        name: 'Leaflet 2D Map',
        viewId: leafletMainView.id,
        type : OSH.UI.DiscoveryView.Type.MARKER_GPS
    }, {
        name: 'Cesium 3D Globe',
        viewId: cesiumMainMapView.id,
        type : OSH.UI.DiscoveryView.Type.MARKER_GPS
    },{
        name: 'Video dialog(H264)',
        type : OSH.UI.DiscoveryView.Type.DIALOG_VIDEO_H264
    },{
        name: 'Video dialog(MJPEG)',
        type : OSH.UI.DiscoveryView.Type.DIALOG_VIDEO_MJPEG
    },{
        name: 'Chart dialog',
        type : OSH.UI.DiscoveryView.Type.DIALOG_CHART
    }]
});