OSH Tutorial Part 8

Javascript Toolkit

Outline

  • Main features
  • Toolkit architecture
  • Technologies & frameworks
  • Development environment
  • Build the Toolkit
  • OSH-js by example: H264 video streaming
  • OSH-js by example: Map & styler
  • OSH-js by example: FieldSensor
  • Event based architecture suitable for real-time or playback
  • Temporal synchronization of multiple data stream
  • Styling using configurable stylers
  • Advanced support for video (H264 and MJPEG)
  • Generic parser for SWE data (coming soon)
  • Integrates with any mapping engine, currently
    • Leaflet
    • OpenLayers
    • Cesium JS

Main features

  • Support for SOS & SPS
  • Discovery 
  • Ajax HTTP & Websocket
  • Several existing modules: Orientation, Chart, Video, Map etc..

Main features

Toolkit Architecture

Toolkit Architecture

0. EventManager

  • Use as an Event bus
  • Contains Event enumeration
  • Used by main components to communicate

Toolkit Architecture

1. Data Connector

  • Link between data (Server) and client
  • No logic, gets only data
  • Generic
  • Two are currently available: HTTP & WebSocket

Toolkit Architecture

2. Data Receiver & Data Sender

  • Use a Data Connector to connect to the Server
  • Filter data depending on the needs
  • Generic
  • Support for SOS & SPS

Toolkit Architecture

2. Data Receiver - SOS

  • Use WebSocket Connector
  • Support multithreading using WebWorkers

Toolkit Architecture

2. Data Sender - SPS

  • Use HTTP Ajax Connector

Toolkit Architecture

3. Buffer

  • Synchronize playback & real-time data
  • Handle DataReceivers & Entities
  • Callback results to EventManager
  • Is embedded into the DataReceiverController or can be used as standalone

Toolkit Architecture

3(bis). DataReceiver/DataSender Controller

  • Helper class to wrap and abstract some logic
  • Use a Buffer instance
  • Callback results to EventManager

Toolkit Architecture

4. DataR/S (Receiver/Sender) vs Entity

  • Entity handles a set of DataR/S
  • Makes link between several DataR/S defined by different Views:
    • 1 click onto a View representing a DataR/S can highlight another view representing a DataR/S contained in the same entity
    • Allow firing group events (show/hide/highlight)

Toolkit Architecture

5. Views

  • Represent the data
  • Use the EventManager to Get/Send information
  • Use ViewItem:
    • styler
    • contextual menu
    • entity id(s)
    • dataReceiver/dataSender id(s)
  • Some views don't need viewItem:
    • RangeSlider
    • Discovery
    • Dialog

=> because they don't represent a Data/R/S

 

Toolkit Architecture

6. Stylers

  • Allow styling data 
  • Use function to filter data 
  • Generic
  • Example: data can be displayed as
    • text
    • icon
    • line
    • map point
    • ...
  • Are not mandatory

Toolkit Architecture

7. ContextMenu

  • Represent a menu
  • Abstract
  • Can have many representations
    • circular
    • stack
    • ...
  • Contains menuItem
    • name
    • viewId
    • css classes
  • A menuItem makes the link between View and ContextMenu (using EventManager)

Technologies & Frameworks

Core

  • Pure JavaScript
  • CSS 3
  • Only works on recent browsers
  • Framework prototypejs
    • Ajax interface
    • Utility methods
    • Helper methods
    • Extending the DOM
    • Class Object
    • Prototype extends the OO nature of JS
      Class.create() returns a function that acts like a Ruby class, that when called will fire its own initialize() method.

Technologies & Frameworks

External libs

  • Bootstrap                 main application (broadband + layouts)
  • Cesium                      pure JS 3D Globe
  • Jsonix                         convert XML & JSON structures
  • modernizr                 JS helper utilities
  • nouislider                 time slider bar
  • NVD3                         re-usable chart for d3.js
  • OpenLayer                
  • Leaflet
  • Custom tree
  • FFMPEG                     portage of FFMPEG C++ library into JS to decode H264 frames

Technologies & Frameworks

Build

  • NPM                          package manager for JavaScript
  • Gulp                          build system, automating tasks
    • Gulp modules
    • gulp-jshint
    • gulp-minify
    • gulp-sort
    • cleanCSS
    • ...
  • JSDoc                         build documentation
  • Karma                       for tests

Development environment

  • Git
  • Text editor or IDE (Eclipse-js, WebStorm ...)
  • Any local httpd server (apache-http)
  • Local/Remote OSH server
  • Recent version of Chrome/Chromium/Firefox

Build the Toolkit

$ npm install
$ gulp build
$ gulp build-minify
$ gulp clean
clone git@github.com:opensensorhub/osh-js.git

OSH-js by example

Part 1: create the client "Drone Real-Time Video Draping onto Terrain"

OSH-js by example

What we want

OSH-js by example

<swes:offering>
    <sos:ObservationOffering>
        <swes:description>Archive data from Solo Nav DataStore</swes:description>
        <swes:identifier>urn:osh:solo-nav</swes:identifier>
        <swes:name>Solo Nav DataStore</swes:name>
        <swes:procedure>urn:osh:sensor:mavlink:solo:S115A5800419</swes:procedure>
        <swes:observableProperty>http://www.opengis.net/def/property/OGC/0/PlatformOrientation</swes:observableProperty>
        <swes:observableProperty>http://sensorml.com/ont/swe/property/OSH/0/GimbalOrientation</swes:observableProperty>
        <swes:observableProperty>http://www.opengis.net/def/property/OGC/0/PlatformLocation</swes:observableProperty>
        <sos:phenomenonTime>
            <gml:TimePeriod gml:id="T4">
                <gml:beginPosition>2016-08-30T18:58:46.180Z</gml:beginPosition>
                <gml:endPosition>2016-08-30T19:20:17.002Z</gml:endPosition>
            </gml:TimePeriod>
        </sos:phenomenonTime>
        ...
</swes:offering>
<swes:offering>
    <sos:ObservationOffering>
        <swes:description>Archive data from Solo Video DataStore</swes:description>
        <swes:identifier>urn:osh:solo-video</swes:identifier>
        <swes:name>Solo Video DataStore</swes:name>
        <swes:procedure>urn:osh:sensor:rtpcam:solo:S115A5800419</swes:procedure>
        <swes:observableProperty>http://sensorml.com/ont/swe/property/VideoFrame</swes:observableProperty>
        <swes:observableProperty>http://sensorml.com/ont/swe/property/Image</swes:observableProperty>
        <sos:phenomenonTime>
            <gml:TimePeriod gml:id="T5">
                <gml:beginPosition>2016-08-30T18:59:03.009Z</gml:beginPosition>
                <gml:endPosition>2016-08-30T19:20:16.402Z</gml:endPosition>
            </gml:TimePeriod>
        </sos:phenomenonTime>
        ...
</swes:offering>

Get information reading GetCapabilities

  • Looking for solo data

 

Offering

ObservableProperty

Time

OSH-js by example

DataReceiver

  • PlatformLocation => LatLon
  • GimbalOrientation => EulerOrientation
  • Video (H264) => VideoH264

Views

  • CesiumView => 3D map 
  • DialogView => encapsulates video + chart viewers
  • RangeSlider => control time
  • Nvd3CurveChartView => Chart
  • FFMPEGView => Display Video using FFMPEG JS
  • EntityTreeView => display entities
  • StackMenu => for Tree

 

OSH-js by example

<body>
 <div id="container-1" class="mini-view"></div> <!-- Video View -->
 <div id="container-2" class="mini-view"></div> <!-- Chart View -->
 <div id="container-3" class="mini-view"></div> <!-- Cesium View -->
</body>
  • Define simple HTML layout
  • Need a container for each view

OSH-js by example

// DEFINE TIME
var startTime = "2016-08-30T19:00:40Z";
var endTime = "2016-08-30T19:22:00Z";
var replaySpeed = "1";

// LatLon DataReceiver	
var soloGPS = new OSH.DataReceiver.LatLonAlt("Solo GPS", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "urn:osh:solo-nav",
    observedProperty: "http://www.opengis.net/def/property/OGC/0/PlatformLocation",
    startTime: startTime,
    endTime: endTime,
    replaySpeed: replaySpeed,
    syncMasterTime: true
});

// Orientation DataReceiver
var soloAttitude = new OSH.DataReceiver.EulerOrientation("Solo Attitude", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "urn:osh:solo-nav",
    observedProperty: "http://www.opengis.net/def/property/OGC/0/PlatformOrientation",
    startTime: startTime,
    endTime: endTime,
    replaySpeed: replaySpeed,
    syncMasterTime: true
});

Synchronize the data

OSH-js by example

// Gimbal orientation DataReceiver
var soloGimbal = new OSH.DataReceiver.EulerOrientation("Solo Gimbal", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "urn:osh:solo-nav",
    observedProperty: "http://sensorml.com/ont/swe/property/OSH/0/GimbalOrientation",
    startTime: startTime,
    endTime: endTime,
    replaySpeed: replaySpeed,
    syncMasterTime: true
});



// Video H264 DataReceiver
var soloVideo = new OSH.DataReceiver.VideoH264("Solo Video", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "urn:osh:solo-video",
    observedProperty: "http://sensorml.com/ont/swe/property/VideoFrame",
    startTime: startTime,
    endTime: endTime,
    replaySpeed: replaySpeed,
    timeShift: -100,
    syncMasterTime: true
});

OSH-js by example

//--------------------------------------------------------------//
//-------------------------- Entities --------------------------//
//--------------------------------------------------------------//
var soloEntity = {
    id: "solo1",
    name: "3DR Solo",
    dataSources: [soloGPS, soloAttitude, soloGimbal, soloVideo]
};

  • Create an entity to group dataReceiver
  • The name is the one displayed by the EntityTreeViewer

OSH-js by example

Video View

OSH-js by example

//--------------------------------------------------------------//
//--------------------------- Views  ---------------------------//
//--------------------------------------------------------------//

    
var soloVideoView = new OSH.UI.FFMPEGView("container-1", {
    dataSourceId: soloVideo.getId(),
    entityId : soloEntity.id,
    css: "video",
    cssSelected: "video-selected",
    useWorker: true,
    width: 1280,
    height: 720
});

  • Create a Video View
  • FFMPEGView decode and render frame
  • Setup multithreading using WebWorker

OSH-js by example

Chart View

OSH-js by example

//--------------------------------------------------------------//
//--------------------------- Views  ---------------------------//
//--------------------------------------------------------------//

    
var altChartView = new OSH.UI.Nvd3CurveChartView("container-2",
[{
    styler: new OSH.UI.Styler.Curve({
        valuesFunc: {
            dataSourceIds: [soloGPS.getId()],
            handler: function (rec, timeStamp) {
            	if (rec.alt < 1)
            	    rec.alt *= 1e4; // *10^4 due to bug in Toulouse dataset
                return {
                    x: timeStamp,
                    y: rec.alt+mslToWgs84
                };
            }
        }
    })
}],
{
    dataSourceId: soloGPS.getId(),
    yLabel: 'Altitude (m)',
    xLabel: 'Time',
    maxPoints: 100,
    css:"chart-view",
    cssSelected: "video-selected"
});

  • Using NVD3 View to render altitude

We use styler to apply an altitude rectification

Define Graph View properties

OSH-js by example

//--------------------------------------------------------------//
//--------------------------- Views  ---------------------------//
//--------------------------------------------------------------//

// common point marker
var pointMarker = new OSH.UI.Styler.PointMarker({
	label: "3DR Solo",
    locationFunc : {
        dataSourceIds : [soloGPS.getId()],
        handler : function(rec) {
        	if (rec.alt < 1)
        	    rec.alt *= 1e4; // *10^4 due to bug in Toulouse dataset
            return {
                x : rec.lon,
                y : rec.lat,
                z : rec.alt+mslToWgs84-5. // model offset
            };
        }
    },
    orientationFunc : {
        dataSourceIds : [soloAttitude.getId()],
        handler : function(rec) {
            return {
                heading : rec.heading
            };
        }
    },
    icon: "./models/Drone+06B.glb"
});
  • Define PointMarker to style latLon + Alt data

We use styler to add MSL to LatLon data

Get heading from altitude dataReceiver

OSH-js by example

var imageDrapingStyler = new OSH.UI.Styler.ImageDraping({
    platformLocationFunc: {
        dataSourceIds: [soloGPS.getId()],
        handler: function(rec) {
            if (rec.alt < 1)
                rec.alt *= 1e4; // *10^4 due to bug in Toulouse dataset
            return {
                x: rec.lon,
                y: rec.lat,
                z: rec.alt + mslToWgs84
            };
        }
    },
    platformOrientationFunc: {
        dataSourceIds: [soloAttitude.getId()],
        handler: function(rec) {
            return {
                heading: rec.heading,
                pitch: 0, //rec.pitch,
                roll: 0, //rec.roll
            };
        }
    },
    gimbalOrientationFunc: {
        dataSourceIds: [soloGimbal.getId()],
        handler: function(rec) {
            return {
                heading: rec.heading,
                pitch: -92, //rec.pitch,
                roll: 0, //rec.roll
            };
        }
    },
    /*GoPro Mike*/
    cameraModel: {
        camProj: new Cesium.Matrix3(747.963 / 1280., 0.0, 650.66 / 1280.,
            0.0, 769.576 / 738., 373.206 / 738.,
            0.0, 0.0, 1.0),
        camDistR: new Cesium.Cartesian3(-2.644e-01, 8.4e-02, 0.0),
        camDistT: new Cesium.Cartesian2(-8.688e-04, 6.123e-04)
    },
    imageSrc: $$('#' + soloVideoView.getId() + ' canvas')[0]
})
  • Define ImageDrapingMarker

Location

Camera setup

EulerOrientation

GimbalOrientation

OSH-js by example

Cesium View

OSH-js by example

 var mapView = new OSH.UI.CesiumView("container-3", [
{
     name: "3DR Solo",
     entityId: soloEntity.id,
     styler: pointMarkerStyler
 }, {
     name: "Geolocated Imagery",
     entityId: soloEntity.id,
     styler: imageDrapingStyler
 }]);
  • Create view items with
    • name
    • entity Id
    • Styler

ViewItem 1

ViewItem 2

OSH-js by example

var dataSourceController = new OSH.DataReceiver.DataReceiverController({
        replayFactor: 1.0
});

dataSourceController.addEntity(soloEntity);
dataSourceController.connectAll();
// or fire an event
//OSH.EventManager.fire(OSH.EventManager.EVENT.CONNECT_DATASOURCE,
// {dataSourcesId:[videoDataSource.id]});
  • Finally create the dataReceiverController to manage dataR/S

OSH-js by example

Add extra stuff

OSH-js by example

  • Decorate container using floating dialog windows
  • Add contextual menu
  • Add entity tree viewer
  • Add range slider

OSH-js by example

Dialog window

OSH-js by example

  • Dialog window can decorate any div
  • Extra functionnalities to get a nice render:
    • dock
    • drag 
    • close
    • swap
    • show/hide
    • connect/disconnect any DataReceiver

OSH-js by example

  • Change the main HTML layout
<div id="main-container" class="main-view"></div>
<div id="dialog-main-container" class="video-main-container"></div>

OSH-js by example

  • Decorate FFMPEG View
// video view    
var soloVideoDialog = new OSH.UI.DialogView("dialog-main-container", {
    draggable: false,
    css: "video-dialog",
    name: "UAV Video",
    show: true,
    dockable: true,
    closeable: true,
    canDisconnect : false,
    swapId: "main-container"
});

var soloVideoView = new OSH.UI.FFMPEGView(soloVideoDialog.popContentDiv.id, {
    dataSourceId: soloVideo.getId(),
    entityId : soloEntity.id,
    css: "video",
    cssSelected: "video-selected",
    useWorker: true,
    width: 1280,
    height: 720
});

plug the dialog to a main container

set the dialog div id instead of container id

OSH-js by example

  • Decorate Other Views
// chart view
var altChartDialog = new OSH.UI.DialogView("dialog-main-container", {
    draggable: false,
    css: "dialog",
    name: "Solo Altitude",
    show: false,
    dockable: true,
    closeable: true,
    canDisconnect : false,
    swapId: "main-container"
});

var altChartView = new OSH.UI.Nvd3CurveChartView(altChartDialog.popContentDiv.id, ...);

// Cesium view is plugged as background 
var mapView = new OSH.UI.CesiumView("main-container",...)

plug the dialog to a main container

Cesium is set as background

OSH-js by example

Contextual Menu

OSH-js by example

  • Menu can be used in any View
  • Receive and Send events using EventManager
  • Contain MenuItem with
    • name
    • viewId
    • css
    • action
  • Can be grouped 
    • views use groupId

OSH-js by example

  • Define menu id used in the different views

 

  • Add menuId to Cesium viewItem
// menu ids
var soloTreeMenuId = "solo-tree-menu";
var soloMarkerMenuId = "solo-marker-menu";
var menuGroupId = "allmenus";
// cesium map view
var mapView = new OSH.UI.CesiumView("main-container", [{
  name: "3DR Solo",
  entityId: soloEntity.id,
  styler: pointMarker,
  contextMenuId: soloMarkerMenuId
}
[...]

OSH-js by example

  • Define menu items
var menuItems = [{
    name: "Show Video",
    viewId: soloVideoDialog.getId(),
    css: "fa fa-video-camera",
    action: "show"
}, {
    name: "Show Altitude Chart",
    viewId: altChartDialog.getId(),
    css: "fa fa-bar-chart",
    action: "show"
}, {
    name: "TakeOff",
    viewId: "",
    css: "fa fa-upload",
    action: "uav:takeoff"
}, {
    name: "Land",
    viewId: "",
    css: "fa fa-download",
    action: "uav:land"
}];

var markerMenu = new OSH.UI.ContextMenu.CircularMenu({
    id: soloMarkerMenuId,
    groupId: menuGroupId,
    items: menuItems
});

Can also use EventManager:

OSH.EventManager.EVENT.SHOW_VIEW

OSH-js by example

EntityTreeViewer

OSH-js by example

  • Add EntityTreeView
  • Show the whole list of entities and DataReceivers
  • Has menu to perform actions
  • inherit from OSH.UI.View

OSH-js by example

  • Can be plugged into Dialog
 // tree view
var entityTreeDialog = new OSH.UI.DialogView(document.body.id, {
    css: "tree-dialog",
    name: "Entities",
    show: true,
    draggable: true,
    dockable: false,
    closeable: true
});

var entityTreeView = new OSH.UI.EntityTreeView(entityTreeDialog.popContentDiv.id,
    [{
        entity : soloEntity,
        path: "Sensors/Solo",
        treeIcon : "images/drone.png",
        contextMenuId: soloTreeMenuId
    }],
    {
        css: "tree-container"
    }
);    

OSH-js by example

Range Slider

OSH-js by example

  • Finally add a RangeSlider to change time
  • Send events to change dynamically DataReceiver request
  • Use EventManager to communicate
  • Support playback & Real-time (data cannot be changed in that mode)

OSH-js by example

  • Update HTML Layout & CSS

 

  • Since it uses EventManager to communicate, we have only to instantiate a new Object

 

 

  • For real-time we would set:
<div class="rangeSlider-container">
    <div id="rangeSlider" class="rangeSlider"></div>
</div>
 var rangeSlider = new OSH.UI.RangeSlider("rangeSlider",{
    startTime: "2015-02-16T07:58:00Z",
    endTime: "2015-02-16T08:09:00Z",
    refreshRate:1
});
 var rangeSlider = new OSH.UI.RangeSlider("rangeSlider");

OSH-js by example

Part 2: make your own DataReceiver

OSH-js by example

Get info from GetCapabilities

  • offeringID: fake-offering
  • startTime: 2016-08-30T19:00:40Z
  • endTime: 2016-08-30T19:22:00Z
  • observedProperty: http://www.opengis.net/def/property/OGC/0/PlatformLocation

OSH-js by example

Create a new DataReceiver

  • We suppose the server is sending "timeStamp,fakeLat,fakeLon,fakeAlt"
OSH.DataReceiver.MyFakeLatLonAlt = Class.create(OSH.DataReceiver.DataSource,{

    parseTimeStamp: function($super,data){
        // do something
        // this part extract the timestamp from data
        // the data could be text, binary etc..
    },
    
    parseData: function($super,data){
        // do something
        // this part extract only the data without timestamp
    }
});

OSH-js by example

Extract the timeStamp

  • Every data sent by the server includes a timestamp, the client has to extracted it
  • The server is sending "1977-04-22T06:00:00Z,2.464,5.278,30.0" as a String
parseTimeStamp: function($super,data){
    // get record from websocket
    var record = String.fromCharCode.apply(null, new Uint8Array(data));

    // split the record using "," as separator
    var tokens = record.trim().split(",");

    // return the timeStamp using Data JS Object and getTime (milliseconds)
    return new Date(tokens[0]).getTime();
}

OSH-js by example

Extract the data

  • The server is sending "1977-04-22T06:00:00Z,2.464,5.278,30.0" as a String
  • Extract 1977-04-22T06:00:00Z,2.464,5.278,30.0 and put together in a JS object
parseData: function($super,data){
    // get data from WebSocket
    var record = String.fromCharCode.apply(null, new Uint8Array(data));
    
    // split String using "," as separator
    var tokens = record.trim().split(",");

    // get lat, lon, alt
    var lat = parseFloat(tokens[1]);
    var lon = parseFloat(tokens[2]);
    var alt = parseFloat(tokens[3]);
    
    // return JS object
    return {
      lat : lat,
      lon : lon,
      alt : alt
    };
} 

OSH-js by example

Instantiate your new DataReceiver

var myFakeLatLonAlt = new OSH.DataReceiver.MyFakeLatLonAlt("Fake GPS", {
    protocol : "ws",
    service: "SOS",
    endpointUrl: hostname + "/sensorhub/sos",
    offeringID: "fake-offering",
    observedProperty: "http://www.opengis.net/def/property/OGC/0/PlatformLocation",
    startTime: 2016-08-30T19:00:40Z,
    endTime: 2016-08-30T19:22:00Z,
    replaySpeed: 1,
    syncMasterTime: true
});

OSH-js by example

Part 3: Use the Discovery View

OSH-js by example

  • Select service
  • Select offering
  • Select observableProperty
  • Select Time (start/end)
  • Sync master time or not
  • Attach to an existing entity
  • The kind of view

OSH-js by example

OSH-js by example

OSH-js by example

var discoveryView = new OSH.UI.DiscoveryView("",{
    services: ["http://localhost:8181/","http://sensiasoft.net:8181/"],
    css: "discovery-view",
    dataReceiverController:dataProviderController,
    swapId: "center-container",
    entities: [androidEntity],
    views: [{
        name: 'Leaflet 2D Map',
        viewId: leafletMainView.id,
        type : OSH.UI.DiscoveryView.Type.MARKER_GPS
    }, {
        name: 'Cesium 3D Globe',
        viewId: cesiumMainMapView.id,
        type : OSH.UI.DiscoveryView.Type.MARKER_GPS
    },{
        name: 'Video dialog(H264)',
        type : OSH.UI.DiscoveryView.Type.DIALOG_VIDEO_H264
    },{
        name: 'Video dialog(MJPEG)',
        type : OSH.UI.DiscoveryView.Type.DIALOG_VIDEO_MJPEG
    },{
        name: 'Chart dialog',
        type : OSH.UI.DiscoveryView.Type.DIALOG_CHART
    }]
});

OSH-js by example

Part 4: create the client FieldSensor

OSH-js by example

Get information reading GetCapabilities

  • Looking for solo, geocam, virb & nexus5 data

 

OSH-js by example

What we want

Demo 2

OSH Tutorial - Part 8: Javascript Toolkit

By Alex Robin

OSH Tutorial - Part 8: Javascript Toolkit

Build web based clients with the OSH Javascript Toolkit

  • 204

More from Alex Robin