IoT development with Node-Red

L-mobile I Digitalisierte Softwarelösungen

Imre Szecsődi

Friday, 30.09.2019

Table of content

L-mobile I Digitalisierte Softwarelösungen

What is Node.js?
What is Typescript?
What is Node-Red?
Deployment
- Native
- Docker
Influxdb
Grafana
Live coding example

Node.js

L-mobile I Digitalisierte Softwarelösungen

Open-source, cross-platform JavaScript run-time environment
Runs outside of a web browser
Server-side programming
Based on the V8 JavaScript Engine
- Developed by the Chromium project
Unifies web development around one language, JS
Centralised package manager, NPM
Two version LTS and Current
Companies using Node.js IBM, LinkedIn, Microsoft, Netflix, PayPal, SAP etc.

PayPal study

L-mobile I Digitalisierte Softwarelösungen

In 2011 PayPal had a risk averse strategy regarding technologies
Webpage written in Java + Spring + JSP (Very similar to ASP.NET + Razor pages)
Complicated, hard to expand

PayPal study

L-mobile I Digitalisierte Softwarelösungen

Rewritten in Node.js
Result:
- Built twice as fast with fewer people
- 33% fewer lines of code
- 40% fewer files
- 2x request/sec
- 35% faster response time

Node.js

L-mobile I Digitalisierte Softwarelösungen

Single threaded
Asynchronous event loop
Scheduler for async context switch
Garbage collected
Executes standard ECMAScript
Extended functionality from the V8 Engine, like standard socket and filesystem
"Easily" extendable with native libraries C/C++, etc.
Large open-source community

Node.js

L-mobile I Digitalisierte Softwarelösungen

TypeScript

L-mobile I Digitalisierte Softwarelösungen

JavaScript that scales.
Typed superset of JavaScript, that compiles to plain JavaScript
Open source
Developed by Microsoft
JavaScript is valid Typescript
Typescript must use the transpiler to create valid JavaScript
Node.js can't execute Typescript directly!

TypeScript

L-mobile I Digitalisierte Softwarelösungen

Compatible with packages in NPM
Since most libraries are written in JavaScript, there is a special repository for types, @types
Not all libraries have @types, but the most used ones do
Native Typescript libraries are preferable
There is option to use JavaScript with type checking, but not using Typescript
IDEs like types

Node-Red

L-mobile I Digitalisierte Softwarelösungen

Flow-based programming for the Internet of Things
Programming tool for wiring together hardware devices, APIs and online services
Browser-based editor
Built on Node.js
Uses nodes to make the program flow
Node-Red node made of:
- Logic, written in JavaScript
- HTML component
Uses the older Bootstrap 3 and jQuery

Node-Red

L-mobile I Digitalisierte Softwarelösungen

Deployment

L-mobile I Digitalisierte Softwarelösungen

Requirements for Node.js
- Use version 10 LTS version. . .
- or download the latest anyway, because version 12 will be LTS by end of October
- Windows and MacOS just use the installer
- Linux, find a precompiled version
- or download the source
- Requirements of compiling:
  - GCC toolchain
  - Python 2.7
  - Make

Docker

L-mobile I Digitalisierte Softwarelösungen

Docker containers supported
Get the appropriate version of Docker
Create the common network adapter
Build and deploy the containers
Most containers are based of Alpine linux, but there are some Ubuntu also

InfluxDB

L-mobile I Digitalisierte Softwarelösungen

Database optimal for time based data storage
Data is indexed by time
Queries are defined on time intervals
Data example:
- Stock market
- Analytics
- IoT devices

InfluxDB

L-mobile I Digitalisierte Softwarelösungen

Why not use a relational database?
Just mark one column where you put the time data and its done
There are several reasons:
- Scalability - several million data points per day is too much for a normal table
- Down-sampling - there are a lots of high precision data which has to be stored, influx aggregates data as they get older
- Interval based storage - When the interval fulfils for every addition of a point there is a deletion which is very cost-intensive on a regular database

InfluxDB

L-mobile I Digitalisierte Softwarelösungen

Fast query time, since the index is on the time
Simple window query
Data stored in Shards
Shards can be dropped easily, this helps to phase out old data
Data inserted via HTTP Request
Data format is the "Line protocol", which does not contain scheme, NoSQL style
There is a retention policy, when will the data dropped from the database automatically

InfluxDB

L-mobile I Digitalisierte Softwarelösungen

weather,location=us-midwest temperature=82 1465839830100400200
  |    -------------------- --------------  |
  |             |             |             |
  |             |             |             |
+-----------+--------+-+---------+-+---------+
|measurement|,tag_set| |field_set| |timestamp|
+-----------+--------+-+---------+-+---------+

Measurement - the name of the data, String
Tags - key value pairs, String - String
Fields - key value pairs, observed data
- Types: Int64, Float64, Bool, String
Timestamp - time in nanoseconds

Integration

L-mobile I Digitalisierte Softwarelösungen

Grafana

L-mobile I Digitalisierte Softwarelösungen

Open-source
Feature-rich metrics dashboard
Work well with InfluxDB

Grafana

L-mobile I Digitalisierte Softwarelösungen

Live coding example

L-mobile I Digitalisierte Softwarelösungen

Examples
- OpenWeahterMap node
  - How to build an example node
  - How to get data for external source
- SeemsLegitDatasource
  - How to get data from a very simple data source
  - How to use the DataSet node
  - How to persist data into InfluxDB
  - How to plot data from InfluxDB in Grafana

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

Open API just need to register
Get an API key
Example Query string

api.openweathermap.org/data/2.5/weather?zip=94040,us

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

{
  "coord": {"lon": -122.08,"lat": 37.39},
  "weather": [
    {
      "id": 800,
      "main": "Clear",
      "description": "clear sky",
      "icon": "01d"
    }
  ],
  "base": "stations",
  "main": {
    "temp": 296.71,
    "pressure": 1013,
    "humidity": 53,
    "temp_min": 294.82,
    "temp_max": 298.71
  },
  "visibility": 16093,
  "wind": {
    "speed": 1.5,
    "deg": 350
  },
  "clouds": {
    "all": 1
  },
  "dt": 1560350645,
  "sys": {
    "type": 1,
    "id": 5122,
    "message": 0.0139,
    "country": "US",
    "sunrise": 1560343627,
    "sunset": 1560396563
  },
  "timezone": -25200,
  "id": 420006353,
  "name": "Mountain View",
  "cod": 200
}

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

<script type="text/javascript">
    RED.nodes.registerType('weather-node', {
        category: 'Weather',
        color: '#cfcbce',
        defaults: {
            name: {value: ""},
            url: {value: "api.openweathermap.org/data/2.5/weather"},
            key: {value: "", "required": true},
            "method-of-gathering-data": {value: "city"},
            "country-code-check": {value: ""},
            "city-name": {value: ""},
            "country-code": {value: ""},
            lon: {value: 0, validate:RED.validators.number()},
            lat: {value: 0, validate:RED.validators.number()},
            "city-id": {value: 0, validate:RED.validators.number()}
        },
        inputs: 1,
        outputs: 1,
        icon: "file.png",
        label: function () {
            return this.name || "OpenWeatherMap Connector";
        },

Required fields for the HMTL

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

oneditprepare: function () {
            $("#node-input-method-of-gathering-data").change(function () {
                if ($(this).val() === "id") {
                    $("#city-name-div").hide();
                    $("#coord-div").hide();
                    $("#city-id-div").show();
                } else if ($(this).val() === "coord") {
                    $("#city-name-div").hide();
                    $("#coord-div").show();
                    $("#city-id-div").hide();
                } else {
                    $("#city-name-div").show();
                    $("#coord-div").hide();
                    $("#city-id-div").hide();
                }
            });
            $("#node-input-country-code-check").change(function () {
                if ($(this).is(":checked")) {
                    $("#node-input-country-code").prop("disabled", false);
                } else {
                    $("#node-input-country-code").prop("disabled", true);
                }
            });

        }
    });
</script>

HTML logic to show/hide components

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

<script type="text/x-red" data-template-name="weather-node">
    <div class="form-row">
        <label for="node-input-name"><i class="fa fa-tag"></i> Name</label>
        <input type="text" id="node-input-name">
    </div>

    <div class="form-row">
        <label for="node-input-url"><i class="fa fa-globe"></i> API URL</label>
        <input type="text" name="url" id="node-input-url" placeholder="api.openweathermap.org/data/2.5/weather">
    </div>

    <div class="form-row">
        <label for="node-input-key"><i class="fa fa-key"></i> API Key</label>
        <input type="text" id="node-input-key">
    </div>

    <div class="form-row">
        <label for="node-input-method-of-gathering-data"><i class="fa fa-download"></i> Method of gathering data</label>
        <select name="method-of-gathering-data" id="node-input-method-of-gathering-data" style="display: inline-block; width: auto; vertical-align: top;">
            <option value="city" selected="selected" >City Name</option>
            <option value="coord">Coordinate</option>
            <option value="id">City ID</option>
        </select>

        <div id="city-name-div">
            <div class="form-row">
                <label for="node-input-city-name">City Name</label>
                <input type="text" name="city-name" id="node-input-city-name">
            </div>

            <div class="form-row">
                <input type="checkbox" name="country-code-check" id="node-input-country-code-check"  style="display: inline-block; width: auto; vertical-align: top;">
                <label for="node-input-country-code-check" style="width: 70%">Use country code</label>
            </div>

            <div class="form-row">
                <label for="node-input-country-code">Country code</label>
                <input type="text" name="country-code" id="node-input-country-code" disabled>
            </div>
        </div>

        <div id="coord-div" hidden >
            <div class="form-row">
                <label for="node-input-lon">Longitude</label>
                <input type="text" name="node-input-lon" id="node-input-lon">
            </div>
            <div class="form-row">
                <label for="node-input-lat">Latitude</label>
                <input type="text" name="lat" id="node-input-lat">
            </div>
        </div>

        <div id="city-id-div" hidden class="form-row">
            <label for="node-input-city-id">City ID</label>
            <input type="text" name="city-id" id="node-input-city-id">
        </div>

    </div>

</script>

HTML Design

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

<script type="text/x-red" data-help-name="weather-node">
    <p>Open Weather Map connector</p>
</script>

HTML - Node-red help

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

import {Red} from "node-red";
import * as request from "request";

async function getWeatherData(config: object) {
    return new Promise((resolve, reject) => {
        let uri = `http://${config["url"]}?`;
        if(config["method-of-gathering-data"] === "city") {
            uri += `q=${config['city-name']}`;
            if(config['country-code-check']) {
                uri += `,${config['country-code']}`
            }
        } else if (config["method-of-gathering-data"] === "coord") {
            uri += `lat=${config["lat"]}&lon=${config['lon']}`;
        } else {
            uri += `id=${config["city-id"]}`;
        }
        uri += `&APPID=${config['key']}`;
        request(uri, {method: "get"}, (error, response, body) => {
            if(error) {
                reject(error);
                return;
            }
            let data = { response: response, body: body};
            resolve(data);
        });
    });
}

TS Logic to get the data

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

function redFunction(RED: Red) {
     function owmGetData(config) {
        RED.nodes.createNode(this, config);
        let node = this;
        node.on("input", async function (msg) {
            msg.payload = await getWeatherData(config);
            node.send(msg);
        })
    }
    RED.nodes.registerType("weather-node", owmGetData);
}

export = redFunction;

Seems Legit Data

L-mobile I Digitalisierte Softwarelösungen

{
    "zipf": 2,
    "binomial": 15,
    "exponential": 4,
    "poission": 2
}

Created for demonstration purposes
Response example
- Binomial distribution
- Exponential distribution
- Poisson distribution
- Zipf's law ~ Logarithmic

Seems Legit Data

L-mobile I Digitalisierte Softwarelösungen

Dataset

Code	Name	Type
binomial	binomial	float
poission	poission	float
zipf	zipf	float
exponential	exponential	float

Seems Legit Data

L-mobile I Digitalisierte Softwarelösungen

Building the flow

Seems Legit Data

L-mobile I Digitalisierte Softwarelösungen

Check Grafana for results

Any questions?

L-mobile I Digitalisierte Softwarelösungen