IoT development with Node-Red

L-mobile I Digitalisierte Softwarelösungen

Imre Szecsődi

Friday, 30.09.2019

Table of content

L-mobile I Digitalisierte Softwarelösungen

  • What is Node.js?
  • What is Typescript?
  • What is Node-Red?
  • Deployment
    • Native
    • Docker
  • Influxdb
  • Grafana
  • Live coding example

Node.js

L-mobile I Digitalisierte Softwarelösungen

  • Open-source, cross-platform JavaScript run-time environment
  • Runs outside of a web browser
  • Server-side programming
  • Based on the V8 JavaScript Engine
    • Developed by the Chromium project
  • Unifies web development around one language, JS
  • Centralised package manager, NPM
  • Two version LTS and Current
  • Companies using Node.js IBM, LinkedIn, Microsoft, Netflix, PayPal, SAP etc.

PayPal study

L-mobile I Digitalisierte Softwarelösungen

  • In 2011 PayPal had a risk averse strategy regarding technologies
  • Webpage written in Java + Spring + JSP (Very similar to ASP.NET + Razor pages)
  • Complicated, hard to expand

PayPal study

L-mobile I Digitalisierte Softwarelösungen

  • Rewritten in Node.js
  • Result:
    • Built twice as fast with fewer people
    • 33% fewer lines of code
    • 40% fewer files
    • 2x request/sec
    • 35% faster response time

Node.js

L-mobile I Digitalisierte Softwarelösungen

  • Single threaded
  • Asynchronous event loop
  • Scheduler for async context switch
  • Garbage collected
  • Executes standard ECMAScript
  • Extended functionality from the V8 Engine, like standard socket and filesystem
  • "Easily" extendable with native libraries C/C++, etc.
  • Large open-source community

Node.js

L-mobile I Digitalisierte Softwarelösungen

TypeScript

L-mobile I Digitalisierte Softwarelösungen

  • JavaScript that scales.
  • Typed superset of JavaScript, that compiles to plain JavaScript
  • Open source
  • Developed by Microsoft
  • JavaScript is valid Typescript
  • Typescript must use the transpiler to create valid JavaScript
  • Node.js can't execute Typescript directly!

TypeScript

L-mobile I Digitalisierte Softwarelösungen

  • Compatible with packages in NPM
  • Since most libraries are written in JavaScript, there is a special repository for types, @types
  • Not all libraries have @types, but the most used ones do
  • Native Typescript libraries are preferable
  • There is option to use JavaScript with type checking, but not using Typescript
  • IDEs like types

Node-Red

L-mobile I Digitalisierte Softwarelösungen

  • Flow-based programming for the Internet of Things
  • Programming tool for wiring together hardware devices, APIs and online services
  • Browser-based editor
  • Built on Node.js
  • Uses nodes to make the program flow
  • Node-Red node made of:
    • Logic, written in JavaScript
    • HTML component
  • Uses the older Bootstrap 3 and jQuery

Node-Red

L-mobile I Digitalisierte Softwarelösungen

Deployment

L-mobile I Digitalisierte Softwarelösungen

  • Requirements for Node.js
    • Use version 10 LTS version. . .
    • or download the latest anyway, because version 12 will be LTS by end of October
    • Windows and MacOS just use the installer
    • Linux, find a precompiled version
    • or download the source
    • Requirements of compiling:
      • GCC toolchain
      • Python 2.7
      • Make

Docker

L-mobile I Digitalisierte Softwarelösungen

  • Docker containers supported
  • Get the appropriate version of Docker
  • Create the common network adapter
  • Build and deploy the containers
  • Most containers are based of Alpine linux, but there are some Ubuntu also

InfluxDB

L-mobile I Digitalisierte Softwarelösungen

  • Database optimal for time based data storage
  • Data is indexed by time
  • Queries are defined on time intervals
  • Data example:
    • Stock market
    • Analytics
    • IoT devices

InfluxDB

L-mobile I Digitalisierte Softwarelösungen

  • Why not use a relational database?
  • Just mark one column where you put the time data and its done
  • There are several reasons:
    • Scalability - several million data points per day is too much for a normal table
    • Down-sampling - there are a lots of high precision data which has to be stored, influx aggregates data as they get older
    • Interval based storage - When the interval fulfils for every addition of a point there is a deletion which is very cost-intensive on a regular database

InfluxDB

L-mobile I Digitalisierte Softwarelösungen

  • Fast query time, since the index is on the time
  • Simple window query
  • Data stored in Shards
  • Shards can be dropped easily, this helps to phase out old data
  • Data inserted via HTTP Request
  • Data format is the "Line protocol", which does not contain scheme, NoSQL style
  • There is a retention policy, when will the data dropped from the database automatically

InfluxDB

L-mobile I Digitalisierte Softwarelösungen

weather,location=us-midwest temperature=82 1465839830100400200
  |    -------------------- --------------  |
  |             |             |             |
  |             |             |             |
+-----------+--------+-+---------+-+---------+
|measurement|,tag_set| |field_set| |timestamp|
+-----------+--------+-+---------+-+---------+
  • Measurement - the name of the data, String
  • Tags - key value pairs, String - String
  • Fields - key value pairs, observed data
    • Types: Int64, Float64, Bool, String
  • Timestamp - time in nanoseconds

Integration

L-mobile I Digitalisierte Softwarelösungen

Grafana

L-mobile I Digitalisierte Softwarelösungen

  • Open-source
  • Feature-rich metrics dashboard
  • Work well with InfluxDB

Grafana

L-mobile I Digitalisierte Softwarelösungen

Live coding example

L-mobile I Digitalisierte Softwarelösungen

  • Examples
    • OpenWeahterMap node
      • How to build an example node
      • How to get data for external source
    • SeemsLegitDatasource
      • How to get data from a very simple data source
      • How to use the DataSet node
      • How to persist data into InfluxDB
      • How to plot data from InfluxDB in Grafana

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

  • Open API just need to register
  • Get an API key
  • Example Query string
api.openweathermap.org/data/2.5/weather?zip=94040,us

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

{
  "coord": {"lon": -122.08,"lat": 37.39},
  "weather": [
    {
      "id": 800,
      "main": "Clear",
      "description": "clear sky",
      "icon": "01d"
    }
  ],
  "base": "stations",
  "main": {
    "temp": 296.71,
    "pressure": 1013,
    "humidity": 53,
    "temp_min": 294.82,
    "temp_max": 298.71
  },
  "visibility": 16093,
  "wind": {
    "speed": 1.5,
    "deg": 350
  },
  "clouds": {
    "all": 1
  },
  "dt": 1560350645,
  "sys": {
    "type": 1,
    "id": 5122,
    "message": 0.0139,
    "country": "US",
    "sunrise": 1560343627,
    "sunset": 1560396563
  },
  "timezone": -25200,
  "id": 420006353,
  "name": "Mountain View",
  "cod": 200
}

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

<script type="text/javascript">
    RED.nodes.registerType('weather-node', {
        category: 'Weather',
        color: '#cfcbce',
        defaults: {
            name: {value: ""},
            url: {value: "api.openweathermap.org/data/2.5/weather"},
            key: {value: "", "required": true},
            "method-of-gathering-data": {value: "city"},
            "country-code-check": {value: ""},
            "city-name": {value: ""},
            "country-code": {value: ""},
            lon: {value: 0, validate:RED.validators.number()},
            lat: {value: 0, validate:RED.validators.number()},
            "city-id": {value: 0, validate:RED.validators.number()}
        },
        inputs: 1,
        outputs: 1,
        icon: "file.png",
        label: function () {
            return this.name || "OpenWeatherMap Connector";
        },
  • Required fields for the HMTL

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

oneditprepare: function () {
            $("#node-input-method-of-gathering-data").change(function () {
                if ($(this).val() === "id") {
                    $("#city-name-div").hide();
                    $("#coord-div").hide();
                    $("#city-id-div").show();
                } else if ($(this).val() === "coord") {
                    $("#city-name-div").hide();
                    $("#coord-div").show();
                    $("#city-id-div").hide();
                } else {
                    $("#city-name-div").show();
                    $("#coord-div").hide();
                    $("#city-id-div").hide();
                }
            });
            $("#node-input-country-code-check").change(function () {
                if ($(this).is(":checked")) {
                    $("#node-input-country-code").prop("disabled", false);
                } else {
                    $("#node-input-country-code").prop("disabled", true);
                }
            });

        }
    });
</script>
  • HTML logic to show/hide components

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

<script type="text/x-red" data-template-name="weather-node">
    <div class="form-row">
        <label for="node-input-name"><i class="fa fa-tag"></i> Name</label>
        <input type="text" id="node-input-name">
    </div>

    <div class="form-row">
        <label for="node-input-url"><i class="fa fa-globe"></i> API URL</label>
        <input type="text" name="url" id="node-input-url" placeholder="api.openweathermap.org/data/2.5/weather">
    </div>

    <div class="form-row">
        <label for="node-input-key"><i class="fa fa-key"></i> API Key</label>
        <input type="text" id="node-input-key">
    </div>

    <div class="form-row">
        <label for="node-input-method-of-gathering-data"><i class="fa fa-download"></i> Method of gathering data</label>
        <select name="method-of-gathering-data" id="node-input-method-of-gathering-data" style="display: inline-block; width: auto; vertical-align: top;">
            <option value="city" selected="selected" >City Name</option>
            <option value="coord">Coordinate</option>
            <option value="id">City ID</option>
        </select>

        <div id="city-name-div">
            <div class="form-row">
                <label for="node-input-city-name">City Name</label>
                <input type="text" name="city-name" id="node-input-city-name">
            </div>

            <div class="form-row">
                <input type="checkbox" name="country-code-check" id="node-input-country-code-check"  style="display: inline-block; width: auto; vertical-align: top;">
                <label for="node-input-country-code-check" style="width: 70%">Use country code</label>
            </div>

            <div class="form-row">
                <label for="node-input-country-code">Country code</label>
                <input type="text" name="country-code" id="node-input-country-code" disabled>
            </div>
        </div>

        <div id="coord-div" hidden >
            <div class="form-row">
                <label for="node-input-lon">Longitude</label>
                <input type="text" name="node-input-lon" id="node-input-lon">
            </div>
            <div class="form-row">
                <label for="node-input-lat">Latitude</label>
                <input type="text" name="lat" id="node-input-lat">
            </div>
        </div>

        <div id="city-id-div" hidden class="form-row">
            <label for="node-input-city-id">City ID</label>
            <input type="text" name="city-id" id="node-input-city-id">
        </div>

    </div>

</script>
  • HTML Design

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

<script type="text/x-red" data-help-name="weather-node">
    <p>Open Weather Map connector</p>
</script>
  • HTML - Node-red help

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

import {Red} from "node-red";
import * as request from "request";

async function getWeatherData(config: object) {
    return new Promise((resolve, reject) => {
        let uri = `http://${config["url"]}?`;
        if(config["method-of-gathering-data"] === "city") {
            uri += `q=${config['city-name']}`;
            if(config['country-code-check']) {
                uri += `,${config['country-code']}`
            }
        } else if (config["method-of-gathering-data"] === "coord") {
            uri += `lat=${config["lat"]}&lon=${config['lon']}`;
        } else {
            uri += `id=${config["city-id"]}`;
        }
        uri += `&APPID=${config['key']}`;
        request(uri, {method: "get"}, (error, response, body) => {
            if(error) {
                reject(error);
                return;
            }
            let data = { response: response, body: body};
            resolve(data);
        });
    });
}
  • TS Logic to get the data

OpenWeahterMap

L-mobile I Digitalisierte Softwarelösungen

function redFunction(RED: Red) {
     function owmGetData(config) {
        RED.nodes.createNode(this, config);
        let node = this;
        node.on("input", async function (msg) {
            msg.payload = await getWeatherData(config);
            node.send(msg);
        })
    }
    RED.nodes.registerType("weather-node", owmGetData);
}

export = redFunction;
  • Register Node in Node-Red

Seems Legit Data

L-mobile I Digitalisierte Softwarelösungen

{
    "zipf": 2,
    "binomial": 15,
    "exponential": 4,
    "poission": 2
}
  • Created for demonstration purposes
  • Response example
    • Binomial distribution
    • Exponential distribution
    • Poisson distribution
    • Zipf's law  ~ Logarithmic

Seems Legit Data

L-mobile I Digitalisierte Softwarelösungen

  • Dataset
Code Name Type Unit
binomial binomial float
poission poission float
zipf zipf float
exponential exponential float

Seems Legit Data

L-mobile I Digitalisierte Softwarelösungen

  • Building the flow

Seems Legit Data

L-mobile I Digitalisierte Softwarelösungen

  • Check Grafana for results

Any questions?

L-mobile I Digitalisierte Softwarelösungen

deck

By Imre Secodi

Made with Slides.com

deck

  • 98

More from Imre Secodi