IoT development with Node-Red
L-mobile I Digitalisierte Softwarelösungen
Imre Szecsődi
Friday, 30.09.2019
Table of content
L-mobile I Digitalisierte Softwarelösungen
- What is Node.js?
- What is Typescript?
- What is Node-Red?
- Deployment
- Native
- Docker
- Influxdb
- Grafana
- Live coding example
Node.js
L-mobile I Digitalisierte Softwarelösungen
- Open-source, cross-platform JavaScript run-time environment
- Runs outside of a web browser
- Server-side programming
- Based on the V8 JavaScript Engine
- Developed by the Chromium project
- Unifies web development around one language, JS
- Centralised package manager, NPM
- Two version LTS and Current
- Companies using Node.js IBM, LinkedIn, Microsoft, Netflix, PayPal, SAP etc.
PayPal study
L-mobile I Digitalisierte Softwarelösungen
- In 2011 PayPal had a risk averse strategy regarding technologies
- Webpage written in Java + Spring + JSP (Very similar to ASP.NET + Razor pages)
- Complicated, hard to expand
PayPal study
L-mobile I Digitalisierte Softwarelösungen
- Rewritten in Node.js
- Result:
- Built twice as fast with fewer people
- 33% fewer lines of code
- 40% fewer files
- 2x request/sec
- 35% faster response time
Node.js
L-mobile I Digitalisierte Softwarelösungen
- Single threaded
- Asynchronous event loop
- Scheduler for async context switch
- Garbage collected
- Executes standard ECMAScript
- Extended functionality from the V8 Engine, like standard socket and filesystem
- "Easily" extendable with native libraries C/C++, etc.
- Large open-source community
Node.js
L-mobile I Digitalisierte Softwarelösungen
TypeScript
L-mobile I Digitalisierte Softwarelösungen
- JavaScript that scales.
- Typed superset of JavaScript, that compiles to plain JavaScript
- Open source
- Developed by Microsoft
- JavaScript is valid Typescript
- Typescript must use the transpiler to create valid JavaScript
- Node.js can't execute Typescript directly!
TypeScript
L-mobile I Digitalisierte Softwarelösungen
- Compatible with packages in NPM
- Since most libraries are written in JavaScript, there is a special repository for types, @types
- Not all libraries have @types, but the most used ones do
- Native Typescript libraries are preferable
- There is option to use JavaScript with type checking, but not using Typescript
- IDEs like types
Node-Red
L-mobile I Digitalisierte Softwarelösungen
- Flow-based programming for the Internet of Things
- Programming tool for wiring together hardware devices, APIs and online services
- Browser-based editor
- Built on Node.js
- Uses nodes to make the program flow
- Node-Red node made of:
- Logic, written in JavaScript
- HTML component
- Uses the older Bootstrap 3 and jQuery
Node-Red
L-mobile I Digitalisierte Softwarelösungen
Deployment
L-mobile I Digitalisierte Softwarelösungen
- Requirements for Node.js
- Use version 10 LTS version. . .
- or download the latest anyway, because version 12 will be LTS by end of October
- Windows and MacOS just use the installer
- Linux, find a precompiled version
- or download the source
- Requirements of compiling:
- GCC toolchain
- Python 2.7
- Make
Docker
L-mobile I Digitalisierte Softwarelösungen
- Docker containers supported
- Get the appropriate version of Docker
- Create the common network adapter
- Build and deploy the containers
- Most containers are based of Alpine linux, but there are some Ubuntu also
InfluxDB
L-mobile I Digitalisierte Softwarelösungen
- Database optimal for time based data storage
- Data is indexed by time
- Queries are defined on time intervals
- Data example:
- Stock market
- Analytics
- IoT devices
InfluxDB
L-mobile I Digitalisierte Softwarelösungen
- Why not use a relational database?
- Just mark one column where you put the time data and its done
- There are several reasons:
- Scalability - several million data points per day is too much for a normal table
- Down-sampling - there are a lots of high precision data which has to be stored, influx aggregates data as they get older
- Interval based storage - When the interval fulfils for every addition of a point there is a deletion which is very cost-intensive on a regular database
InfluxDB
L-mobile I Digitalisierte Softwarelösungen
- Fast query time, since the index is on the time
- Simple window query
- Data stored in Shards
- Shards can be dropped easily, this helps to phase out old data
- Data inserted via HTTP Request
- Data format is the "Line protocol", which does not contain scheme, NoSQL style
- There is a retention policy, when will the data dropped from the database automatically
InfluxDB
L-mobile I Digitalisierte Softwarelösungen
weather,location=us-midwest temperature=82 1465839830100400200
| -------------------- -------------- |
| | | |
| | | |
+-----------+--------+-+---------+-+---------+
|measurement|,tag_set| |field_set| |timestamp|
+-----------+--------+-+---------+-+---------+
- Measurement - the name of the data, String
- Tags - key value pairs, String - String
- Fields - key value pairs, observed data
- Types: Int64, Float64, Bool, String
- Timestamp - time in nanoseconds
Integration
L-mobile I Digitalisierte Softwarelösungen
Grafana
L-mobile I Digitalisierte Softwarelösungen
- Open-source
- Feature-rich metrics dashboard
- Work well with InfluxDB
Grafana
L-mobile I Digitalisierte Softwarelösungen
Live coding example
L-mobile I Digitalisierte Softwarelösungen
- Examples
- OpenWeahterMap node
- How to build an example node
- How to get data for external source
- SeemsLegitDatasource
- How to get data from a very simple data source
- How to use the DataSet node
- How to persist data into InfluxDB
- How to plot data from InfluxDB in Grafana
- OpenWeahterMap node
OpenWeahterMap
L-mobile I Digitalisierte Softwarelösungen
- Open API just need to register
- Get an API key
- Example Query string
api.openweathermap.org/data/2.5/weather?zip=94040,us
OpenWeahterMap
L-mobile I Digitalisierte Softwarelösungen
{
"coord": {"lon": -122.08,"lat": 37.39},
"weather": [
{
"id": 800,
"main": "Clear",
"description": "clear sky",
"icon": "01d"
}
],
"base": "stations",
"main": {
"temp": 296.71,
"pressure": 1013,
"humidity": 53,
"temp_min": 294.82,
"temp_max": 298.71
},
"visibility": 16093,
"wind": {
"speed": 1.5,
"deg": 350
},
"clouds": {
"all": 1
},
"dt": 1560350645,
"sys": {
"type": 1,
"id": 5122,
"message": 0.0139,
"country": "US",
"sunrise": 1560343627,
"sunset": 1560396563
},
"timezone": -25200,
"id": 420006353,
"name": "Mountain View",
"cod": 200
}
OpenWeahterMap
L-mobile I Digitalisierte Softwarelösungen
<script type="text/javascript">
RED.nodes.registerType('weather-node', {
category: 'Weather',
color: '#cfcbce',
defaults: {
name: {value: ""},
url: {value: "api.openweathermap.org/data/2.5/weather"},
key: {value: "", "required": true},
"method-of-gathering-data": {value: "city"},
"country-code-check": {value: ""},
"city-name": {value: ""},
"country-code": {value: ""},
lon: {value: 0, validate:RED.validators.number()},
lat: {value: 0, validate:RED.validators.number()},
"city-id": {value: 0, validate:RED.validators.number()}
},
inputs: 1,
outputs: 1,
icon: "file.png",
label: function () {
return this.name || "OpenWeatherMap Connector";
},
- Required fields for the HMTL
OpenWeahterMap
L-mobile I Digitalisierte Softwarelösungen
oneditprepare: function () {
$("#node-input-method-of-gathering-data").change(function () {
if ($(this).val() === "id") {
$("#city-name-div").hide();
$("#coord-div").hide();
$("#city-id-div").show();
} else if ($(this).val() === "coord") {
$("#city-name-div").hide();
$("#coord-div").show();
$("#city-id-div").hide();
} else {
$("#city-name-div").show();
$("#coord-div").hide();
$("#city-id-div").hide();
}
});
$("#node-input-country-code-check").change(function () {
if ($(this).is(":checked")) {
$("#node-input-country-code").prop("disabled", false);
} else {
$("#node-input-country-code").prop("disabled", true);
}
});
}
});
</script>
- HTML logic to show/hide components
OpenWeahterMap
L-mobile I Digitalisierte Softwarelösungen
<script type="text/x-red" data-template-name="weather-node">
<div class="form-row">
<label for="node-input-name"><i class="fa fa-tag"></i> Name</label>
<input type="text" id="node-input-name">
</div>
<div class="form-row">
<label for="node-input-url"><i class="fa fa-globe"></i> API URL</label>
<input type="text" name="url" id="node-input-url" placeholder="api.openweathermap.org/data/2.5/weather">
</div>
<div class="form-row">
<label for="node-input-key"><i class="fa fa-key"></i> API Key</label>
<input type="text" id="node-input-key">
</div>
<div class="form-row">
<label for="node-input-method-of-gathering-data"><i class="fa fa-download"></i> Method of gathering data</label>
<select name="method-of-gathering-data" id="node-input-method-of-gathering-data" style="display: inline-block; width: auto; vertical-align: top;">
<option value="city" selected="selected" >City Name</option>
<option value="coord">Coordinate</option>
<option value="id">City ID</option>
</select>
<div id="city-name-div">
<div class="form-row">
<label for="node-input-city-name">City Name</label>
<input type="text" name="city-name" id="node-input-city-name">
</div>
<div class="form-row">
<input type="checkbox" name="country-code-check" id="node-input-country-code-check" style="display: inline-block; width: auto; vertical-align: top;">
<label for="node-input-country-code-check" style="width: 70%">Use country code</label>
</div>
<div class="form-row">
<label for="node-input-country-code">Country code</label>
<input type="text" name="country-code" id="node-input-country-code" disabled>
</div>
</div>
<div id="coord-div" hidden >
<div class="form-row">
<label for="node-input-lon">Longitude</label>
<input type="text" name="node-input-lon" id="node-input-lon">
</div>
<div class="form-row">
<label for="node-input-lat">Latitude</label>
<input type="text" name="lat" id="node-input-lat">
</div>
</div>
<div id="city-id-div" hidden class="form-row">
<label for="node-input-city-id">City ID</label>
<input type="text" name="city-id" id="node-input-city-id">
</div>
</div>
</script>
- HTML Design
OpenWeahterMap
L-mobile I Digitalisierte Softwarelösungen
<script type="text/x-red" data-help-name="weather-node">
<p>Open Weather Map connector</p>
</script>
- HTML - Node-red help
OpenWeahterMap
L-mobile I Digitalisierte Softwarelösungen
import {Red} from "node-red";
import * as request from "request";
async function getWeatherData(config: object) {
return new Promise((resolve, reject) => {
let uri = `http://${config["url"]}?`;
if(config["method-of-gathering-data"] === "city") {
uri += `q=${config['city-name']}`;
if(config['country-code-check']) {
uri += `,${config['country-code']}`
}
} else if (config["method-of-gathering-data"] === "coord") {
uri += `lat=${config["lat"]}&lon=${config['lon']}`;
} else {
uri += `id=${config["city-id"]}`;
}
uri += `&APPID=${config['key']}`;
request(uri, {method: "get"}, (error, response, body) => {
if(error) {
reject(error);
return;
}
let data = { response: response, body: body};
resolve(data);
});
});
}
- TS Logic to get the data
OpenWeahterMap
L-mobile I Digitalisierte Softwarelösungen
function redFunction(RED: Red) {
function owmGetData(config) {
RED.nodes.createNode(this, config);
let node = this;
node.on("input", async function (msg) {
msg.payload = await getWeatherData(config);
node.send(msg);
})
}
RED.nodes.registerType("weather-node", owmGetData);
}
export = redFunction;
- Register Node in Node-Red
Seems Legit Data
L-mobile I Digitalisierte Softwarelösungen
{
"zipf": 2,
"binomial": 15,
"exponential": 4,
"poission": 2
}
- Created for demonstration purposes
- Response example
- Binomial distribution
- Exponential distribution
- Poisson distribution
- Zipf's law ~ Logarithmic
Seems Legit Data
L-mobile I Digitalisierte Softwarelösungen
- Dataset
Code | Name | Type | Unit |
---|---|---|---|
binomial | binomial | float | |
poission | poission | float | |
zipf | zipf | float | |
exponential | exponential | float |
Seems Legit Data
L-mobile I Digitalisierte Softwarelösungen
- Building the flow
Seems Legit Data
L-mobile I Digitalisierte Softwarelösungen
- Check Grafana for results
Any questions?
L-mobile I Digitalisierte Softwarelösungen
deck
By Imre Secodi
deck
- 94