Using Node-Red and Grafana WorldMap for geolocalized data visualization

Based on my previous posts we are now able to build a system that can receive, store and visualize data by using Node-Red, InfluxDB and Graphana. Grafana allows us build dashboards, query and visualize the stored data across time efficiently by using, in our case, the InfluxDB database engine. So far we’ve used simple line/bar charts to visualize data but we can use both Node-Red and Grafana to plot data onto a map:

  1. NodeRed Contrib World Map: Openstreet UI based map for plotting data with several options, including icon types, vectors, circles and heatmaps totally controlled through nodered flows.
  2. Grafana WorldMap plugin: Grafana panel with also an OpenStreet map for visualizing data.

Both have pros and cons, but the main differences between the two is that Node-Red Worldmap is suited more to real time display, and the Grafana plugin is better adapted to display data based on some time based query. Other major difference is that Node-Red Worldmap would require some coding, but, at least I consifer it, at an easy level, and the Grafana plugin is much harder to make it work.

Mapping data using Node-Red Worldmap:
One of the easiest ways for mapping data in realtime is using Node Red Worldmap node. The map is plotted and updated in real time.

cd ~
cd .node-red
npm install node-red-contrib-web-worldmap

After restarting and deploying a worldmap node, the map should be available at: http://server:1880/worldmap or other URL depending on the Node-Red base configuration.

One thing to keep in mind is that Node-Red is single user, so all instances of world maps (several different clients/browsers) will always have the same view.
The simplest way to start using the worldmap is just to copy and deploy the demo workflow provided by the node, but the key concept is that each point has a name and a set of coordinates.

msg.payload = {}; = "CentralLX"; = 38.7223;
msg.payload.lon = -9.1393;
msg.payload.layer = "SensorData";
msg.payload.UVLevel = getUVLevel() ;
msg.payload.Temperature = getTemp();

The cool thing is that if we inject repeatedly the above message (keeping the same name) but with different coordinates, the data point will move across the map in real time, and as I said earlier, the move will be reflect onto every client.

So all we need is an Inject node to a function node with the above code and feed it to the world map:

At the end we get this at the URL http://server:1880/worldmap:

Mapping data using Grafana Worldmap plugin:

The Grafana Worldmap plugin can get the location data in several ways. One of them is to use geohash data that is associated to the values/measurements.
There is a Node Red Geohash node that generates the geohash value from the latitude and longitude of data location. As usual we install the node:

cd ~
cd .node-red
npm install node-red-node-geohash

and then the Grafana plugin. We just follow the plugin instructions:

cd ~
grafana-cli plugins install grafana-worldmap-panel
/etc/init.d/grafana-server restart

With this per-requisites installed we can now feed data onto the database, in our case InfluxDB, that will be used by Grafana. We just need make sure that we add the geohash field. The geohash node will calculate from the node-red message properties lat – latitude and lon – longitude the required info:

A simple example:

Using the Influx tool, we can query our database and see that the geohash localization is now set:

> select * from SensorData limit 2
name: DemoValue
time                Temp UVLevel geohash    lat       lon        
----                ---- ------- -------   ---------- ----------  
1490706639630929463 22   8       eyckpjywh 38.7045055 -9.1754669  
1490706651606553077 21   7       eyckpjzjr 38.7044008 -9.1746488 

Anyway for setting up the World map plugin to display the above data was not straight forward, so the following instructions are more for a startup point rather than a solution.

The first thing to know is that the plugin is waiting for two fields: geohash and metric. With this in mind, before wasting too much time with the map plugin, a table panel that is filled with the required query is a precious tool to debug the query:

After we infer from the table that the data is more or less the data we want, we just transfer the query to WorldMap plugin:

Notice two important things: The aliasing for the query field to metric with the alias(metric) instruction, and the Format as: Table.

We can now setup the specific Worldmap settings:

On the Map Visual Options , I’ve centered the map in my location and set the zoom level. Fiddling around here can be seen in real time.

On the Map Data Options for this specific example, the Location Data comes from a table filled with the previous query (hence the format as table on the query output), and we want to see the current values with no aggregation.

When hoovering around a spot plotted on the map we can see a label: value, and the label used is obtained from a table field. In my case I just used geohash (not really useful…). Anyway these changes only work after saving and reloading the panel with F5 in my experience.

At the end we have now graphed data and localized data:

If we drag the selector on the left graphic panel, or select another time interval on top right menu of the grafana dashboard, the visualized information on the map changes.

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 3: Single point of access – Reverse proxy the services with nginx

Since we will be running a lot of services, each running on its own port, the following configuration, is optional, but allows to access all services through the same entry point by using Nginx server as a reverse proxy to Node-Red, Node-Red UI/Dashboard, Node-Red Worldmap and Grafana.

With this configuration the base URL is always the same without any appended ports, and the only thing that changes are the URL path:


To allow this we install and configure Nginx:

apt-get install nginx

The configuration files will reside in /etc/nginx directory. Under that directory there are two directories: sites-available and sites-enable where the later normally contains a link to configuration files located at sites-available.
At that directory there is a file named default that defines the default web site configuration used by Nginx. This is the file where we will add the reverse proxy directives.

Reverse proxy for Node-Red and Node-Red Contrib Worldmap
For setting up the reverse proxy for Node-Red we must first change the base URL for Node Red from / (root) to something else that we can map the reverse proxy.

For this we will need to edit the settings.js file located on the .node-red directory on the home path of the user running Node-Red.

We need to uncomment and change the entry httpRoot to point to our new base URL.

   httpRoot: '/nodered',

Don’t forget the trailing comma.

We need to restart now Node-Red and it should be accessible at the URL http://server:1880/nodered instead of http://server:1880/.

To configure Nginx, we edit the file default at /etc/nginx/sites-available and add the following section:

        location  /nodered {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_pass "";

        location / {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_pass "";


Note the following: The first location defines the reverse proxy URL /nodered to be served by the backend server The incoming path, /nodered, will be passed to the backend server URL /nodered, since paths are passed directly. No need to add the /nodered path to the backend server definition.
Also I’m using the address instead of localhost to avoid the IPv6 mapping to the localhost. In this way I’m sure that IPv4 will be used.

The location mapping for /nodered will make all the functionality of node red to work as it should at the base url /nodered. But some nodes, like node-red-contrib-worldmap will request to the proxy server ignoring the node-red base root map. Hence the / mapping. It will allow the worldmap nodes to work, but will stop this mapping to be used for something else.

Reverse proxy for Grafana

Setting up the reverse proxy for Grafana we can, and should use the following documentation: Grafana Reverse Poxy. For me the following configuration worked:

First edit the [server] section on the Grafana configuration file grafana.ini located at /etc/grafana.

Uncomment and edit the following lines:

# Protocol (http or https)
protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain =

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url =

Note the ending slash at the root_url. The same applies to the Nginx configuration

The files for the Nginx configuration are the same as the above configuration for reverse proxy.

We just need to add the following section after the previous location directives:

        location /grafana/ {
                proxy_pass http://localhost:3000/;

We should now restart nginx to refresh the configuration, and all should be working as it should by accessing the Grafana dashboard at

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 2: Database configuration and data collection

On the previous post we’ve installed and the base software for our Grafana based dash board.

We need now to configure our InfluxDB database and Node Red to start collecting data.

Configuring InfluxDB:
Detailed instructions for configuring an InfluxDB database are on this InfluxDB documentation link..

The main concepts that we need to be aware when using the InfluxDB is that record of data has a time stamp, a set of tags and a measured value. This allows, for example to create a value named Temperature and tag it depending on the source sensor:

Temperature: Value=22.1 , Sensor=Kitchen
Temperature: Value=21.9 , Sensor=Room1

This allows to process all the data or only process data based on a certain tag or tags. Values and tags can be created on the fly without previously define them, which is a bit different from standard RDBMS engines.

Creating an InfluxDB database:
To create the database, we need to access the machine hosting the InfluxDB server and execute the command influx:

odroid@odroid:~$ influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> create database SensorData
> show databases
name: databases


Now we have our database created and I’ve named SensorData. To make an example with the above temperature data we can do the following:

> insert Temperature,Sensor=kitchen value=22.1
ERR: {"error":"database is required"}

Note: error may be due to not setting a database or retention policy.
Please set a database with the command "use " or
> use SensorData
Using database SensorData

As we can see we need first to select the database where we are going to insert data with the command use SensorData:

> use SensorData
Using database SensorData
> insert Temperature, Sensor=kitchen value=22.1
ERR: {"error":"unable to parse 'Temperature, Sensor=kitchen value=22.1': missing tag key"}

> insert Temperature,Sensor=kitchen value=22.1
> insert Temperature,Sensor=Room1 value=21.9
> select * from Temperature
name: Temperature
time                Sensor  value
----                ------  -----
1487939008959909164 kitchen 22.1
1487939056354678353 Room1   21.9

Note that we can’t use spaces between the Measure name and the tags. The correct syntax is as follows:

 insert MeasureName,tag1=t1,tag2=t2,...   value1=val1,value2=val2,value3=val3,....

Also note that no DDL (data definition language) was used to create the tags or the measured value, we’ve just inserted data for our measurement with the our tags and value(s) without the need of previously define the schema.

Configuring Node-Red
Since we now have a database we can configure the InfluxDB Node Red nodes to store data onto the database:

There are two types of InfluxDB nodes, one that has an Input and Output and other that only has Input. The former is for making queries to the database where we provide on the input node the query, and on the output the results are returned. The later is for storing data only onto the database.
For both nodes we need to configure an InfluxDB server:

InfluxDB Server Configuration

We need to press the Pen icon right next to the server to add or reconfigure a new InfluxDB server:

InfluxDB server

A set of credentials are required, but since I’ve yet configured security, we can just put admin/admin as username and password. In a real deployment we must activate security.

From now on it is rather simple. Referring to InfluxDB node configuration screenshot (Not the InfluxDB server configuration) we have a configuration field named Measurement. This is our measure name that we associate a value. Picking up on the above example with the Insert command it will be Temperature, for example.

Now if the msg.payload provided has input to the node is a single value, let’s say 21, this is equivalent to do:

Insert Temperature value=12

We other formats for msg.payload that allows to associate tags and measures. Just check the Info tab for the node.

Simple example:

The following flow shows a simple example of a value received through MQTT, in this case the free heap from one of my ESP8266 and its storage in InfluxDB:

Sample Flow

[{"id":"20bec5de.8881c2","type":"mqtt in","z":"ced40abb.3c92e","name":"Heap","topic":"/outbox/ESP12DASH/Heap","qos":"2","broker":"2a552b3c.de8d2c","x":83.16668701171875,"y":206.41668701171875,"wires":[["e0d9c912.8c57f8","876fb151.6f2fa"]]},{"id":"876fb151.6f2fa","type":"debug","z":"ced40abb.3c92e","name":"","active":true,"console":"false","complete":"false","x":408.5,"y":177,"wires":[]},{"id":"e0d9c912.8c57f8","type":"influxdb out","z":"ced40abb.3c92e","influxdb":"bbd62a93.1a7108","name":"","measurement":"heap","x":446.1666717529297,"y":224.58335876464844,"wires":[]},{"id":"2a552b3c.de8d2c","type":"mqtt-broker","broker":"","port":"1883","clientid":"node-red","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":15,"cleansession":true,"willQos":"0","birthQos":"0"},{"id":"bbd62a93.1a7108","type":"influxdb","z":"","hostname":"","port":"8086","protocol":"http","database":"SensorData","name":"ODroid InfluxDB"}]

We can see with this flow the data stored in InfluxDB:

> select * from heap;
name: heap
time                value
----                -----
1487946319638000000 41600
1487946440913000000 41600
1487946562206000000 41600
1487946683474000000 41600
1487946804751000000 41600
1487946926061000000 41600
1487947047309000000 41616
1487947168594000000 41600

Now we have data that we can graph with Grafana, subject of my next posts.

Node Red Dashboard and UPS Monitoring

Just a quick hack to use the Node Red dashboard to monitor some of the UPS values that is attached to My Synology NAS.

Gathering the data and feeding it to Node-Red
First I thought to do some sort of Python or NodeJS program to run the upsc command, process the output and feed it, through MQTT, to Node Red.
But since it seemed to me a bit of overkill to just process a text output, transform it to JSON and push it through MQTT by using a program, I decided that I’ll use some shell scripting, bash to be more explicit.

I’m running on my Odroid C1+ “server” all the necessary components, namely Node Red with the Dashboard UI module.

So on Odroid I also have the ups monitoring tools, and upsc outputs a text with the ups status:

odroid@odroid:~$ upsc ups@
Init SSL without certificate database
battery.charge: 100
battery.charge.low: 10
input.transfer.high: 300
input.transfer.low: 140
input.voltage: 230.0
ups.load: 7
ups.mfr: American Power Conversion
ups.model: Back-UPS XS 700U  

So all we need now is to transform the above output from that text format to JSON and feed it to MQTT.
This means that we need to put between ” the parameter names and values, replace the : by , and also we need to replace the . on parameter names to _ so that in Node Red javascript we don’t have problems working with the parameter names.

Since I’m processing each line of the output, I’m using gawk/awk that allows some text processing. The awk program is as follow:

BEGIN {print "{"}
   print lline  "\42" $1 "\42:\42"$2"\42"
 {lline =", "}
END {print "}"}

This will at the beginning print the opening JSON bracket, then line by line the parameter name and value between ” and separated by : .
The lline variable at the first line is empty, so it prints nothing, but at the following lines it prints , which separates the JSON values.
We just need awk now to recognize parameters and values, and that is easy since they are separated by :

So if the above code is saved as procupsc.awk file, then the following command:

 upsc ups@ 2>/dev/null | awk -F: -f ~/upsmon/procupsc.awk |  sed 's/[.]/_/g'

Transforms the upsc output into a JSON output, including the replacement of . on variable names into _

"battery_charge":" 100"
, "battery_charge_low":" 10"
, "battery_charge_warning":" 50"
, "ups_load":" 7"
, "ups_vendorid":" xxxx"

Now all we need is to feed the output to the MQTT broker, and for this I’ll use the mosquitto_pub command, that has a switch that accepts the message from the standard input:

upsc ups@ 2>/dev/null | awk -F: -f /home/odroid/upsmon/procupsc.awk |  sed 's/[.]/_/g' | mosquitto_pub -h -t upsmon -s

So we define the host and the topic: upsmon and the message is the output of the previous command (the -s switch).

All we need now is on Node Red to subscribe to the upsmon topic and process the received JSON object.

Since I’m running this periodically on crontab, I also add the PATH variable so that all files and commands are found.
The complete script is as follows:

upsc ups@ 2>/dev/null | awk -F: -f /home/odroid/upsmon/procupsc.awk |  sed 's/[.]/_/g' | mosquitto_pub -h -t upsmon -s

and on Crontab:

# m h  dom mon dow   command
*/5 * * * * /home/odroid/upsmon/

Node Red processing and visualization
On node red side, now is easy. We receive the above upsc JSON object as a string on msg.payload, and we use the JSON node to separate into different msg.# variables.
From here we just feed the data to charts and gauges. The code is:

[{"id":"603732e7.bb8464","type":"mqtt in","z":"a8b82890.09ca7","name":"","topic":"upsmon","qos":"2","broker":"2a552b3c.de8d2c","x":114.5,"y":91,"wires":[["193a550.2b0ea2b"]]},{"id":"f057bae4.8e4678","type":"debug","z":"a8b82890.09ca7","name":"","active":true,"console":"false","complete":"payload","x":630.5,"y":88,"wires":[]},{"id":"563ec2f5.6475e4","type":"ui_gauge","z":"a8b82890.09ca7","name":"UPS Load","group":"ba196e43.b35398","order":0,"width":0,"height":0,"gtype":"donut","title":"Load","label":"%","format":"{{value}}","min":0,"max":"100","colors":["#00b500","#e6e600","#ca3838"],"x":630.5,"y":173,"wires":[]},{"id":"dbdb6731.531e8","type":"function","z":"a8b82890.09ca7","name":"UPS_Load","func":"msg.payload = Number(msg.payload.ups_load);\nreturn msg;","outputs":1,"noerr":0,"x":372.5,"y":175,"wires":[["563ec2f5.6475e4","f057bae4.8e4678","57bc92ab.ce4234"]]},{"id":"193a550.2b0ea2b","type":"json","z":"a8b82890.09ca7","name":"To Json","x":129.5,"y":178,"wires":[["dbdb6731.531e8","5433b116.a93b4","52604271.71f43c","11ad64b4.f70ad3","589635b3.9bacc4"]]},{"id":"57bc92ab.ce4234","type":"ui_chart","z":"a8b82890.09ca7","name":"Ups Load/Time","group":"ba196e43.b35398","order":0,"width":0,"height":0,"label":"Load/Time","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","ymin":"0","ymax":"100","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"x":639.5,"y":226,"wires":[[],[]]},{"id":"5433b116.a93b4","type":"function","z":"a8b82890.09ca7","name":"Battery Status","func":"var Vbats = msg.payload.battery_voltage;\n\nvar Vbat = Vbats.replace(\"_\",\".\");\n\nmsg.payload = Number(Vbat);\n\nreturn msg;","outputs":1,"noerr":0,"x":400.5,"y":336,"wires":[["8c6f269f.87b618"]]},{"id":"52604271.71f43c","type":"function","z":"a8b82890.09ca7","name":"V IN","func":"var Vins = msg.payload.input_voltage;\n\nvar Vin = Vins.replace(\"_\",\".\");\n\nmsg.payload = Number(Vin);\nreturn msg;","outputs":1,"noerr":0,"x":369.5,"y":570,"wires":[["240f6e31.7aa5da","b37dffd2.1cb0d"]]},{"id":"8c6f269f.87b618","type":"ui_gauge","z":"a8b82890.09ca7","name":"","group":"421e19.192041e8","order":0,"width":0,"height":0,"gtype":"gage","title":"Curr. Bat. Voltage","label":"V. Bat","format":"{{value}}","min":"11","max":"15","colors":["#b50000","#e6e600","#00b500"],"x":616.5,"y":332,"wires":[]},{"id":"240f6e31.7aa5da","type":"ui_gauge","z":"a8b82890.09ca7","name":"Input AC Voltage","group":"4e74439e.ee7e74","order":0,"width":0,"height":0,"gtype":"gage","title":"VIN AC","label":"V AC","format":"{{value}}","min":"190","max":"240","colors":["#00b500","#e6e600","#ca3838"],"x":660.5,"y":569,"wires":[]},{"id":"11ad64b4.f70ad3","type":"function","z":"a8b82890.09ca7","name":"Bat Runtime","func":"msg.payload = Number(msg.payload.battery_runtime);\nreturn msg;","outputs":1,"noerr":0,"x":397.5,"y":397,"wires":[["98292081.f4f718","62bb1456.0f45ec"]]},{"id":"98292081.f4f718","type":"ui_gauge","z":"a8b82890.09ca7","name":"UPS Level","group":"fccb6f27.7691d8","order":0,"width":0,"height":0,"gtype":"wave","title":"UPS Runtime","label":"UPS Level","format":"{{value}}","min":0,"max":"1500","colors":["#00b500","#e6e600","#ca3838"],"x":642.5,"y":391,"wires":[]},{"id":"589635b3.9bacc4","type":"function","z":"a8b82890.09ca7","name":"Bat Charge","func":"var Vcharges = msg.payload.battery_charge;\n\nvar Vcharge = Vcharges.replace(\"_\",\".\");\n\nmsg.payload = Number(Vcharge);\n\n\nreturn msg;","outputs":1,"noerr":0,"x":405,"y":495,"wires":[["d2ab2543.d53a68"]]},{"id":"d2ab2543.d53a68","type":"ui_gauge","z":"a8b82890.09ca7","name":"Battery Charge","group":"421e19.192041e8","order":0,"width":0,"height":0,"gtype":"gage","title":"Battery Charge","label":"%","format":"{{value}}","min":0,"max":"100","colors":["#ff0000","#e6e600","#00ff01"],"x":661.5,"y":494,"wires":[]},{"id":"62bb1456.0f45ec","type":"ui_chart","z":"a8b82890.09ca7","name":"","group":"fccb6f27.7691d8","order":0,"width":0,"height":0,"label":"Runtime (sec)","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"x":625.5,"y":441,"wires":[[],[]]},{"id":"b37dffd2.1cb0d","type":"ui_chart","z":"a8b82890.09ca7","name":"Vin/Time","group":"4e74439e.ee7e74","order":0,"width":0,"height":0,"label":"VAC In/Time","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","ymin":"190","ymax":"240","removeOlder":"12","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"x":635.5,"y":641,"wires":[[],[]]},{"id":"2a552b3c.de8d2c","type":"mqtt-broker","broker":"","port":"1883","clientid":"node-red","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":15,"cleansession":true,"willQos":"0","birthQos":"0"},{"id":"ba196e43.b35398","type":"ui_group","z":"","name":"UPS Load","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"421e19.192041e8","type":"ui_group","z":"","name":"UPS Battery","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"4e74439e.ee7e74","type":"ui_group","z":"","name":"Input Voltage","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"fccb6f27.7691d8","type":"ui_group","z":"","name":"UPS Runtime","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"61ec3881.53526","type":"ui_tab","z":"","name":"UPS","icon":"dashboard","order":2}]

The final output is as follow:

Node Red UPS Monitoring

Node Red UPS Monitoring

Upgrading NodeJs and Node Red on Odroid

I run many services on my Odroid C1+ including Node-Red. But since NodeJs on Odroid C1+ is version v0.10 is starting to be seriously old for running Node-Red or other NodeJS dependent software.

So my quick instructions for upgrading NodeJS and Node-Red on the Odroid C1+

Upgrading NodeJS

First verify what version is available/installed on the Odroid:

odroid@odroid:~$ node -v
odroid@odroid:~$ nodejs -v

Since I’ve already had previously installed a more recent version of NodeJS (the node command), the version used by Node-Red is v0.12.14 while the default NodeJS version is v0.10.25.

We can also, and should, check the npm version:

odroid@odroid:~$ npm -v

We also need to find what architecture we are using, just for completeness since ODroid C1+ is an ARM7 based architecture:

odroid@odroid:~$ uname -a
Linux odroid 3.10.96-151 #1 SMP PREEMPT Wed Jun 15 18:47:37 BRT 2016 armv7l armv7l armv7l GNU/Linux

This will allow us to download the correct version of the NodeJS binaries from the NodeJS site: NodeJS downloads.
In our case we choose the ARM7 architecture binaries, which at the current time is file: node-v6.9.2-linux-armv7l.tar.xz
So I’ve just copied the link from the NodeJS site and did a wget on the Odroid:


I then created a working directory and “untared” the file:

odroid@odroid:~$ mkdir nodework
odroid@odroid:~$ cd nodework
odroid@odroid:~/nodework$ tar xvf ../node-v6.9.2-linux-armv7l.tar.xz
odroid@odroid:~/nodework$ cd node-v6.9.2-linux-armv7l/

Since there isn’t an install script we need to move the new NodeJS files to the correct locations:

  1. Binaries to /usr/bin
  2. Include files to /usr/include
  3. Libs files to /usr/lib

Copy the binaries, replacing, if existing the older versions:

$ sudo cp -i node /usr/bin
cp: overwrite ‘/usr/bin/node’? y

Copy the include files:

$ sudo cp -R node  /usr/include/

and copy the libraries

$ sudo cp -R node_modules /usr/lib

and finally:

$ sudo cp -R . /usr/share

We need now to make npm to point to the correct nodejs script so, we need to delete the npm link at the /usr/bin and /usr/local/bin directories:

odroid@odroid:~$ sudo rm /usr/bin/npm
odroid@odroid:~$ sudo rm /usr/local/bin/npm

and re-create the correct links:

odroid@odroid:~$ sudo ln -s /usr/lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm
odroid@odroid:~$ sudo ln -s /usr/lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm

Running now the node and npm commands should report the latest versions:

odroid@odroid:~$ node -v
odroid@odroid:~$ npm -v


Upgrading Node-Red

From the Node-Red startup log, we can see the previous versions of node-red and nodejs used:

Welcome to Node-RED

28 Dec 17:55:40 - [info] Node-RED version: v0.15.2
28 Dec 17:55:40 - [info] Node.js  version: v0.12.14
28 Dec 17:55:40 - [info] Linux 3.10.96-151 arm LE
28 Dec 17:55:42 - [info] Loading palette nodes
28 Dec 17:55:50 - [info] Dashboard version 2.1.0 started at /ui
28 Dec 17:55:54 - [warn] ------------------------------------------------------
28 Dec 17:55:54 - [warn] [rpi-gpio] Info : Ignoring Raspberry Pi specific node
28 Dec 17:55:54 - [warn] ------------------------------------------------------

we can upgrade now Node-Red according to the Node Red upgrading instructions:

odroid@odroid:~$ sudo npm cache clean
odroid@odroid:~$ sudo npm install -g --unsafe-perm node-red

and after a while the upgrade should be done.

Before starting up node-red I went to the node-red module directories, and did an update:

odroid@odroid:~/.node-red$ npm update
└── crypto-js@3.1.8 

Starting up Node-Red should show now the new software versions:

Welcome to Node-RED

1 Jan 20:35:46 - [info] Node-RED version: v0.15.2
1 Jan 20:35:46 - [info] Node.js  version: v6.9.2
1 Jan 20:35:46 - [info] Linux 3.10.96-151 arm LE
1 Jan 20:35:47 - [info] Loading palette nodes
1 Jan 20:35:54 - [info] Dashboard version 2.2.1 started at /ui


ESP8266 – Logging data in a backend – AES and Crypto-JS

After building, on the previous posts, the Node-Red based backend to support E2EE (End to End Encryption) so we can log data into a central server/database, from our devices securely, without using HTTPS, we need now to build the firmware for the ESP8266 that allows it to call our E2EE backend.

The firmware for the ESP8266 must gather the data that it wants to send, get or generate the current sequence number for the data (to avoid replay attacks), encrypt the data and send it to the backend.
On the backend we are using the Java script library for cryptographic functions Crypto-js, and specifically we are encrypting data with the encryption algorithm AES. So all we need is to encrypt our data with AES on the ESP8266, send it to the Node-Red Crypto-js backend, decrypt it and store it, easy right?

Not quite, let’s see why:

Crypto-js and AES:
We can see that on my Node-Red function code and testing programs I’m using something similar to the following code example:

var CryptoJS = require("crypto-js");
var message  = "Message to encrypt";
var AESKey   = '2B7E151628AED2A6ABF7158809CF4F3C';

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text in Base64: " ,  ciphertext.toString(CryptoJS.enc.base64) );

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Several points regarding the above code need clarification:

The code variable AESKey the way it is used on the above example encrypt and decrypt functions isn’t really a key but a passphrase from where the real key and an initialization vector or salt value is generated (I’m using the names interchangeably, but they are not exactly the same thing except they are public viewable data that must change over time/calls).
The use for the generated key is self explanatory, but the initialization vector (IV) or salt value is used to allow that the encrypted data output to be not the same for the same initial message. While the key is kept constant and secret to both parties, the IV/salt changes between calls, which means that the above code, when run multiple times, will never produce the same output for the same initial message.

Still referring to the above code, the algorithm that generates the key from the passphrase is the PBKDF2 algorithm. More info at Crypto-js documentation. At the end of the encryption the output is a OpenSSL salted format that means that the output begins by the signature id: Salted_, followed by an eight byte salt value, and after the salt, the encrypted data.

So if we want use the API has above on the node-js/crypto-js side, we need to implement on the ESP8266 side both the AES and PBKDF2 algorithms.

I decided not to do that, first because finding a C/C++ implementation of the PBKDF2 algorithm that could be portable and worked on the ESP822 proved difficult, and second the work for porting it to the ESP8266 won’t be needed if I use a KEY/IV directly, and so I decided to use the more standard way of providing an AES key and an initialization vector for encrypting and decrypting data.

In the case of Node-JS and Crypto-JS when using an explicit key and IV the code looks like:

var CryptoJS = require("crypto-js");
var request = require('request');

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":145 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);

var iv = CryptoJS.enc.Hex.parse('0000000000000000');
var key= CryptoJS.enc.Hex.parse(AESKey);

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, key , { iv: iv } );

//console.log("Cypher: ", ciphertext );
console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");

console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Now, with above code, where the IV is always initialized to the same value, in this case ‘0000000000000000’, we can see when running the above code several times that the output is always the same since the IV is kept constant. Also the encrypted output is now just the raw encrypted data and not the Openssl format.

So to make the above code secure we must randomize the IV value for producing an output that is always different, even from several program runs when encrypting the same source data.

As a final note, if we count the number of HEX characters on the Key string, we find out that they are 16 bytes, which gives a total of 128 key bits. So the above example is using AES128 encryption, and with default Crypto-js block mode and padding algorithms which are CBC (Chain block mode) and pkcs7.

Interfacing Crypto-js and the ESP8266:
Since we are using AES for encrypting data and decrypting data, we need first to have an AES library for the ESP8266. The AES library that I’m using is this one Spaniakos AES library for Arduino and RPi. This library uses AES128, CBC and pkcs7 padding, so it ticks all boxes for compatibility with Crypto-js…

I just added the code from the above library to my Sming project and also added this Base64 library so that I can encode to and from Base64.

The only remaining issue was to securely generate a truly random initialization vector. And while at first I’ve used some available libraries to generate pseudo-random numbers to real random numbers, I’ve found out that the ESP8266 seems to have already a random number generator that is undocumented: Random number generator

So to generate a random IV value is as easy as:

uint8_t getrnd() {
    uint8_t really_random = *(volatile uint8_t *)0x3FF20E44;
    return really_random;

// Generate a random initialization vector
void gen_iv(byte  *iv) {
    for (int i = 0 ; i < N_BLOCK ; i++ ) {
        iv[i]= (byte) getrnd();

So our ESP8266 code is as follow:

Global variables declarations:
The N_Block defines the encryption block size, that for AES128 is 16 bytes.

#include "AES.h"
#include "base64.h"

// The AES library object.
AES aes;

// Our AES key. Note that is the same that is used on the Node-Js side but as hex bytes.
byte key[] = { 0x2B, 0x7E, 0x15, 0x16, 0x28, 0xAE, 0xD2, 0xA6, 0xAB, 0xF7, 0x15, 0x88, 0x09, 0xCF, 0x4F, 0x3C };

// The unitialized Initialization vector
byte my_iv[N_BLOCK] = { 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};

// Our message to encrypt. Static for this example.
String msg = "{\"data\":{\"value\":300}, \"SEQN\":700 , \"msg\":\"IT WORKS!!\" }";

The example function ESP8266 Sming code is:

void testAES128()  {

    char b64data[200];
    byte cipher[1000];
    byte iv [N_BLOCK] ;
    Serial.println("Let's encrypt:");
    aes.set_key( key , sizeof(key));  // Get the globally defined key
    gen_iv( my_iv );                  // Generate a random IV
    // Print the IV
    base64_encode( b64data, (char *)my_iv, N_BLOCK);
    Serial.println(" IV b64: " + String(b64data));
    Serial.println(" Mensagem: " + msg );
    int b64len = base64_encode(b64data, (char *)msg.c_str(),msg.length());
    Serial.println (" Message in B64: " + String(b64data) );
    Serial.println (" The lenght is:  " + String(b64len) );
    // For sanity check purpose
    //base64_decode( decoded , b64data , b64len );
    //Serial.println("Decoded: " + String(decoded));
    // Encrypt! With AES128, our key and IV, CBC and pkcs7 padding    
    aes.do_aes_encrypt((byte *)b64data, b64len , cipher, key, 128, my_iv);
    Serial.println("Encryption done!");
    Serial.println("Cipher size: " + String(aes.get_size()));
    base64_encode(b64data, (char *)cipher, aes.get_size() );
    Serial.println ("Encrypted data in base64: " + String(b64data) );

When the above code/function is executed on the ESP8266 it outputs the following:

Let's encrypt:
 IV b64: cAFviaDMHejlteGn9/4eQQ==
 Mensagem: {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }
 Message in B64: eyJkYXRhIjp7InZhbHVlIjozMDB9LCAiU0VRTiI6NzAwICwgIm1zZyI6IklUIFdPUktTISIgfQ==
 The lenght is:  76
Encryption done!
Cipher size: 80
Encrypted data in base64: /1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=

And now we can feed the above Base64 IV and encrypted data to our decoding program in Node-Js using Crypto-JS:

var CryptoJS = require("crypto-js");
var request = require('request');

var esp8266_msg = '/1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=';
var esp8266_iv  = 'cAFviaDMHejlteGn9/4eQQ==';

// The AES encryption/decryption key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

var plain_iv =  new Buffer( esp8266_iv , 'base64').toString('hex');
var iv = CryptoJS.enc.Hex.parse( plain_iv );
var key= CryptoJS.enc.Hex.parse( AESKey );

console.log("Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt( esp8266_msg, key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Base64);
var decoded_b64msg =  new Buffer(plaintext , 'base64').toString('ascii');
var decoded_msg =     new Buffer( decoded_b64msg , 'base64').toString('ascii');

console.log("Decrypted message: ", decoded_msg);

and the output is:

Decrypted message:  {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }

So, as the message shows, it WORKS!. AES encryption on the ESP8266 to a Node-JS Crypto-JS based code where decryption occurs.

Final notes:
So all is needed now is to build on the ESP8266 side the message with the encrypted data and the IV, and send through plain HTTP a JSON object to the Node-Red back end.


On the Node-Red back end the decryption can now be done easily as was shown on the above testing code.

ESP8266 – Logging data in a backend with end to end encryption – Storing Data

As we can see in my previous post, for securely store data from a device, we should make the data secure during transport and tamper proof. This can be achieved by using encryption, but the use of encryption only does not solve some attacks, like replay attacks.

So a scheme where data is encrypted and associated with a sequence number is used, to have transport security and replay attack protection, and on the previous post we’ve implemented the base framework that allows the support of our requirements.

So the device generates, for example, the following message:

 { "data":{"value": 300} , "SEQN": 120 }

meaning that it wants to send to the backend the JSON Object {“value”: 300}, and the current Sequence number is 120. On the next message the sequence number should be 121, but that is supposing that algorithm for the sequence number is to increase it one by one per message.

Giving the message, we encrypt it now with our device key, and build another JSON object, the one that will be sent to the Node-red based backend:

 { "msg":"U2FsdGVkX1..."}

Since we still don’t have our device ready to send messages, we will build a simple NodeJs program to generate the messages:

mkdir ~/gen
cd ~/gen
npm init   (Just accept the defaults)
npm install crypto-js --save
npm install request --save

And the code to generate our message is as follow:

var CryptoJS = require("crypto-js");
var request = require('request');

// API endpoint.
var apiEP = 'http://localhost:1880/data/';
var deviceID = '12FA';

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":121 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);
console.log("Message: " , message );

// Encode the string to base64. Not really needed.
message = new Buffer(message).toString("base64");
console.log("Message B64: " , message );

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");
console.log("Let's call the Node-Red API end point: ");

var URL = apiEP + deviceID;
var rawdata = ciphertext.toString(CryptoJS.enc.base64);

console.log(" Calling end point: " , URL);
console.log(" RawData: " , rawdata );

// Let's call the REST API end point.
request( {
    url: URL,
    method: "POST",
    json: true,
    body: { "msg": rawdata}
    } ,
    function (error, response, body) {
        if (!error && response.statusCode == 200) {
            console.log(" ");
            console.log("REST API Output: ");
console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);
console.log(" ");

console.log("Decrypted message: " , new Buffer(plaintext , 'base64').toString('ascii'));

We can modify the message by modifying the msgObjs variable. We run this program by executing node index.js and at the end the Node-Red answer should be displayed under REST API Output.

If everything is setup correctly the REST API should return:

REST API Output: 
{ status: 'OK' }

But calling a second time without modifying the sequence number should fail:

REST API Output: 
{ status: 'NOT OK' }

The Node-Red flow:

The Node-Red flow, receives the REST POST request for the device ID, obtains from the database the private key associated to the device, decrypts the payload, and checks the message sequence number vs the database sequence number. By design it allows the device to send messages above the current sequence number to allow message gaps (messages that where lost). We can, if needed, process this gaps so we can have an idea of how many messages that we are loosing.

If the decryption is successful and the sequence number is ok, we increment the sequence number to the next value, and store the data:

Node-Red Data Storage flow

Node-Red Data Storage flow

The code is as follow:

[{"id":"501864bd.eea2ec","type":"sqlitedb","z":"ee002ffe.ffd9e8","db":"/home/odroid/Databases/wsn.db"},{"id":"9e2ef.10b2b512","type":"http in","z":"ee002ffe.ffd9e8","name":"setSensorData","url":"/data/:id","method":"post","swaggerDoc":"","x":142.1666717529297,"y":812.0833587646484,"wires":[["fa205b8f.8de96"]]},{"id":"fa205b8f.8de96","type":"function","z":"ee002ffe.ffd9e8","name":"Build Query","func":"// Get the device id\nvar deviceId =;\n\n// Build the query. The SQLITE node requires the query in msg.topic\nmsg.topic=\"Select * from Devices where deviceID='\" + deviceId +\"'\";\n\n// Let's pass these parameters forward on its on variables:\nmsg.deviceid = deviceId;\nmsg.rawdata  = msg.payload;\n\nreturn msg;","outputs":1,"noerr":0,"x":199.1666717529297,"y":882.7499694824219,"wires":[["94004285.aa1418"]]},{"id":"94004285.aa1418","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Get Device AES Key","x":404.16668701171875,"y":817.0833435058594,"wires":[["7dd8f03c.ab2778"]]},{"id":"d0ed1bda.b15428","type":"http response","z":"ee002ffe.ffd9e8","name":"","x":1023.1666870117188,"y":956.8333435058594,"wires":[]},{"id":"7dd8f03c.ab2778","type":"function","z":"ee002ffe.ffd9e8","name":"Decrypt Request","func":"var cryptojs =;\ntry {\n    // Get the key, Sequence number and the raw encrypted data\n    var AESKey = msg.payload[0].deviceKey;\n    msg.dbSQN = msg.payload[0].deviceSQN;   // Obtain also the currrent SQN on the database\n    \n    var rawdata= msg.rawdata.msg;\n    \n    node.log(  msg.devSQN );\n    // Decrypt the payload data with the device key. \n    // It returns a string sequence of bytes.\n    var bytes = cryptojs.AES.decrypt( rawdata, AESKey );\n    \n    // Convert bytes to an UTF8 plain string\n    var plaintext = bytes.toString(cryptojs.enc.Utf8);\n    node.log( plaintext );\n    // Convert from base64 to string\n    msg.payload  = new Buffer(plaintext , 'base64').toString('ascii');\n\n    return [ null , msg ];  // Exit the function at output 2.\n    \n} catch (err) {\n    msg.payload = { \"status\":\"NOT OK\"};\n    msg.statusCode = 500;     // Set internal server error\n    node.log(\"Invalid deviceID request for GET SQN Operation\");\n    return [ msg , null ];  // Exit the function at output 1\n}\nreturn msg;","outputs":"2","noerr":0,"x":668.1666870117188,"y":817.7499694824219,"wires":[["d0ed1bda.b15428"],["c7be8b09.893b9"]]},{"id":"c7be8b09.893b9","type":"function","z":"ee002ffe.ffd9e8","name":"Verify MSG SQN","func":"// At this point we should have:\n// msg.payload with the sequence number in JSON: { SEQN: 200} and the rest of the message\n// msg.deviceid with the device id\ntry {\n    var msgObj = JSON.parse(msg.payload);\n    var msgSeqn = msgObj.SEQN;   // The sequence number that the device as sent\n    var msgData =;   // The data sent\n  \n    if ( msg.dbSQN <= msgSeqn )  { // Valid Sequence number\n      node.log( \"Sequence is VALID!!!!!\");\n      msg.payload = { \"status\":\"OK\"};\n = msgData;\n      msg.devSQN= msgSeqn;\n      return [ msg, msg];\n    } else {\n        node.log( \"Sequence is invalid!!!!\");\n        msg.payload = { \"status\":\"NOT OK\"};\n        return [ null , msg ];\n    }\n} catch( e ) {\n    node.log(\"Error verifying SEQN...\") \n    msg.payload = { \"status\":\"Internal Error\"};\n    return [ null , msg ];\n}","outputs":"2","noerr":0,"x":164.1666717529297,"y":996.7499694824219,"wires":[["c8ce0d1c.589058","533d8275.e04474"],["d0ed1bda.b15428"]]},{"id":"c8ce0d1c.589058","type":"function","z":"ee002ffe.ffd9e8","name":"Update Sequence ","func":"\n    var dbSQN = msg.devSQN + 1;   // It allows message gaps.\n\n    msg.topic=\"Update Devices Set deviceSQN= \" + dbSQN + \" Where deviceID='\" + msg.deviceid +\"'\";\n \n    //node.log(\"SQL: \" + msg.topic );\nreturn msg;","outputs":1,"noerr":0,"x":584.1666870117188,"y":1034.5833740234375,"wires":[["bef21dc4.4642a"]]},{"id":"bef21dc4.4642a","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Set SQN","x":780.1666870117188,"y":1034.4166259765625,"wires":[[]]},{"id":"533d8275.e04474","type":"function","z":"ee002ffe.ffd9e8","name":"Update Data","func":"  // Let's extract the data.\n  // This step should be modified as needed.\n  var data =;\n  msg.topic=\"Insert into Data Values ( '\" + msg.deviceid + \"', CURRENT_TIMESTAMP , \" + data + \")\";\n \n  //node.log (\"Data: \" + data );\nreturn msg;","outputs":1,"noerr":0,"x":579.1666870117188,"y":922.5833435058594,"wires":[["770abb44.6f54d4"]]},{"id":"770abb44.6f54d4","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Save Data","x":782.1666870117188,"y":922.4165954589844,"wires":[[]]}]

And so the only thing missing is the ESP8266 code to call and setting data using the encrypted transport. For implementing that we will be using the Sming framework that will use and call the above defined API on this post and previous.