The Air Quality Monitor – Data processing

After implementing and deploying the DSM501a based Air quality monitor, data was collected into an InfluxDB database and consumed by a Grafana Dashboard:

DSM501a and BMP180 Grafana Dashboard

While we can see that Temperature (yes it’s hot!) and Pressure looks ok, data collected from the DSM501a sensor is just a mess. It’s just a bunch of samples jumping around across several values, and so doesn’t look very promising that we can take meaningful data out of it.

So we might need to process the data so that it makes sense, and for that based on the fact that:

  1. Data sample is slow: 2 samples per minute
  2. We don’t want high variations, but smooth out the data

I’ve choosen to filter data using an IIR (Infinite Response Filter) LPF (Low pass filter) to remove any high frequency data/noise, and hence obtain a smoother output.
Did it work? Yes it did:

Original data vs IIR LPF filtered data

As we can see for each collected particle size of 1.0 and 2.5 we’ve filtered it with an IIR LPF that smoothed out any wild transitions while keeping the fundamental and underlying data validation.

Implementation:
IIR implementation is quite simple since it is only a set of additions, subtractions and multiplications with some factors that define the behavior of the filter.

IIR Filter

(The picture was taken from here that also explains nicely what is an IIR and FIR filters).

The input x[]n is DSM501a sample time at t=0, t=30s, t=60s, … and so on, and y[]n is the corresponding output. The b0,b1,b2 and a0,a1 and a2 are the filter factors, that define the filter response. For testing purposes I’ve just choose factors for a 1KHz Low Pass filter and tested it during several days, and hence the above output that can be seen on the Grafana dashboard.

The IIR filtering process is done on Node-Red but it can be done easily also on the ESP8266 since there is no complicated math/algorithms involved.

Node-Red IIR LPF filter

The function that implements the IIR LPF filter is (Note that on the code I use the a’s as the input factors and b’s as the output factors which is the contrary of the above IIR picture):

// IIR LPF factors
  f_a0 = 0.0010227586546542474;     // Input factors
  f_a1 = 0.002045517309308495;
  f_a2 = 0.0010227586546542474;
  
  f_b1 = -1.9066459797557103;       // Output factors
  f_b2 = 0.9107370143743273;

// PPM 1.0 input variables
var i0_c10 = msg.payload.cPM10;
var i1_c10 = context.get('i1_c10') || 0;
var i2_c10 = context.get('i2_c10') || 0;

// PPM 1.0 output variables
var o0_c10 = context.get('o0_c10') || 0;
var o1_c10 = context.get('o1_c10') || 0;


// Calculate the IIR
var lpf =   i0_c10 * f_a0 + 
            i1_c10 * f_a1 + 
            i2_c10 * f_a2 -         // We add the negative output factors
            o0_c10 * f_b1 - 
            o1_c10 * f_b2;
            
// Memorize the variables
context.set( 'i2_c10' , i1_c10 );
context.set( 'i1_c10' , i0_c10 );

context.set( 'o1_c10' , o0_c10 );
context.set( 'o0_c10' , lpf );

// PPM 2.5 input variables
var i0_c25 = msg.payload.cPM25;
var i1_c25 = context.get('i1_c25') || 0;
var i2_c25 = context.get('i2_c25') || 0;

// PPM 1.0 output variables
var o0_c25 = context.get('o0_c25') || 0;
var o1_c25 = context.get('o1_c25') || 0;


// Calculate the IIR
var lpf25 =   i0_c25 * f_a0 + 
              i1_c25 * f_a1 + 
              i2_c25 * f_a2 -         // We add the negative output factors
              o0_c25 * f_b1 - 
              o1_c25 * f_b2;
            
// Memorize the variables
context.set( 'i2_c25' , i1_c25 );
context.set( 'i1_c25' , i0_c25 );

context.set( 'o1_c25' , o0_c25 );
context.set( 'o0_c25' , lpf25 );

msg.payload = {}
msg.payload.cfP10 = lpf;
msg.payload.cfP25 = lpf25;

return msg;

We maintain the filter state (the two previous samples from the sensor) on Node-Red global variables (which will be reset if Node-red is restarted), and calculate for each PM1.0 and PM2.5 sample the filtered value, which depends on the previous samples. The final output is then fed to an InfluxDB sink node which saves the filtered data.
The complete code is at this gist.

Conclusion:
While still this being a test by using a probably LPF filter that is not adequate to the sampled data (it was designed for Audio at 96Khz sample rate), it shows that we can do some simple processing to clean up inbound data so that it makes more sense. This mechanisms of using digital filtering signals (DSP) are applied widely in other systems such as electrocardiogram sensors or other biometric sensors the remove or damp signal noise. In this case we can see that after the filtering data looks much more promising to be processed and so be used to calculate the Air Quality Index without the index jumping around as the samples jump.

PZEM-004T ESP8266 software

Following up the home energy meter post based on an ESP8266 and PZEM-004T hardware, this post describes succinctly the software for using the energy meter.

There are at least two components to the solution:

  1. ESP8266 software for driving the power meter and make the measurements.
  2. The backend software for receiving and processing data.

The ESP8266 software:
The power meter software for the ESP8266 available on this GitHub repository, uses an available PZEM-004T library for accessing the power meter, and sends the collected data through MQTT to any subscribers of the power meter topic.
I’m using the convention that is also used on Thingsboard, namely an MQTT attributes topic to publish the device status, and a telemetry topic to post the data in JSON format.
Around lines 80 on main.cpp of PowerMeter sources, the topics are defined as:

  1. Attributes: “iot/device/” + String(MQTT_ClientID) + “/attributes”
  2. Telemetry: “iot/device/” + String(MQTT_ClientID) + “/telemetry”

MQTT_ClientID is defined on the secrets.h file, where we also define a list of available WIFI connections for our ESP8266. The attributes topic periodically sends the current device status (RSSI, HEAP, wifi SSID), while the data on the telemetry topic is fed into a timeseries database such as InfluxDB where then a Grafana Dashboard shows and allows to see the captured data across time.

As also my previous post regarding framework and libraries versions, I needed to block the ESP8266 framework version and the SoftwareSerial library because the combination of these with the PZE-004T library was (is ?) broken of more recent versions. As is currently defined on the platformio.ini file, the current set of versions, work fine.

A lot of people had problems working with the use of SoftwareSerial library for the PZEM library to communicate with the hardware. The issue, that I accidentally found out, are related with timing issues to communicate with the PZEM hardware. There are periods of time that the PZEM is not responsive, probably because is making some measurement.

The solution to this issue is at start up to try the connection during some time, at 3 seconds interval until it succeeds. After the connection is successful, we need to keep an interval around one minute between reads to encounter no issues/failures . If this interval is kept, the connection to the PZEM hardware works flawlessly, at least with the hardware that I have.

So the connection phase is checked and tried several times to synchronize the ESP8266 with the PZEM, and them every single minute there is a data read. If the interval is shorter, lets say, 30s, it will fail, until the elapsed time to one minute is completed.

The firmware solves the above issue, and after reading the data, it posts it to a MQTT broker. The firmware also makes available a web page with the current status and measurements:

Power Meter Web Page

Then there are other bits, namely since the meter will be on the electric mains board, an UDP logging facility that allows on the computer to run an UDP server and see what is going on.

The back-end software:
I’ve not done much on this area, since most of it is just standard stuff. An MQTT broker and Node-Red flow. The flow just receives the data, saves it into an InfluxDB database and creates a Node-Red UI dashboard.

Power Meter Node-Red UI

This screenshot shows some of the information that was collected on the last minute and it is updated in real time as soon the PowerMeter information arrives to the MQTT broker.

Future work:
Basically what is missing is two things:

  1. Grafana Dashboard based on the InfluxDB data (Already done, to be described in a future post).
  2. Some kind of exporter to CSV or Spreadsheat to allow further data analysis such as the daily power consumption totals.

Node-Red: Checking network service port status + UI status indicator

This post is about how to do two simple things using Node-Red:

  1. Check if network service on the machine running Node-Red is available by checking the corresponding listening port.
  2. The Node-Red UI doesn’t have a status indicator available, so I’ve built one

The only limitation on the following solution is that it only tests for ports for services that are running on the same server, where Node-Red is also running.

Preparation:

We need to install the Is Port Available NPM Module and make it available into our Node-Red instance.

For doing so, in Linux we must do the following:

root@server:~# cd .node-red/
root@server:~/.node-red# npm i --save is-port-available

We need now to make this node module available to Node-Red by editing the settings.js file:

root@server:~/.node-red# vi settings.js

Add the module to the global context on the function named functionGlobalContext:

    functionGlobalContext: {
        // os:require('os'),
        // octalbonescript:require('octalbonescript'),
        // jfive:require("johnny-five"),
        // j5board:require("johnny-five").Board({repl:false})

        portavail:require('is-port-available')
    },

You might have other modules configured, so we need to add the above portavail:require(‘is-port-available’) line to that list preceded by a comma.

We need to restart Node-Red to make the module available to the flows.

The testing flow
In our Function nodes, we can now use the global context object portavail to access the is-port-available module.

For example for testing the InfluxDB server port (1086/TCP) we can write the following function:

    // Instantiate locally on the flow the is-port-available module
    const isPortAvailable = context.global.portavail;

    msg.payload = {};   // Zero out the message. Not really necessary
     
    var port = 1086; // Replace this with your service port number. In this case 1086 is the Influx DB port
    
    isPortAvailable(port).then( status => {
        if(status) {
            //console.log('Port ' + port + ' IS available!');
            msg.payload = {'InfluxDB':false,"title":"InfluxDB","color":"red"};   // The port is available, hence the server is NOT running
            node.send(msg);
        } else {
            //console.log('Port ' + port + ' IS NOT available!');
            //console.log('Reason : ' + isPortAvailable.lastError);
            msg.payload = {'InfluxDB':true,"title":"InfluxDB","color":"green"};    // The port is not available, so the server MIGHT be running
            node.send(msg);
           
        }
    });

    // Note that we DO NOT return a message here since the above code is asynchronous and it will emit the message in the future. 

Since the test is using promises, Node-Red will continue executing without waiting for the test response (the isPortAvailable(port) code ). So we do not send any message further on the normal Node-Red execution flow (hence there is no return msg; object) and the message is only emitted when the promise fulfils. When that happens we just send the message with the node.send(msg) statement.

The message payload can be anything, being the only important properties the title and color that are used for creating the UI status indicator.

The status indicator is a simple Angularjs template that displays the title and a status circle with the chosen colour.

Since pasting CSS and HTML code in WordPress is recipe to disaster, the template code can be accessed on this gist or on the complete test flow below:

[{"id":"1f506795.4be25","type":"inject","z":"53f8b852.885c6","name":"Check todos os 60s","topic":"","payload":"","payloadType":"date","repeat":"60","crontab":"","once":true,"x":260,"y":96,"wires":[["5d180fc7.9ad06","27e67f9b.4f9158"]]},{"id":"5d180fc7.9ad06","type":"function","z":"53f8b852.885c6","name":"Test Influx DB","func":"    const isPortAvailable = context.global.portavail;\n    msg.payload = {};\n     \n    var port = 8086;\n    \n    isPortAvailable(port).then( status =>{\n        if(status) {\n            //console.log('Port ' + port + ' IS available!');\n            msg.payload = {'InfluxDB':false,\"title\":\"InfluxDB\",\"color\":\"red\"};   // The port is available, hence the server is NOT running\n            node.send(msg);\n        } else {\n            //console.log('Port ' + port + ' IS NOT available!');\n            //console.log('Reason : ' + isPortAvailable.lastError);\n            msg.payload = {'InfluxDB':true,\"title\":\"InfluxDB\",\"color\":\"green\"};    // The port is not available, so the server MIGHT be running\n            node.send(msg);\n           \n        }\n    });\n    ","outputs":1,"noerr":0,"x":533.5,"y":97,"wires":[["3f3f8226.c9bfb6"]]},{"id":"3f3f8226.c9bfb6","type":"ui_template","z":"53f8b852.885c6","group":"44e5d7ea.043b2","name":"Status Icon","order":0,"width":0,"height":0,"format":"\n.dot {\n    height: 25px;\n    width: 25px;\n    background-color: #bbb;\n    border-radius: 50%;\n    display: inline-block;\n    float: right;\n}\n\n\n
{{msg.payload.title}}\n \n
","storeOutMessages":true,"fwdInMessages":true,"x":780,"y":96,"wires":[[]]},{"id":"27e67f9b.4f9158","type":"function","z":"53f8b852.885c6","name":"Test MongoDB","func":" const isPortAvailable = context.global.portavail;\n msg.payload = {};\n \n var port = 27017;\n \n isPortAvailable(port).then( status =>{\n if(status) {\n //console.log('Port ' + port + ' IS available!');\n msg.payload = {'MongoDB':false,\"title\":\"MongoDB\",\"color\":\"red\"}; // The port is available, hence the server is NOT running\n node.send(msg);\n } else {\n //console.log('Port ' + port + ' IS NOT available!');\n //console.log('Reason : ' + isPortAvailable.lastError);\n msg.payload = {'MongoDB':true,\"title\":\"MongoDB\",\"color\":\"green\"}; // The port is not available, so the server MIGHT be running\n node.send(msg);\n \n }\n });\n ","outputs":1,"noerr":0,"x":533,"y":158,"wires":[["2e85d9d.cc25126"]]},{"id":"2e85d9d.cc25126","type":"ui_template","z":"53f8b852.885c6","group":"44e5d7ea.043b2","name":"Status Icon","order":0,"width":0,"height":0,"format":"\n.dot {\n height: 25px;\n width: 25px;\n background-color: #bbb;\n border-radius: 50%;\n display: inline-block;\n float: right;\n}\n\n\n
{{msg.payload.title}}\n \n
","storeOutMessages":true,"fwdInMessages":true,"x":781,"y":161,"wires":[[]]},{"id":"44e5d7ea.043b2","type":"ui_group","z":"","name":"System Status","tab":"7011ff77.15cb18","disp":true,"width":"6"},{"id":"7011ff77.15cb18","type":"ui_tab","z":"","name":"Home","icon":"dashboard"}]

The result:

The above flow and Node UI status indicator template should produce the following result:

NR UI Status Indicator
Node-Red UI Status Indicator

Using Node-Red and Grafana WorldMap for geolocalized data visualization

Based on my previous posts we are now able to build a system that can receive, store and visualize data by using Node-Red, InfluxDB and Graphana. Grafana allows us build dashboards, query and visualize the stored data across time efficiently by using, in our case, the InfluxDB database engine. So far we’ve used simple line/bar charts to visualize data but we can use both Node-Red and Grafana to plot data onto a map:

  1. NodeRed Contrib World Map: Openstreet UI based map for plotting data with several options, including icon types, vectors, circles and heatmaps totally controlled through nodered flows.
  2. Grafana WorldMap plugin: Grafana panel with also an OpenStreet map for visualizing data.

Both have pros and cons, but the main differences between the two is that Node-Red Worldmap is suited more to real time display, and the Grafana plugin is better adapted to display data based on some time based query. Other major difference is that Node-Red Worldmap would require some coding, but, at least I consider it, at an easy level, and the Grafana plugin is much harder to make it work.

Mapping data using Node-Red Worldmap:
One of the easiest ways for mapping data in real time is using Node Red Worldmap node. The map is plotted and updated in real time.

cd ~
cd .node-red
npm install node-red-contrib-web-worldmap

After restarting and deploying a worldmap node, the map should be available at: http://server:1880/worldmap or other URL depending on the Node-Red base configuration.

One thing to keep in mind is that Node-Red is single user, so all instances of world maps (several different clients/browsers) will always have the same view.
The simplest way to start using the worldmap is just to copy and deploy the demo workflow provided by the node, but the key concept is that each point has a name and a set of coordinates.

msg.payload = {};
msg.payload.name = "CentralLX";
msg.payload.lat = 38.7223;
msg.payload.lon = -9.1393;
msg.payload.layer = "SensorData";
msg.payload.UVLevel = getUVLevel() ;
msg.payload.Temperature = getTemp();

The cool thing is that if we inject repeatedly the above message (keeping the same name) but with different coordinates, the data point will move across the map in real time, and as I said earlier, the move will be reflect onto every client.

So all we need is an Inject node to a function node with the above code and feed it to the world map:

At the end we get this at the URL http://server:1880/worldmap:

Mapping data using Grafana Worldmap plugin:

The Grafana Worldmap plugin can get the location data in several ways. One of them is to use geohash data that is associated to the values/measurements.
There is a Node Red Geohash node that generates the geohash value from the latitude and longitude of data location. As usual we install the node:

cd ~
cd .node-red
npm install node-red-node-geohash

and then the Grafana plugin. We just follow the plugin instructions:

cd ~
grafana-cli plugins install grafana-worldmap-panel
/etc/init.d/grafana-server restart

With this per-requisites installed we can now feed data onto the database, in our case InfluxDB, that will be used by Grafana. We just need make sure that we add the geohash field. The geohash node will calculate from the node-red message properties lat – latitude and lon – longitude the required info:

A simple example:

Using the Influx tool, we can query our database and see that the geohash localization is now set:

> select * from SensorData limit 2
name: DemoValue
time                Temp UVLevel geohash    lat       lon        
----                ---- ------- -------   ---------- ----------  
1490706639630929463 22   8       eyckpjywh 38.7045055 -9.1754669  
1490706651606553077 21   7       eyckpjzjr 38.7044008 -9.1746488 

Anyway for setting up the World map plugin to display the above data was not straight forward, so the following instructions are more for a startup point rather than a solution.

The first thing to know is that the plugin is waiting for two fields: geohash and metric. With this in mind, before wasting too much time with the map plugin, a table panel that is filled with the required query is a precious tool to debug the query:

After we infer from the table that the data is more or less the data we want, we just transfer the query to WorldMap plugin:

Notice two important things: The aliasing for the query field to metric with the alias(metric) instruction, and the Format as: Table.

We can now setup the specific Worldmap settings:

On the Map Visual Options , I’ve centered the map in my location and set the zoom level. Fiddling around here can be seen in real time.

On the Map Data Options for this specific example, the Location Data comes from a table filled with the previous query (hence the format as table on the query output), and we want to see the current values with no aggregation.

When hoovering around a spot plotted on the map we can see a label: value, and the label used is obtained from a table field. In my case I just used geohash (not really useful…). Anyway these changes only work after saving and reloading the panel with F5 in my experience.

At the end we have now graphed data and localized data:

If we drag the selector on the left graphic panel, or select another time interval on top right menu of the grafana dashboard, the visualized information on the map changes.

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 3: Single point of access – Reverse proxy the services with nginx

Since we will be running a lot of services, each running on its own port, the following configuration, is optional, but allows to access all services through the same entry point by using Nginx server as a reverse proxy to Node-Red, Node-Red UI/Dashboard, Node-Red Worldmap and Grafana.

With this configuration the base URL is always the same without any appended ports, and the only thing that changes are the URL path:

http://server/nodered
http://server/nodered/worldmap
http://server/grafana

To allow this we install and configure Nginx:

apt-get install nginx

The configuration files will reside in /etc/nginx directory. Under that directory there are two directories: sites-available and sites-enable where the later normally contains a link to configuration files located at sites-available.
At that directory there is a file named default that defines the default web site configuration used by Nginx. This is the file where we will add the reverse proxy directives.

Reverse proxy for Node-Red and Node-Red Contrib Worldmap
For setting up the reverse proxy for Node-Red we must first change the base URL for Node Red from / (root) to something else that we can map the reverse proxy.

For this we will need to edit the settings.js file located on the .node-red directory on the home path of the user running Node-Red.

We need to uncomment and change the entry httpRoot to point to our new base URL.

   httpRoot: '/nodered',

Don’t forget the trailing comma.

We need to restart now Node-Red and it should be accessible at the URL http://server:1880/nodered instead of http://server:1880/.

To configure Nginx, we edit the file default at /etc/nginx/sites-available and add the following section:


        location  /nodered {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_pass "http://127.0.0.1:1880";
        }

        location /socket.io {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_pass "http://127.0.0.1:1880";

        }

Note the following: The first location defines the reverse proxy URL /nodered to be served by the backend server http://127.0.0.1:1880. The incoming path, /nodered, will be passed to the backend server URL /nodered, since paths are passed directly. No need to add the /nodered path to the backend server definition.
Also I’m using the 127.0.0.1 address instead of localhost to avoid the IPv6 mapping to the localhost. In this way I’m sure that IPv4 will be used.

The location mapping for /nodered will make all the functionality of node red to work as it should at the base url /nodered. But some nodes, like node-red-contrib-worldmap will request to the proxy server ignoring the node-red base root map. Hence the /socket.io mapping. It will allow the worldmap nodes to work, but will stop this mapping to be used for something else.

Reverse proxy for Grafana

Setting up the reverse proxy for Grafana we can, and should use the following documentation: Grafana Reverse Poxy. For me the following configuration worked:

First edit the [server] section on the Grafana configuration file grafana.ini located at /etc/grafana.

Uncomment and edit the following lines:

[server]
# Protocol (http or https)
protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = server.domain.com

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = http://server.domain.com/grafana/

Note the ending slash at the root_url. The same applies to the Nginx configuration

The files for the Nginx configuration are the same as the above configuration for reverse proxy.

We just need to add the following section after the previous location directives:

        location /grafana/ {
                proxy_pass http://localhost:3000/;
        }

We should now restart nginx to refresh the configuration, and all should be working as it should by accessing the Grafana dashboard at http://server.domain.com/grafana

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 2: Database configuration and data collection

On the previous post we’ve installed and the base software for our Grafana based dash board.

We need now to configure our InfluxDB database and Node Red to start collecting data.

Configuring InfluxDB:
Detailed instructions for configuring an InfluxDB database are on this InfluxDB documentation link..

The main concepts that we need to be aware when using the InfluxDB is that record of data has a time stamp, a set of tags and a measured value. This allows, for example to create a value named Temperature and tag it depending on the source sensor:

Temperature: Value=22.1 , Sensor=Kitchen
Temperature: Value=21.9 , Sensor=Room1

This allows to process all the data or only process data based on a certain tag or tags. Values and tags can be created on the fly without previously define them, which is a bit different from standard RDBMS engines.

Creating an InfluxDB database:
To create the database, we need to access the machine hosting the InfluxDB server and execute the command influx:

odroid@odroid:~$ influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> create database SensorData
> show databases
name: databases
name
----
_internal
SensorData

> 

Now we have our database created and I’ve named SensorData. To make an example with the above temperature data we can do the following:

> insert Temperature,Sensor=kitchen value=22.1
ERR: {"error":"database is required"}

Note: error may be due to not setting a database or retention policy.
Please set a database with the command "use " or
INSERT INTO . 
> use SensorData
Using database SensorData
> 

As we can see we need first to select the database where we are going to insert data with the command use SensorData:

> use SensorData
Using database SensorData
> insert Temperature, Sensor=kitchen value=22.1
ERR: {"error":"unable to parse 'Temperature, Sensor=kitchen value=22.1': missing tag key"}

> insert Temperature,Sensor=kitchen value=22.1
> insert Temperature,Sensor=Room1 value=21.9
> select * from Temperature
name: Temperature
time                Sensor  value
----                ------  -----
1487939008959909164 kitchen 22.1
1487939056354678353 Room1   21.9

Note that we can’t use spaces between the Measure name and the tags. The correct syntax is as follows:

 insert MeasureName,tag1=t1,tag2=t2,...   value1=val1,value2=val2,value3=val3,....

Also note that no DDL (data definition language) was used to create the tags or the measured value, we’ve just inserted data for our measurement with the our tags and value(s) without the need of previously define the schema.

Configuring Node-Red
Since we now have a database we can configure the InfluxDB Node Red nodes to store data onto the database:

There are two types of InfluxDB nodes, one that has an Input and Output and other that only has Input. The former is for making queries to the database where we provide on the input node the query, and on the output the results are returned. The later is for storing data only onto the database.
For both nodes we need to configure an InfluxDB server:

InfluxDB Server Configuration

We need to press the Pen icon right next to the server to add or reconfigure a new InfluxDB server:

InfluxDB server

A set of credentials are required, but since I’ve yet configured security, we can just put admin/admin as username and password. In a real deployment we must activate security.

From now on it is rather simple. Referring to InfluxDB node configuration screenshot (Not the InfluxDB server configuration) we have a configuration field named Measurement. This is our measure name that we associate a value. Picking up on the above example with the Insert command it will be Temperature, for example.

Now if the msg.payload provided has input to the node is a single value, let’s say 21, this is equivalent to do:

Insert Temperature value=12

We other formats for msg.payload that allows to associate tags and measures. Just check the Info tab for the node.

Simple example:

The following flow shows a simple example of a value received through MQTT, in this case the free heap from one of my ESP8266 and its storage in InfluxDB:

Sample Flow

[{"id":"20bec5de.8881c2","type":"mqtt in","z":"ced40abb.3c92e","name":"Heap","topic":"/outbox/ESP12DASH/Heap","qos":"2","broker":"2a552b3c.de8d2c","x":83.16668701171875,"y":206.41668701171875,"wires":[["e0d9c912.8c57f8","876fb151.6f2fa"]]},{"id":"876fb151.6f2fa","type":"debug","z":"ced40abb.3c92e","name":"","active":true,"console":"false","complete":"false","x":408.5,"y":177,"wires":[]},{"id":"e0d9c912.8c57f8","type":"influxdb out","z":"ced40abb.3c92e","influxdb":"bbd62a93.1a7108","name":"","measurement":"heap","x":446.1666717529297,"y":224.58335876464844,"wires":[]},{"id":"2a552b3c.de8d2c","type":"mqtt-broker","broker":"192.168.1.17","port":"1883","clientid":"node-red","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":15,"cleansession":true,"willQos":"0","birthQos":"0"},{"id":"bbd62a93.1a7108","type":"influxdb","z":"","hostname":"127.0.0.1","port":"8086","protocol":"http","database":"SensorData","name":"ODroid InfluxDB"}]

We can see with this flow the data stored in InfluxDB:

> select * from heap;
name: heap
time                value
----                -----
1487946319638000000 41600
1487946440913000000 41600
1487946562206000000 41600
1487946683474000000 41600
1487946804751000000 41600
1487946926061000000 41600
1487947047309000000 41616
1487947168594000000 41600

Now we have data that we can graph with Grafana, subject of my next posts.

Node Red Dashboard and UPS Monitoring

Just a quick hack to use the Node Red dashboard to monitor some of the UPS values that is attached to My Synology NAS.

Gathering the data and feeding it to Node-Red
First I thought to do some sort of Python or NodeJS program to run the upsc command, process the output and feed it, through MQTT, to Node Red.
But since it seemed to me a bit of overkill to just process a text output, transform it to JSON and push it through MQTT by using a program, I decided that I’ll use some shell scripting, bash to be more explicit.

I’m running on my Odroid C1+ “server” all the necessary components, namely Node Red with the Dashboard UI module.

So on Odroid I also have the ups monitoring tools, and upsc outputs a text with the ups status:

odroid@odroid:~$ upsc ups@192.168.1.16
Init SSL without certificate database
battery.charge: 100
battery.charge.low: 10
...
input.transfer.high: 300
input.transfer.low: 140
input.voltage: 230.0
...
ups.load: 7
ups.mfr: American Power Conversion
...
ups.model: Back-UPS XS 700U  
...

So all we need now is to transform the above output from that text format to JSON and feed it to MQTT.
This means that we need to put between ” the parameter names and values, replace the : by , and also we need to replace the . on parameter names to _ so that in Node Red javascript we don’t have problems working with the parameter names.

Since I’m processing each line of the output, I’m using gawk/awk that allows some text processing. The awk program is as follow:

BEGIN {print "{"}
 {
   print lline  "\42" $1 "\42:\42"$2"\42"
 }
 {lline =", "}
END {print "}"}

This will at the beginning print the opening JSON bracket, then line by line the parameter name and value between ” and separated by : .
The lline variable at the first line is empty, so it prints nothing, but at the following lines it prints , which separates the JSON values.
We just need awk now to recognize parameters and values, and that is easy since they are separated by :

So if the above code is saved as procupsc.awk file, then the following command:

 upsc ups@192.168.1.16 2>/dev/null | awk -F: -f ~/upsmon/procupsc.awk |  sed 's/[.]/_/g'

Transforms the upsc output into a JSON output, including the replacement of . on variable names into _

{
"battery_charge":" 100"
, "battery_charge_low":" 10"
, "battery_charge_warning":" 50"
...
, "ups_load":" 7"
..
, "ups_vendorid":" xxxx"
}

Now all we need is to feed the output to the MQTT broker, and for this I’ll use the mosquitto_pub command, that has a switch that accepts the message from the standard input:

upsc ups@192.168.1.16 2>/dev/null | awk -F: -f /home/odroid/upsmon/procupsc.awk |  sed 's/[.]/_/g' | mosquitto_pub -h 192.168.1.17 -t upsmon -s

So we define the host and the topic: upsmon and the message is the output of the previous command (the -s switch).

All we need now is on Node Red to subscribe to the upsmon topic and process the received JSON object.

Since I’m running this periodically on crontab, I also add the PATH variable so that all files and commands are found.
The complete script is as follows:

upsmon.sh

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
upsc ups@192.168.1.16 2>/dev/null | awk -F: -f /home/odroid/upsmon/procupsc.awk |  sed 's/[.]/_/g' | mosquitto_pub -h 192.168.1.17 -t upsmon -s

and on Crontab:

# m h  dom mon dow   command
*/5 * * * * /home/odroid/upsmon/upsmon.sh

Node Red processing and visualization
On node red side, now is easy. We receive the above upsc JSON object as a string on msg.payload, and we use the JSON node to separate into different msg.# variables.
From here we just feed the data to charts and gauges. The code is:

[{"id":"603732e7.bb8464","type":"mqtt in","z":"a8b82890.09ca7","name":"","topic":"upsmon","qos":"2","broker":"2a552b3c.de8d2c","x":114.5,"y":91,"wires":[["193a550.2b0ea2b"]]},{"id":"f057bae4.8e4678","type":"debug","z":"a8b82890.09ca7","name":"","active":true,"console":"false","complete":"payload","x":630.5,"y":88,"wires":[]},{"id":"563ec2f5.6475e4","type":"ui_gauge","z":"a8b82890.09ca7","name":"UPS Load","group":"ba196e43.b35398","order":0,"width":0,"height":0,"gtype":"donut","title":"Load","label":"%","format":"{{value}}","min":0,"max":"100","colors":["#00b500","#e6e600","#ca3838"],"x":630.5,"y":173,"wires":[]},{"id":"dbdb6731.531e8","type":"function","z":"a8b82890.09ca7","name":"UPS_Load","func":"msg.payload = Number(msg.payload.ups_load);\nreturn msg;","outputs":1,"noerr":0,"x":372.5,"y":175,"wires":[["563ec2f5.6475e4","f057bae4.8e4678","57bc92ab.ce4234"]]},{"id":"193a550.2b0ea2b","type":"json","z":"a8b82890.09ca7","name":"To Json","x":129.5,"y":178,"wires":[["dbdb6731.531e8","5433b116.a93b4","52604271.71f43c","11ad64b4.f70ad3","589635b3.9bacc4"]]},{"id":"57bc92ab.ce4234","type":"ui_chart","z":"a8b82890.09ca7","name":"Ups Load/Time","group":"ba196e43.b35398","order":0,"width":0,"height":0,"label":"Load/Time","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","ymin":"0","ymax":"100","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"x":639.5,"y":226,"wires":[[],[]]},{"id":"5433b116.a93b4","type":"function","z":"a8b82890.09ca7","name":"Battery Status","func":"var Vbats = msg.payload.battery_voltage;\n\nvar Vbat = Vbats.replace(\"_\",\".\");\n\nmsg.payload = Number(Vbat);\n\nreturn msg;","outputs":1,"noerr":0,"x":400.5,"y":336,"wires":[["8c6f269f.87b618"]]},{"id":"52604271.71f43c","type":"function","z":"a8b82890.09ca7","name":"V IN","func":"var Vins = msg.payload.input_voltage;\n\nvar Vin = Vins.replace(\"_\",\".\");\n\nmsg.payload = Number(Vin);\nreturn msg;","outputs":1,"noerr":0,"x":369.5,"y":570,"wires":[["240f6e31.7aa5da","b37dffd2.1cb0d"]]},{"id":"8c6f269f.87b618","type":"ui_gauge","z":"a8b82890.09ca7","name":"","group":"421e19.192041e8","order":0,"width":0,"height":0,"gtype":"gage","title":"Curr. Bat. Voltage","label":"V. Bat","format":"{{value}}","min":"11","max":"15","colors":["#b50000","#e6e600","#00b500"],"x":616.5,"y":332,"wires":[]},{"id":"240f6e31.7aa5da","type":"ui_gauge","z":"a8b82890.09ca7","name":"Input AC Voltage","group":"4e74439e.ee7e74","order":0,"width":0,"height":0,"gtype":"gage","title":"VIN AC","label":"V AC","format":"{{value}}","min":"190","max":"240","colors":["#00b500","#e6e600","#ca3838"],"x":660.5,"y":569,"wires":[]},{"id":"11ad64b4.f70ad3","type":"function","z":"a8b82890.09ca7","name":"Bat Runtime","func":"msg.payload = Number(msg.payload.battery_runtime);\nreturn msg;","outputs":1,"noerr":0,"x":397.5,"y":397,"wires":[["98292081.f4f718","62bb1456.0f45ec"]]},{"id":"98292081.f4f718","type":"ui_gauge","z":"a8b82890.09ca7","name":"UPS Level","group":"fccb6f27.7691d8","order":0,"width":0,"height":0,"gtype":"wave","title":"UPS Runtime","label":"UPS Level","format":"{{value}}","min":0,"max":"1500","colors":["#00b500","#e6e600","#ca3838"],"x":642.5,"y":391,"wires":[]},{"id":"589635b3.9bacc4","type":"function","z":"a8b82890.09ca7","name":"Bat Charge","func":"var Vcharges = msg.payload.battery_charge;\n\nvar Vcharge = Vcharges.replace(\"_\",\".\");\n\nmsg.payload = Number(Vcharge);\n\n\nreturn msg;","outputs":1,"noerr":0,"x":405,"y":495,"wires":[["d2ab2543.d53a68"]]},{"id":"d2ab2543.d53a68","type":"ui_gauge","z":"a8b82890.09ca7","name":"Battery Charge","group":"421e19.192041e8","order":0,"width":0,"height":0,"gtype":"gage","title":"Battery Charge","label":"%","format":"{{value}}","min":0,"max":"100","colors":["#ff0000","#e6e600","#00ff01"],"x":661.5,"y":494,"wires":[]},{"id":"62bb1456.0f45ec","type":"ui_chart","z":"a8b82890.09ca7","name":"","group":"fccb6f27.7691d8","order":0,"width":0,"height":0,"label":"Runtime (sec)","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"x":625.5,"y":441,"wires":[[],[]]},{"id":"b37dffd2.1cb0d","type":"ui_chart","z":"a8b82890.09ca7","name":"Vin/Time","group":"4e74439e.ee7e74","order":0,"width":0,"height":0,"label":"VAC In/Time","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","ymin":"190","ymax":"240","removeOlder":"12","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"x":635.5,"y":641,"wires":[[],[]]},{"id":"2a552b3c.de8d2c","type":"mqtt-broker","broker":"192.168.1.17","port":"1883","clientid":"node-red","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":15,"cleansession":true,"willQos":"0","birthQos":"0"},{"id":"ba196e43.b35398","type":"ui_group","z":"","name":"UPS Load","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"421e19.192041e8","type":"ui_group","z":"","name":"UPS Battery","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"4e74439e.ee7e74","type":"ui_group","z":"","name":"Input Voltage","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"fccb6f27.7691d8","type":"ui_group","z":"","name":"UPS Runtime","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"61ec3881.53526","type":"ui_tab","z":"","name":"UPS","icon":"dashboard","order":2}]

The final output is as follow:

Node Red UPS Monitoring
Node Red UPS Monitoring

Upgrading NodeJs and Node Red on Odroid

I run many services on my Odroid C1+ including Node-Red. But since NodeJs on Odroid C1+ is version v0.10 is starting to be seriously old for running Node-Red or other NodeJS dependent software.

So my quick instructions for upgrading NodeJS and Node-Red on the Odroid C1+

Upgrading NodeJS

First verify what version is available/installed on the Odroid:

odroid@odroid:~$ node -v
v0.12.14
odroid@odroid:~$ nodejs -v
v0.10.25

Since I’ve already had previously installed a more recent version of NodeJS (the node command), the version used by Node-Red is v0.12.14 while the default NodeJS version is v0.10.25.

We can also, and should, check the npm version:

odroid@odroid:~$ npm -v
2.15.1

We also need to find what architecture we are using, just for completeness since ODroid C1+ is an ARM7 based architecture:

odroid@odroid:~$ uname -a
Linux odroid 3.10.96-151 #1 SMP PREEMPT Wed Jun 15 18:47:37 BRT 2016 armv7l armv7l armv7l GNU/Linux

This will allow us to download the correct version of the NodeJS binaries from the NodeJS site: NodeJS downloads.
In our case we choose the ARM7 architecture binaries, which at the current time is file: node-v6.9.2-linux-armv7l.tar.xz
So I’ve just copied the link from the NodeJS site and did a wget on the Odroid:

wget https://nodejs.org/dist/v6.9.2/node-v6.9.2-linux-armv7l.tar.xz

I then created a working directory and “untared” the file:

odroid@odroid:~$ mkdir nodework
odroid@odroid:~$ cd nodework
odroid@odroid:~/nodework$ tar xvf ../node-v6.9.2-linux-armv7l.tar.xz
odroid@odroid:~/nodework$ cd node-v6.9.2-linux-armv7l/
odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l$ 

Since there isn’t an install script we need to move the new NodeJS files to the correct locations:

  1. Binaries to /usr/bin
  2. Include files to /usr/include
  3. Libs files to /usr/lib

Copy the binaries, replacing, if existing the older versions:

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/bin
$ sudo cp -i node /usr/bin
cp: overwrite ‘/usr/bin/node’? y
odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/bin$ 

Copy the include files:

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/include
$ sudo cp -R node  /usr/include/

and copy the libraries

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/lib
$ sudo cp -R node_modules /usr/lib

and finally:

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/share
$ sudo cp -R . /usr/share

We need now to make npm to point to the correct nodejs script so, we need to delete the npm link at the /usr/bin and /usr/local/bin directories:

odroid@odroid:~$ sudo rm /usr/bin/npm
odroid@odroid:~$ sudo rm /usr/local/bin/npm

and re-create the correct links:

odroid@odroid:~$ sudo ln -s /usr/lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm
odroid@odroid:~$ sudo ln -s /usr/lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm

Running now the node and npm commands should report the latest versions:

odroid@odroid:~$ node -v
v6.9.2
odroid@odroid:~$ npm -v
3.10.9
odroid@odroid:~$ 

Success!

Upgrading Node-Red

From the Node-Red startup log, we can see the previous versions of node-red and nodejs used:

Welcome to Node-RED
===================

28 Dec 17:55:40 - [info] Node-RED version: v0.15.2
28 Dec 17:55:40 - [info] Node.js  version: v0.12.14
28 Dec 17:55:40 - [info] Linux 3.10.96-151 arm LE
28 Dec 17:55:42 - [info] Loading palette nodes
28 Dec 17:55:50 - [info] Dashboard version 2.1.0 started at /ui
28 Dec 17:55:54 - [warn] ------------------------------------------------------
28 Dec 17:55:54 - [warn] [rpi-gpio] Info : Ignoring Raspberry Pi specific node
28 Dec 17:55:54 - [warn] ------------------------------------------------------

we can upgrade now Node-Red according to the Node Red upgrading instructions:

odroid@odroid:~$ sudo npm cache clean
odroid@odroid:~$ sudo npm install -g --unsafe-perm node-red

and after a while the upgrade should be done.

Before starting up node-red I went to the node-red module directories, and did an update:

odroid@odroid:~/.node-red$ npm update
/home/odroid/.node-red
└── crypto-js@3.1.8 

Starting up Node-Red should show now the new software versions:

Welcome to Node-RED
===================

1 Jan 20:35:46 - [info] Node-RED version: v0.15.2
1 Jan 20:35:46 - [info] Node.js  version: v6.9.2
1 Jan 20:35:46 - [info] Linux 3.10.96-151 arm LE
1 Jan 20:35:47 - [info] Loading palette nodes
1 Jan 20:35:54 - [info] Dashboard version 2.2.1 started at /ui

Done!

ESP8266 – Logging data in a backend – AES and Crypto-JS

After building, on the previous posts, the Node-Red based backend to support E2EE (End to End Encryption) so we can log data into a central server/database, from our devices securely, without using HTTPS, we need now to build the firmware for the ESP8266 that allows it to call our E2EE backend.

The firmware for the ESP8266 must gather the data that it wants to send, get or generate the current sequence number for the data (to avoid replay attacks), encrypt the data and send it to the backend.
On the backend we are using the Java script library for cryptographic functions Crypto-js, and specifically we are encrypting data with the encryption algorithm AES. So all we need is to encrypt our data with AES on the ESP8266, send it to the Node-Red Crypto-js backend, decrypt it and store it, easy right?

Not quite, let’s see why:

Crypto-js and AES:
We can see that on my Node-Red function code and testing programs I’m using something similar to the following code example:

var CryptoJS = require("crypto-js");
var message  = "Message to encrypt";
var AESKey   = '2B7E151628AED2A6ABF7158809CF4F3C';

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text in Base64: " ,  ciphertext.toString(CryptoJS.enc.base64) );

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Several points regarding the above code need clarification:

The code variable AESKey the way it is used on the above example encrypt and decrypt functions isn’t really a key but a passphrase from where the real key and an initialization vector or salt value is generated (I’m using the names interchangeably, but they are not exactly the same thing except they are public viewable data that must change over time/calls).
The use for the generated key is self explanatory, but the initialization vector (IV) or salt value is used to allow that the encrypted data output to be not the same for the same initial message. While the key is kept constant and secret to both parties, the IV/salt changes between calls, which means that the above code, when run multiple times, will never produce the same output for the same initial message.

Still referring to the above code, the algorithm that generates the key from the passphrase is the PBKDF2 algorithm. More info at Crypto-js documentation. At the end of the encryption the output is a OpenSSL salted format that means that the output begins by the signature id: Salted_, followed by an eight byte salt value, and after the salt, the encrypted data.

So if we want use the API has above on the node-js/crypto-js side, we need to implement on the ESP8266 side both the AES and PBKDF2 algorithms.

I decided not to do that, first because finding a C/C++ implementation of the PBKDF2 algorithm that could be portable and worked on the ESP822 proved difficult, and second the work for porting it to the ESP8266 won’t be needed if I use a KEY/IV directly, and so I decided to use the more standard way of providing an AES key and an initialization vector for encrypting and decrypting data.

In the case of Node-JS and Crypto-JS when using an explicit key and IV the code looks like:

var CryptoJS = require("crypto-js");
var request = require('request');

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":145 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);

var iv = CryptoJS.enc.Hex.parse('0000000000000000');
var key= CryptoJS.enc.Hex.parse(AESKey);

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, key , { iv: iv } );

//console.log("Cypher: ", ciphertext );
console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");

console.log("=============================================================================");
console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Now, with above code, where the IV is always initialized to the same value, in this case ‘0000000000000000’, we can see when running the above code several times that the output is always the same since the IV is kept constant. Also the encrypted output is now just the raw encrypted data and not the Openssl format.

So to make the above code secure we must randomize the IV value for producing an output that is always different, even from several program runs when encrypting the same source data.

As a final note, if we count the number of HEX characters on the Key string, we find out that they are 16 bytes, which gives a total of 128 key bits. So the above example is using AES128 encryption, and with default Crypto-js block mode and padding algorithms which are CBC (Chain block mode) and pkcs7.

Interfacing Crypto-js and the ESP8266:
Since we are using AES for encrypting data and decrypting data, we need first to have an AES library for the ESP8266. The AES library that I’m using is this one Spaniakos AES library for Arduino and RPi. This library uses AES128, CBC and pkcs7 padding, so it ticks all boxes for compatibility with Crypto-js…

I just added the code from the above library to my Sming project and also added this Base64 library so that I can encode to and from Base64.

The only remaining issue was to securely generate a truly random initialization vector. And while at first I’ve used some available libraries to generate pseudo-random numbers to real random numbers, I’ve found out that the ESP8266 seems to have already a random number generator that is undocumented: Random number generator

So to generate a random IV value is as easy as:

uint8_t getrnd() {
    uint8_t really_random = *(volatile uint8_t *)0x3FF20E44;
    return really_random;
}

// Generate a random initialization vector
void gen_iv(byte  *iv) {
    for (int i = 0 ; i < N_BLOCK ; i++ ) {
        iv[i]= (byte) getrnd();
    }

So our ESP8266 code is as follow:

Global variables declarations:
The N_Block defines the encryption block size, that for AES128 is 16 bytes.

#include "AES.h"
#include "base64.h"

// The AES library object.
AES aes;

// Our AES key. Note that is the same that is used on the Node-Js side but as hex bytes.
byte key[] = { 0x2B, 0x7E, 0x15, 0x16, 0x28, 0xAE, 0xD2, 0xA6, 0xAB, 0xF7, 0x15, 0x88, 0x09, 0xCF, 0x4F, 0x3C };

// The unitialized Initialization vector
byte my_iv[N_BLOCK] = { 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};

// Our message to encrypt. Static for this example.
String msg = "{\"data\":{\"value\":300}, \"SEQN\":700 , \"msg\":\"IT WORKS!!\" }";


The example function ESP8266 Sming code is:

void testAES128()  {

    char b64data[200];
    byte cipher[1000];
    byte iv [N_BLOCK] ;
    
    Serial.println("Let's encrypt:");
    
    aes.set_key( key , sizeof(key));  // Get the globally defined key
    gen_iv( my_iv );                  // Generate a random IV
    
    // Print the IV
    base64_encode( b64data, (char *)my_iv, N_BLOCK);
    Serial.println(" IV b64: " + String(b64data));
       
    Serial.println(" Mensagem: " + msg );
 
    int b64len = base64_encode(b64data, (char *)msg.c_str(),msg.length());
    Serial.println (" Message in B64: " + String(b64data) );
    Serial.println (" The lenght is:  " + String(b64len) );
    
    // For sanity check purpose
    //base64_decode( decoded , b64data , b64len );
    //Serial.println("Decoded: " + String(decoded));
    
    // Encrypt! With AES128, our key and IV, CBC and pkcs7 padding    
    aes.do_aes_encrypt((byte *)b64data, b64len , cipher, key, 128, my_iv);
    
    Serial.println("Encryption done!");
    
    Serial.println("Cipher size: " + String(aes.get_size()));
    
    base64_encode(b64data, (char *)cipher, aes.get_size() );
    Serial.println ("Encrypted data in base64: " + String(b64data) );
      
    Serial.println("Done...");
}

When the above code/function is executed on the ESP8266 it outputs the following:

Let's encrypt:
 IV b64: cAFviaDMHejlteGn9/4eQQ==
 Mensagem: {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }
 Message in B64: eyJkYXRhIjp7InZhbHVlIjozMDB9LCAiU0VRTiI6NzAwICwgIm1zZyI6IklUIFdPUktTISIgfQ==
 The lenght is:  76
Encryption done!
Cipher size: 80
Encrypted data in base64: /1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=
Done...

And now we can feed the above Base64 IV and encrypted data to our decoding program in Node-Js using Crypto-JS:

var CryptoJS = require("crypto-js");
var request = require('request');

var esp8266_msg = '/1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=';
var esp8266_iv  = 'cAFviaDMHejlteGn9/4eQQ==';

// The AES encryption/decryption key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

var plain_iv =  new Buffer( esp8266_iv , 'base64').toString('hex');
var iv = CryptoJS.enc.Hex.parse( plain_iv );
var key= CryptoJS.enc.Hex.parse( AESKey );

console.log("Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt( esp8266_msg, key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Base64);
var decoded_b64msg =  new Buffer(plaintext , 'base64').toString('ascii');
var decoded_msg =     new Buffer( decoded_b64msg , 'base64').toString('ascii');

console.log("Decrypted message: ", decoded_msg);

and the output is:

Decrypted message:  {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }

So, as the message shows, it WORKS!. AES encryption on the ESP8266 to a Node-JS Crypto-JS based code where decryption occurs.

Final notes:
So all is needed now is to build on the ESP8266 side the message with the encrypted data and the IV, and send through plain HTTP a JSON object to the Node-Red back end.

{
  "msg":"/1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=",
  "iv":"cAFviaDMHejlteGn9/4eQQ=="
}

On the Node-Red back end the decryption can now be done easily as was shown on the above testing code.

Further reading:

More information also available at: ESP8266 and AES128 for end-to-end encryption and Establishing secure ESP8266 and NodeJs communication by using Diffie-Hellman key exchange and Elliptic Curves

ESP8266 – Logging data in a backend with end to end encryption – Storing Data

As we can see in my previous post, for securely store data from a device, we should make the data secure during transport and tamper proof. This can be achieved by using encryption, but the use of encryption only does not solve some attacks, like replay attacks.

So a scheme where data is encrypted and associated with a sequence number is used, to have transport security and replay attack protection, and on the previous post we’ve implemented the base framework that allows the support of our requirements.

So the device generates, for example, the following message:

 { "data":{"value": 300} , "SEQN": 120 }

meaning that it wants to send to the backend the JSON Object {“value”: 300}, and the current Sequence number is 120. On the next message the sequence number should be 121, but that is supposing that algorithm for the sequence number is to increase it one by one per message.

Giving the message, we encrypt it now with our device key, and build another JSON object, the one that will be sent to the Node-red based backend:

 { "msg":"U2FsdGVkX1..."}

Since we still don’t have our device ready to send messages, we will build a simple NodeJs program to generate the messages:

mkdir ~/gen
cd ~/gen
npm init   (Just accept the defaults)
npm install crypto-js --save
npm install request --save

And the code to generate our message is as follow:

var CryptoJS = require("crypto-js");
var request = require('request');

// API endpoint.
var apiEP = 'http://localhost:1880/data/';
var deviceID = '12FA';

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":121 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);
console.log("Message: " , message );

// Encode the string to base64. Not really needed.
message = new Buffer(message).toString("base64");
console.log("Message B64: " , message );

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");
console.log("Let's call the Node-Red API end point: ");

var URL = apiEP + deviceID;
var rawdata = ciphertext.toString(CryptoJS.enc.base64);

console.log(" Calling end point: " , URL);
console.log(" RawData: " , rawdata );

// Let's call the REST API end point.
request( {
    url: URL,
    method: "POST",
    json: true,
    body: { "msg": rawdata}
    } ,
    function (error, response, body) {
        if (!error && response.statusCode == 200) {
            console.log(" ");
            console.log("REST API Output: ");
            console.log(body)
        }
    }
);
console.log("=============================================================================");
console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);
console.log(" ");

console.log("Decrypted message: " , new Buffer(plaintext , 'base64').toString('ascii'));

We can modify the message by modifying the msgObjs variable. We run this program by executing node index.js and at the end the Node-Red answer should be displayed under REST API Output.

If everything is setup correctly the REST API should return:

REST API Output: 
{ status: 'OK' }

But calling a second time without modifying the sequence number should fail:

REST API Output: 
{ status: 'NOT OK' }

The Node-Red flow:

The Node-Red flow, receives the REST POST request for the device ID, obtains from the database the private key associated to the device, decrypts the payload, and checks the message sequence number vs the database sequence number. By design it allows the device to send messages above the current sequence number to allow message gaps (messages that where lost). We can, if needed, process this gaps so we can have an idea of how many messages that we are loosing.

If the decryption is successful and the sequence number is ok, we increment the sequence number to the next value, and store the data:

Node-Red Data Storage flow
Node-Red Data Storage flow

The code is as follow:

[{"id":"501864bd.eea2ec","type":"sqlitedb","z":"ee002ffe.ffd9e8","db":"/home/odroid/Databases/wsn.db"},{"id":"9e2ef.10b2b512","type":"http in","z":"ee002ffe.ffd9e8","name":"setSensorData","url":"/data/:id","method":"post","swaggerDoc":"","x":142.1666717529297,"y":812.0833587646484,"wires":[["fa205b8f.8de96"]]},{"id":"fa205b8f.8de96","type":"function","z":"ee002ffe.ffd9e8","name":"Build Query","func":"// Get the device id\nvar deviceId = msg.req.params.id;\n\n// Build the query. The SQLITE node requires the query in msg.topic\nmsg.topic=\"Select * from Devices where deviceID='\" + deviceId +\"'\";\n\n// Let's pass these parameters forward on its on variables:\nmsg.deviceid = deviceId;\nmsg.rawdata  = msg.payload;\n\nreturn msg;","outputs":1,"noerr":0,"x":199.1666717529297,"y":882.7499694824219,"wires":[["94004285.aa1418"]]},{"id":"94004285.aa1418","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Get Device AES Key","x":404.16668701171875,"y":817.0833435058594,"wires":[["7dd8f03c.ab2778"]]},{"id":"d0ed1bda.b15428","type":"http response","z":"ee002ffe.ffd9e8","name":"","x":1023.1666870117188,"y":956.8333435058594,"wires":[]},{"id":"7dd8f03c.ab2778","type":"function","z":"ee002ffe.ffd9e8","name":"Decrypt Request","func":"var cryptojs = context.global.cryptojs;\ntry {\n    // Get the key, Sequence number and the raw encrypted data\n    var AESKey = msg.payload[0].deviceKey;\n    msg.dbSQN = msg.payload[0].deviceSQN;   // Obtain also the currrent SQN on the database\n    \n    var rawdata= msg.rawdata.msg;\n    \n    node.log(  msg.devSQN );\n    // Decrypt the payload data with the device key. \n    // It returns a string sequence of bytes.\n    var bytes = cryptojs.AES.decrypt( rawdata, AESKey );\n    \n    // Convert bytes to an UTF8 plain string\n    var plaintext = bytes.toString(cryptojs.enc.Utf8);\n    node.log( plaintext );\n    // Convert from base64 to string\n    msg.payload  = new Buffer(plaintext , 'base64').toString('ascii');\n\n    return [ null , msg ];  // Exit the function at output 2.\n    \n} catch (err) {\n    msg.payload = { \"status\":\"NOT OK\"};\n    msg.statusCode = 500;     // Set internal server error\n    node.log(\"Invalid deviceID request for GET SQN Operation\");\n    return [ msg , null ];  // Exit the function at output 1\n}\nreturn msg;","outputs":"2","noerr":0,"x":668.1666870117188,"y":817.7499694824219,"wires":[["d0ed1bda.b15428"],["c7be8b09.893b9"]]},{"id":"c7be8b09.893b9","type":"function","z":"ee002ffe.ffd9e8","name":"Verify MSG SQN","func":"// At this point we should have:\n// msg.payload with the sequence number in JSON: { SEQN: 200} and the rest of the message\n// msg.deviceid with the device id\ntry {\n    var msgObj = JSON.parse(msg.payload);\n    var msgSeqn = msgObj.SEQN;   // The sequence number that the device as sent\n    var msgData = msgObj.data;   // The data sent\n  \n    if ( msg.dbSQN <= msgSeqn )  { // Valid Sequence number\n      node.log( \"Sequence is VALID!!!!!\");\n      msg.payload = { \"status\":\"OK\"};\n      msg.data = msgData;\n      msg.devSQN= msgSeqn;\n      return [ msg, msg];\n    } else {\n        node.log( \"Sequence is invalid!!!!\");\n        msg.payload = { \"status\":\"NOT OK\"};\n        return [ null , msg ];\n    }\n} catch( e ) {\n    node.log(\"Error verifying SEQN...\") \n    msg.payload = { \"status\":\"Internal Error\"};\n    return [ null , msg ];\n}","outputs":"2","noerr":0,"x":164.1666717529297,"y":996.7499694824219,"wires":[["c8ce0d1c.589058","533d8275.e04474"],["d0ed1bda.b15428"]]},{"id":"c8ce0d1c.589058","type":"function","z":"ee002ffe.ffd9e8","name":"Update Sequence ","func":"\n    var dbSQN = msg.devSQN + 1;   // It allows message gaps.\n\n    msg.topic=\"Update Devices Set deviceSQN= \" + dbSQN + \" Where deviceID='\" + msg.deviceid +\"'\";\n \n    //node.log(\"SQL: \" + msg.topic );\nreturn msg;","outputs":1,"noerr":0,"x":584.1666870117188,"y":1034.5833740234375,"wires":[["bef21dc4.4642a"]]},{"id":"bef21dc4.4642a","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Set SQN","x":780.1666870117188,"y":1034.4166259765625,"wires":[[]]},{"id":"533d8275.e04474","type":"function","z":"ee002ffe.ffd9e8","name":"Update Data","func":"  // Let's extract the data.\n  // This step should be modified as needed.\n  var data = msg.data.value;\n  msg.topic=\"Insert into Data Values ( '\" + msg.deviceid + \"', CURRENT_TIMESTAMP , \" + data + \")\";\n \n  //node.log (\"Data: \" + data );\nreturn msg;","outputs":1,"noerr":0,"x":579.1666870117188,"y":922.5833435058594,"wires":[["770abb44.6f54d4"]]},{"id":"770abb44.6f54d4","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Save Data","x":782.1666870117188,"y":922.4165954589844,"wires":[[]]}]

And so the only thing missing is the ESP8266 code to call and setting data using the encrypted transport. For implementing that we will be using the Sming framework that will use and call the above defined API on this post and previous.