ESP8266 – Logging data in a backend with end to end encryption – Storing Data

As we can see in my previous post, for securely store data from a device, we should make the data secure during transport and tamper proof. This can be achieved by using encryption, but the use of encryption only does not solve some attacks, like replay attacks.

So a scheme where data is encrypted and associated with a sequence number is used, to have transport security and replay attack protection, and on the previous post we’ve implemented the base framework that allows the support of our requirements.

So the device generates, for example, the following message:

 { "data":{"value": 300} , "SEQN": 120 }

meaning that it wants to send to the backend the JSON Object {“value”: 300}, and the current Sequence number is 120. On the next message the sequence number should be 121, but that is supposing that algorithm for the sequence number is to increase it one by one per message.

Giving the message, we encrypt it now with our device key, and build another JSON object, the one that will be sent to the Node-red based backend:

 { "msg":"U2FsdGVkX1..."}

Since we still don’t have our device ready to send messages, we will build a simple NodeJs program to generate the messages:

mkdir ~/gen
cd ~/gen
npm init   (Just accept the defaults)
npm install crypto-js --save
npm install request --save

And the code to generate our message is as follow:

var CryptoJS = require("crypto-js");
var request = require('request');

// API endpoint.
var apiEP = 'http://localhost:1880/data/';
var deviceID = '12FA';

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":121 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);
console.log("Message: " , message );

// Encode the string to base64. Not really needed.
message = new Buffer(message).toString("base64");
console.log("Message B64: " , message );

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");
console.log("Let's call the Node-Red API end point: ");

var URL = apiEP + deviceID;
var rawdata = ciphertext.toString(CryptoJS.enc.base64);

console.log(" Calling end point: " , URL);
console.log(" RawData: " , rawdata );

// Let's call the REST API end point.
request( {
    url: URL,
    method: "POST",
    json: true,
    body: { "msg": rawdata}
    } ,
    function (error, response, body) {
        if (!error && response.statusCode == 200) {
            console.log(" ");
            console.log("REST API Output: ");
            console.log(body)
        }
    }
);
console.log("=============================================================================");
console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);
console.log(" ");

console.log("Decrypted message: " , new Buffer(plaintext , 'base64').toString('ascii'));

We can modify the message by modifying the msgObjs variable. We run this program by executing node index.js and at the end the Node-Red answer should be displayed under REST API Output.

If everything is setup correctly the REST API should return:

REST API Output: 
{ status: 'OK' }

But calling a second time without modifying the sequence number should fail:

REST API Output: 
{ status: 'NOT OK' }

The Node-Red flow:

The Node-Red flow, receives the REST POST request for the device ID, obtains from the database the private key associated to the device, decrypts the payload, and checks the message sequence number vs the database sequence number. By design it allows the device to send messages above the current sequence number to allow message gaps (messages that where lost). We can, if needed, process this gaps so we can have an idea of how many messages that we are loosing.

If the decryption is successful and the sequence number is ok, we increment the sequence number to the next value, and store the data:

Node-Red Data Storage flow

Node-Red Data Storage flow

The code is as follow:

[{"id":"501864bd.eea2ec","type":"sqlitedb","z":"ee002ffe.ffd9e8","db":"/home/odroid/Databases/wsn.db"},{"id":"9e2ef.10b2b512","type":"http in","z":"ee002ffe.ffd9e8","name":"setSensorData","url":"/data/:id","method":"post","swaggerDoc":"","x":142.1666717529297,"y":812.0833587646484,"wires":[["fa205b8f.8de96"]]},{"id":"fa205b8f.8de96","type":"function","z":"ee002ffe.ffd9e8","name":"Build Query","func":"// Get the device id\nvar deviceId = msg.req.params.id;\n\n// Build the query. The SQLITE node requires the query in msg.topic\nmsg.topic=\"Select * from Devices where deviceID='\" + deviceId +\"'\";\n\n// Let's pass these parameters forward on its on variables:\nmsg.deviceid = deviceId;\nmsg.rawdata  = msg.payload;\n\nreturn msg;","outputs":1,"noerr":0,"x":199.1666717529297,"y":882.7499694824219,"wires":[["94004285.aa1418"]]},{"id":"94004285.aa1418","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Get Device AES Key","x":404.16668701171875,"y":817.0833435058594,"wires":[["7dd8f03c.ab2778"]]},{"id":"d0ed1bda.b15428","type":"http response","z":"ee002ffe.ffd9e8","name":"","x":1023.1666870117188,"y":956.8333435058594,"wires":[]},{"id":"7dd8f03c.ab2778","type":"function","z":"ee002ffe.ffd9e8","name":"Decrypt Request","func":"var cryptojs = context.global.cryptojs;\ntry {\n    // Get the key, Sequence number and the raw encrypted data\n    var AESKey = msg.payload[0].deviceKey;\n    msg.dbSQN = msg.payload[0].deviceSQN;   // Obtain also the currrent SQN on the database\n    \n    var rawdata= msg.rawdata.msg;\n    \n    node.log(  msg.devSQN );\n    // Decrypt the payload data with the device key. \n    // It returns a string sequence of bytes.\n    var bytes = cryptojs.AES.decrypt( rawdata, AESKey );\n    \n    // Convert bytes to an UTF8 plain string\n    var plaintext = bytes.toString(cryptojs.enc.Utf8);\n    node.log( plaintext );\n    // Convert from base64 to string\n    msg.payload  = new Buffer(plaintext , 'base64').toString('ascii');\n\n    return [ null , msg ];  // Exit the function at output 2.\n    \n} catch (err) {\n    msg.payload = { \"status\":\"NOT OK\"};\n    msg.statusCode = 500;     // Set internal server error\n    node.log(\"Invalid deviceID request for GET SQN Operation\");\n    return [ msg , null ];  // Exit the function at output 1\n}\nreturn msg;","outputs":"2","noerr":0,"x":668.1666870117188,"y":817.7499694824219,"wires":[["d0ed1bda.b15428"],["c7be8b09.893b9"]]},{"id":"c7be8b09.893b9","type":"function","z":"ee002ffe.ffd9e8","name":"Verify MSG SQN","func":"// At this point we should have:\n// msg.payload with the sequence number in JSON: { SEQN: 200} and the rest of the message\n// msg.deviceid with the device id\ntry {\n    var msgObj = JSON.parse(msg.payload);\n    var msgSeqn = msgObj.SEQN;   // The sequence number that the device as sent\n    var msgData = msgObj.data;   // The data sent\n  \n    if ( msg.dbSQN <= msgSeqn )  { // Valid Sequence number\n      node.log( \"Sequence is VALID!!!!!\");\n      msg.payload = { \"status\":\"OK\"};\n      msg.data = msgData;\n      msg.devSQN= msgSeqn;\n      return [ msg, msg];\n    } else {\n        node.log( \"Sequence is invalid!!!!\");\n        msg.payload = { \"status\":\"NOT OK\"};\n        return [ null , msg ];\n    }\n} catch( e ) {\n    node.log(\"Error verifying SEQN...\") \n    msg.payload = { \"status\":\"Internal Error\"};\n    return [ null , msg ];\n}","outputs":"2","noerr":0,"x":164.1666717529297,"y":996.7499694824219,"wires":[["c8ce0d1c.589058","533d8275.e04474"],["d0ed1bda.b15428"]]},{"id":"c8ce0d1c.589058","type":"function","z":"ee002ffe.ffd9e8","name":"Update Sequence ","func":"\n    var dbSQN = msg.devSQN + 1;   // It allows message gaps.\n\n    msg.topic=\"Update Devices Set deviceSQN= \" + dbSQN + \" Where deviceID='\" + msg.deviceid +\"'\";\n \n    //node.log(\"SQL: \" + msg.topic );\nreturn msg;","outputs":1,"noerr":0,"x":584.1666870117188,"y":1034.5833740234375,"wires":[["bef21dc4.4642a"]]},{"id":"bef21dc4.4642a","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Set SQN","x":780.1666870117188,"y":1034.4166259765625,"wires":[[]]},{"id":"533d8275.e04474","type":"function","z":"ee002ffe.ffd9e8","name":"Update Data","func":"  // Let's extract the data.\n  // This step should be modified as needed.\n  var data = msg.data.value;\n  msg.topic=\"Insert into Data Values ( '\" + msg.deviceid + \"', CURRENT_TIMESTAMP , \" + data + \")\";\n \n  //node.log (\"Data: \" + data );\nreturn msg;","outputs":1,"noerr":0,"x":579.1666870117188,"y":922.5833435058594,"wires":[["770abb44.6f54d4"]]},{"id":"770abb44.6f54d4","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Save Data","x":782.1666870117188,"y":922.4165954589844,"wires":[[]]}]

And so the only thing missing is the ESP8266 code to call and setting data using the encrypted transport. For implementing that we will be using the Sming framework that will use and call the above defined API on this post and previous.

OrangePi, rtl_tcp and power line communications.

The initial reason for buying an Orange Pi PC was to remotely used it as a server for the SDR dongle and run the rtl_tcp program. In my desktop PC, running Arch Linux and using Gqrx, I can connect remotely and avoid the massive interferences produced by my desktop.

Direct connection through ethernet cabling and 1GB switch showed that the Orange PI PC + RTL SDR + rtl_sdr, after some tweaking, would work just fine, with no annoying lag between commands and response to/from the rtl_tcp program when using the Gqrx.

The issue is/was that the location for my remote SDR would mean that the connection would be made through PLC: Power Line communications. PLC uses the power lines distributing electricity around the house as the medium to connect devices without using ethernet cables or wifi.

I had doubts that with my Devolo 200MBps PLC that would provide stable connectivity for the I/Q stream from rtl_tcp, but I was wrong. Using the rtl_tcp through the 200MBps PLC works like a charm, and I only have a slight lag (around 1s) between commands and response.

The problem with PLC is that them by themselves are also huge interference generators, and so I still have some noise issues, but nothing compared to the rtl sdr being near my computer. The 868Mhz ISM band is now pretty much clear and we can see all the devices communicating.

Orange Pi PC, Armbian and SDR

A few weeks ago I’ve bought an SDR RTL2832U+R820T2 dongle to do some tests with Software Defined radio. Despite of being able to catch some signals with the provided antenna, I have a huge interference problem originated from my desktop PC. Using the SDR dongle connected to my laptop and with the desktop PC off, most part of those interferences disappear. Still reception was poor due to the antenna quality and my office location.

So one solution for the above issues is to remotely put the SDR dongle in a better location and with a better antenna and connect the SDR software running on my desktop PC to this remote SDR dongle by using the rtl_tcp program.

Since I didn’t want to shell out a lot of money again for a RPi or Odroid C1/C2, I’ve decided to by an Orange Pi PC from Aliexpress for about 16.5€, postage included.

The Orange PI PC
The Orange PI PC is a small form computer like the Raspberry Pi and Odroid C1/C2 for example. It has a quad-core Allwinner processor that runs at 1.2GHz (under Armbian), with 1GB of memory, 3 USB2 ports and HDMI.
The Orange Pi provided operating system seems to overclock the CPU to 1.6GHz which brings a lot of stability and heat issues to the device.
The Armbian version seems to not suffer from such issues and works out of the box, including HDMI video output and apparently video acceleration (Haven’t tested it yet).
I’ve also bought, separately the acrylic box and a 2A 5V charger with the correct plug to connect to the Orange PI. The Orange PI, box and power supply took less than 3 weeks to arrive.

Starting up
The Orange PI needs an small form micro SD card to have the operating system installed and space for the file system. The recommend cards are Sandisk UHS-1 or Samsung UHS-1, but I’ve bought a Toshiba micro SD Exceria UHS-1 card, that works fine.
The card comes with a standard SD card adapter to be used when connected to a card reader.

After copying the Armbian operating system to the card on my desktop computer, and putting it the micro SD card slot on the Orange PI, I’ve just connected the network, HDMI and power.

The initial power up sequence can take several minutes, since it will expand the file-system on the SD card and probably sets up other things.

At the end there was a RED led steadily lit and a blinking GREEN led, with the Armbian desktop on my monitor.

Setting up
The following steps can be done through the desktop environment or through ssh. I’ve done all these steps through ssh:

– Change the root password from the default 1234 to a secure password.
– Change the Time Zone to my time zone: dpkg-reconfigure tzdata
– Change the hostname: vi /etc/hostname
– Add a working user: adduser opi
– Add the opi user to the sudo group: usermod -aG sudo opi
– Change the password of the opi user: passwd opi
– Move the network from DHCP to a fixed IP: vi /etc/network/interfaces

# Wired adapter #1
auto eth0
#iface eth0 inet dhcp
iface eth0 inet static
address 192.168.1.5
netmask 255.255.255.0
gateway 192.168.1.254

– Update the software: apt-get update and apt-get upgrade
– Install the rtl sdr software: apt-get install rtl-sdr gqrx-sdr librtlsdr-dev libusb-1.0-0-dev
– Blacklist the dvb modules so that the RTL software can load: vi /etc/modprobe.d/rtl-sdr-blacklist.conf

# This system has librtlsdr0 installed in order to
# use digital video broadcast receivers as generic
# software defined radios.
blacklist dvb_usb
blacklist dvb_core
blacklist dvb_usb_rtl2832u
blacklist dvb_usb_rtl28xxu
blacklist e4000
blacklist rtl2832

And finally we can reboot and connect our RTL SDR dongle.

Some final notes:
The SD card speed with the Toshiba Exceria card is:

hdparm -Tt /dev/mmcblk0p1

/dev/mmcblk0p1:
 Timing cached reads:   856 MB in  2.00 seconds = 427.67 MB/sec
 Timing buffered disk reads:  58 MB in  3.03 seconds =  19.12 MB/sec

About 40% slower than my Odroid emmc disk.

To allow the remote Gqrx SDR software to connect we need to run:

 rtl_tcp -a 192.168.1.5

And configure Gqrx as following:

Gqrx Device Config

and start using GQRX.

The Orange Pi CPU and temperatures with rtl_tcp running and connected never rouse above 7/8% CPU and 43ºC, so it looks good!

Gqrx tunning
Just one final note regarding GQRX:

On the Orange Pi, the rtl_tcp program was outputting a lot of ll+:### where ### is an increasing number, and I had several seconds of lag between the change of frequency on the Gqrx aplication.

After checking the rtl_tcp code source, these messages are related to buffering issues, so I’ve changed the Gqrx connection string to:

rtl_tcp=192.168.1.5:1234,buffers=384,psize=65536

and all ll+ messages where eliminated never rising above 5.

ESP8266 – Logging data in a backend with end to end encryption

One off my posts is how to log data that is sent from an ESP8266 device into a MySql database: Logging data on a MySQL database by using the Nodemcu firmware and some PHP on the server side. Keep in mind that post/solution is/was just a quick hack to move data from the ESP8266 into a database. There are better solutions for implementing a process that receives data and stores data from the ESP8266, or any other device, into a database.

Why we should improve the “hack”? Well there is no security or authentication on that example. If deployed in a real world scenario, someone could just inject random value data into MySQL or do some other nasty things, like SQL Injection.

So the idea of this post is to do something more generic and more robust than the original post, namely build a server backend that can satisfy these following points:

  1. Authentication – Only allow authorized devices to log data into a database
  2. E2EE: End to End Encryption – Secure data while in transit from the device to the final storage
  3. MySQL database is hardcoded: – The original example is hardcoded to MySQL. Let’s do something more generic, namely something that can store data on other databases.

One natural candidate for building this generic backend is IBM Node Red. Node Red would allows us to keep developments to a minimum and still have great flexibility.

Options to connect to the backend:
There are several options to get data from the devices into a backend database server securely. Let’s see some of them:

Implement a native database driver on the device:
Ok, this crosses the mind of several people as a possible solution. But we are talking on implementing a database driver on devices that have constrained resources (memory, CPU, connectivity!). It also means that the database must be exposed directly on the network so that the devices can access it directly. Also, in most of the database drivers, the database connection needs to have a constant network connection. So this solution might make sense if the device is an RPi or some other larger/powerful device with a permanent network connection, but it’s not a good idea for a device like an ESP8266 (Not to mention that it might be just plain impossible to port the database driver).

HTTP based interface: REST, SOAP, plain HTTP:
This is far easier to implement since it uses plain HTTP (as the my original example from the ESP8266 to MySQL example), needs no permanent connection to the server, and we can implement authentication and E2EE either by configuring the backend server to use HTTPS or by encrypting data by ourselves. Still HTTPS could be difficult to use, but HTTP is almost universal. So HTTP(S) call to a REST/SOAP server is a good candidate.

MQTT data publishing:
MQTT is also a good candidate but it won’t work well for devices that have intermittent connectivity since MQTT needs a permanent TCP/IP connection to be available from the device to the MQTT broker. MQTT recognizes this limitation, and there is MQTT-SN, for sensor networks, that doesn’t demand a TCP/IP connection.
Authentication and E2EE can be implemented at the protocol level, namely by using authentication on the broker and accessing the broker with TLS/SSL. Also MQTT is not the only alternative, COAP and AMQP are also valid solutions.

So using a native database driver on the device is out of question and so far it seems that the HTTP, MQTT(-SN) and/or COAP protocols are the best way to transfer data from the devices to the backend. Still even implementing these protocols on the end devices might not be good enough for very constrained devices, but that is another issue…

Anyway HTTP and MQTT support, is as of today, pretty standard on all the available firmwares for the ESP8266. But using HTTPS and MQTT over SSL support might still not possible to use1.

So on this post I’ll just use plain REST over HTTP with the payload encrypted with AES.
This means that before we send our data, we will encrypt it with a key (our device key) and send it to the server. The server also holds our device key, and decrypts the data.

The basics:

Since we want to encrypt data during transmission and then decoding at the other, we need to setup a secure channel between our source and our destination database. For the sake of an example I’ll just AES encrypt the source data and decrypt it at the destination. Encryption will solve the possibility of someone eavesdropping the data but it won’t protect from replay attacks, which means someone can grab the full encrypted data and try to flood our server with N repetitions of the same data.

To solve this latest issue, we will add a sequence number to each request before is encrypted and at the receive side we check if the sequence number is not repeated. The sequence number will also allow us to see if we have lost any messages.

So we will use just a plain private shared key, without any kind of key management, or any kind of Public key based solution, to setup our encrypted channel.

Setting up Node-Red:

For setting up Node-red we need to do the following:

– Install the Sqlite node for accessing Sqlite databases:

cd ~/.node-red
npm install node-red-node-sqlite

Install the CryptoJS libraries to allow the use of AES crypto functions in our workflows.

Restart Node-Red so the new Sqlite nodes appear at the nodes pallet.

Implementing the backend server side:
Our backend server based on Node-red should now allow to do the following:

– Decrypt the receiving data sent by the devices.
– Check if the sequence number for the sending device is valid. If not reject the request.
– Store the received data on the database.

The requesting device at startup, since it might not have any kind of permanent storage, might not know what is/was the current/correct sequence number. There are at least three solutions for this:

  1. Case 1: The device requests what is the current sequence number for itself at startup.
  2. Case 2: The device starts at zero, again, but informs the backend that it has reset the sequence to zero
  3. Case 3: The device starts at a random number and informs the backend what is now the current sequence number.

As we can see Case 2 is just a special case of Case 3. We just need to make sure that:

  1. On the first case the backend response with the current sequence number is in fact from a trusted backend
  2. On the other cases the backend must validate that the device is who claims to be and sets the new sequence number.

Also, as for the orinial problem of setting data with protection from replay attacks, Case 2 and Case 3, suffer from the same issue. We also should also protect this calls from replay attacks, but for the sake of simplicity, we not going to do that on this cases.

For implementing the solution we need two tables: One for the device id-> device key association and current sequence number, and the table for storing data. In this last case let’s just suppose it is a simple value.

TABLE Devices
      deviceId: String,
      deviceKey: String,
      deviceSQN: integer

TABLE Data
      deviceId: String,
      dataDate: date
      dataValue: integer

We can create both tables on MySQL/PostGres/SQLITE, or for a real example, create the Devices table on a relational standard RDBMS as MySQL/SQLITE, and the Data “table” available on a timeseries database like InfluxDB.

We going to keep everything, for now on SQLITE, since we going to use the Node-Red Sqlite node:

Creating the database:
On the machine running Node-Red, create the database and tables:

cd ~
mkdir Databases
cd Databases

We create now the database and tables:

pcortex@pcortex:~/Databases$ sqlite3 wsn.db 
SQLite version 3.8.2 
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .databases
seq  name             file                                                      
---  ---------------  ----------------------------------------------------------
0    main             /home/pcortex/Databases/wsn.db                             

And now we create the tables:

sqlite> create table if not exists Devices ( deviceID String, deviceKey String, deviceSQN INTEGER);
sqlite> create table if not exists Data ( deviceID String, dataDate Date, dataValue Integer);
sqlite> .tables
Data     Devices
sqlite> 

We can now just insert some dummy data into the device table:

sqlite> insert into Devices values( '12FA', '2B7E151628AED2A6ABF7158809CF4F3C', 0);
sqlite> select * from Devices;
12FA|2B7E151628AED2A6ABF7158809CF4F3C|0
sqlite> 

The process of filling up this tableis another story in itself, so for the purpose of this post we just insert the device id, in this case 12FA, an AES key and the sequence number 0.

Retrieving the Sequence Number (Case 1):

Our device will retrieve the current sequence number using a REST Api GET operation by calling our backend, running on Node-Red. The request for retrieving the sequence number will return an encrypted JSON object as follow:

{"SEQN": 20 }

Where in this case, 20 is the current sequence number stored in the database.

The REST API is built as follow:

Node RED REST API

Node RED REST API

The HTTP input node will listen to the URL /wsn/:id where id is the device ID for the device that we want to obtain the sequence number. We then, based on the device ID, query the SQLITE Devices table, obtain the current sequence number, encrypt it with the device key and send the response to the requesting device. The requesting party using the device key, decrypts the answer and obtains the sequence number.

[{"id":"501864bd.eea2ec","type":"sqlitedb","z":"ee002ffe.ffd9e8","db":"/home/odroid/Databases/wsn.db"},{"id":"27489da4.ff5382","type":"http in","z":"ee002ffe.ffd9e8","name":"getSensorSEQN","url":"/wsn/:id","method":"get","swaggerDoc":"","x":169.1666717529297,"y":96.0833511352539,"wires":[["e9c38500.a29dc"]]},{"id":"313d3819.34686","type":"http response","z":"ee002ffe.ffd9e8","name":"","x":801.1666870117188,"y":92.00000762939453,"wires":[]},{"id":"aec88434.e67e98","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Get Device SEQN","x":384.16668701171875,"y":232.75,"wires":[["108630b.69050cf","33e20204.5cbc66"]]},{"id":"108630b.69050cf","type":"function","z":"ee002ffe.ffd9e8","name":"Encrypt response","func":"// Import the global Crypto-js module defined on Node-Red settings.js file\nvar cryptojs = context.global.cryptojs;\ntry {\n    // Get the key and the sequence number.\n    var AESKey = msg.payload[0].deviceKey;\n    var SEQNum = msg.payload[0].deviceSQN;\n\n    // Build the new payload. Just the sequence number\n    msg.payload = '{\"SEQN\":' + SEQNum + '}';\n\n    // Move data to base64\n    var bdata = new Buffer(msg.payload).toString('base64');\n\n    // Encrypt the data with the device key.\n    var ciphertext = cryptojs.AES.encrypt(bdata, AESKey );\n\n    // The payload is now the encrypted data\n    msg.payload = ciphertext.toString();\n} catch (err) {\n    msg.payload = \"\";\n    node.log(\"Invalid deviceID request for GET SQN Operation\");\n}\nreturn msg;\n","outputs":1,"noerr":0,"x":650.1666870117188,"y":233.58334350585938,"wires":[["313d3819.34686","33e20204.5cbc66"]]},{"id":"e9c38500.a29dc","type":"function","z":"ee002ffe.ffd9e8","name":"Build Query","func":"var deviceId = msg.req.params.id;\nmsg.topic=\"Select * from Devices where deviceID='\" + deviceId +\"'\";\nreturn msg;","outputs":1,"noerr":0,"x":182.1666717529297,"y":232.58334350585938,"wires":[["aec88434.e67e98"]]},{"id":"9d945f85.62b8a","type":"catch","z":"ee002ffe.ffd9e8","name":"","scope":null,"x":471.16668701171875,"y":93.0833511352539,"wires":[["313d3819.34686"]]},{"id":"33e20204.5cbc66","type":"debug","z":"ee002ffe.ffd9e8","name":"","active":false,"console":"false","complete":"payload","x":635.1666870117188,"y":323.66668701171875,"wires":[]},{"id":"9a0c49f6.f84f","type":"comment","z":"ee002ffe.ffd9e8","name":"REST API - GET operation","info":"Obtains the current Sequence Number for the device","x":144.1666717529297,"y":36.91667175292969,"wires":[]}]

Just copy the above Node-Red flow and use the Import function on Node-Red to paste the code.

We can test the API through wget or curl:

 wget -qO- http://localhost:1880/wsn/12FA
or
 curl -X GET  http://localhost:1880/wsn/12FA
Outputs:
 U2FsdGVkX1+VRlCORJjs2mxxTljfcdu6Z7G8JVyFx7b+jaqKMLeBx4ecLQnjUOYp

Setting the Sequence Number (Case 2 and 3):

For this case the device defines what sequence number it should use, zero or other random number, encrypts with device key the JSON object { SEQN: # } where # is the sequence number, and wraps it on a JSON POST request to our backend:
For example:

 { msg:"U2FsdGVkX1+VRlCORJjs2mxxTljfcdu6Z7G8JVyFx7b+jaqKMLeBx4ecLQnjUOYp" }

The above encrypted data is: {“SEQN”:120}.

As we’ve referred earlier we wont protect this call from replay attacks, so calling the API with same message over and over will just reset the sequence.

The flow to process the request for reseting the sequence is as follow:
Set Sequence flow

and the code is:

[{"id":"501864bd.eea2ec","type":"sqlitedb","z":"ee002ffe.ffd9e8","db":"/home/odroid/Databases/wsn.db"},{"id":"60f28b66.30c0b4","type":"http in","z":"ee002ffe.ffd9e8","name":"setSensorSEQN","url":"/wsn/:id","method":"post","swaggerDoc":"","x":166.1666717529297,"y":467.0833740234375,"wires":[["78ba9614.8925f"]]},{"id":"78ba9614.8925f","type":"function","z":"ee002ffe.ffd9e8","name":"Build Query","func":"// Get the device id\nvar deviceId = msg.req.params.id;\n\n// Build the query. The SQLITE node requires the query in msg.topic\nmsg.topic=\"Select * from Devices where deviceID='\" + deviceId +\"'\";\n\n// Let's pass these parameters forward on its on variables:\nmsg.deviceid = deviceId;\nmsg.rawdata  = msg.payload;\n\nreturn msg;\n","outputs":1,"noerr":0,"x":373.16668701171875,"y":467.2499694824219,"wires":[["76185c6b.95edbc"]]},{"id":"76185c6b.95edbc","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Get Device AES Key","x":188.1666717529297,"y":532.0833740234375,"wires":[["7fd8ec4.4d1e114"]]},{"id":"7fd8ec4.4d1e114","type":"function","z":"ee002ffe.ffd9e8","name":"Decrypt Request","func":"var cryptojs = context.global.cryptojs;\ntry {\n    // Get the key and the raw encrypted data\n    var AESKey = msg.payload[0].deviceKey;\n    var rawdata= msg.rawdata.msg;\n\n    // Decrypt the payload data with the device key. \n    // It returns a string sequence of bytes.\n    var bytes = cryptojs.AES.decrypt( rawdata, AESKey );\n    \n    // Convert bytes to an UTF8 plain string\n    var plaintext = bytes.toString(cryptojs.enc.Utf8);\n   \n    // Convert from base64 to string\n    msg.payload  = new Buffer(plaintext , 'base64').toString('ascii');\n\n    return [ null , msg ];  // Exit the function at output 2.\n    \n} catch (err) {\n    msg.payload = \"\";\n    msg.statusCode=500;     // Set internal server error\n    node.log(\"Invalid deviceID request for GET SQN Operation\");\n    return [ msg , null ];  // Exit the function at output 1\n}\nreturn msg;","outputs":"2","noerr":0,"x":176.1666717529297,"y":597.25,"wires":[["4c3674c1.cdde1c"],["8eba7bdc.d8358"]]},{"id":"4c3674c1.cdde1c","type":"http response","z":"ee002ffe.ffd9e8","name":"","x":596.1666870117188,"y":588.8334045410156,"wires":[]},{"id":"6a9430f9.542d08","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Update SQN","x":435.16668701171875,"y":679.0833740234375,"wires":[["4c3674c1.cdde1c"]]},{"id":"8eba7bdc.d8358","type":"function","z":"ee002ffe.ffd9e8","name":"Build UPD Query","func":"// At this point we should have:\n// msg.payload with the sequence number in JSON: { SEQN: 200}\n// msg.deviceid with the device id\nvar seqnObj = JSON.parse(msg.payload);\nvar seqn = seqnObj.SEQN;\n\nnode.log(\"Seq load: \" + seqn);\n\nmsg.topic = \"update Devices set deviceSQN = \" + seqn +\" where deviceId = '\" + msg.deviceid + \"'\";\n\nreturn msg;","outputs":1,"noerr":0,"x":179.1666717529297,"y":678.75,"wires":[["6a9430f9.542d08"]]}]

For testing we can use sqlite to reset the sequence:

sqlite3 wsn.db 
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> update Devices set deviceSQN=30;
sqlite> select * from Devices;
12FA|2B7E151628AED2A6ABF7158809CF4F3C|30

and do the encrypted request:

curl -H "Content-Type: application/json" -X POST -d     '{"msg":"U2FsdGVkX1+VRlCORJjs2mxxTljfcdu6Z7G8JVyFx7b+jaqKMLeBx4ecLQnjUOYp"}' http://localhost:1880/wsn/12FA

and see the result:

sqlite> select * from Devices;
12FA|2B7E151628AED2A6ABF7158809CF4F3C|120

We can repeat the request over and over again, and it will be accepted every time since we have no mechanism to stop that.

Conclusion:
We can see that the most secure way to use the Sequence Number is to only use Case 1, requesting it, and not Case 2 and Case 3 where the devices set’s the sequence and is vulnerable to replay attacks.

So this is already a long post and there are still several things to do:
– The data storage request handler so we can finally transmit securely our data.
– The device code on the ESP8266 to execute the requests.

Comming up in the next posts!

Node-Red and the Crypto-JS library

Just a quick install instructions to use the Crypto-Js library on Node-Red, so that we can use encryption/decryption functions and hash functions like AES and SHA256 on workflows.

Installation and configuration:
First move to the Node-Red install directory. The right directory can vary from platform to platform.
In my case I’m running Node-Red on my Odroid C1+ controlled by the PM2 process manager:

cd .node-red
npm install crypto-js

Edit the settings.js file to add the global defined library:

    functionGlobalContext: {
        // os:require('os'),
        // octalbonescript:require('octalbonescript'),
        // jfive:require("johnny-five"),
        // j5board:require("johnny-five").Board({repl:false})
        gcm:require('node-gcm'),
        cryptojs:require('crypto-js')
    },

I already have the node-gcm module to use it for Android notifications from Node-Red workflows.

Restart Node-Red. In my case as: pm2 restart node-red

How to use:
We can now use the Crypto-js functions on our workflow functions like this:

// Import the global Crypto-js module defined on Node-Red settings.js file
var cryptojs = context.global.cryptojs;

// Move data to base64
var bdata = new Buffer(msg.payload).toString('base64');

// Encrypt the data with the device key.
var ciphertext = cryptojs.AES.encrypt(bdata, "2B7E151628AED2A6ABF7158809CF4F3C" );

// The payload is now the encrypted data
msg.payload = ciphertext.toString();

return msg;

That’s it. We can now access the provided functions by the Crypto-js library from our Node-Red workflows.

Synology Reverse Proxy with DSM 6.0

The latest Synology DSM 6.0 version now supports out of the box reverse proxy configuration. So no need to build and edit internal configuration files. Everything can be done now on the DSM web frontend.

To configure the reverse proxy we need to go to the DSM web application, select the Control Panel and then the Application Portal icon:

Application portal Configuration

Application portal Configuration

We can see that I already have some applications configured and with the HTTP internal ports defined. For example the Notes application is accessed internally by the URL http://diskstation:9350. Note that I haven’t defined a HTTPS port because I’ll use the reverse proxy as the HTTPS frontend.

So we take note of the ports for the applications that we want to make available at the reverse proxy, in my case the port 9350 for the Notes application, and create a new reverse proxy configuration by selecting the Reverse Proxy tab and pressing the Create button:

Notes Reverse Proxy Configuration

Notes Reverse Proxy Configuration

Take note of the following:

– I’m using one of the available domains provided by synology myds.me.
– The Synology domains, at least the myds.me supports sub-domain wildcarding.
– So I can have the primalcortex.myds.me as my main domain
– And I can have all subdomains below the primalcortex.myds.me, like, for example, notes.primalcortex.myds.me

So I will make the Notes application available at the notes.primalcortex.myds.me and that is the hostname that I need to define under my reverse proxy configuration.

So with the above rule all requests to https://notes.primalcortex.myds.me will be routed to the localhost server running on the port 9350…

And that’s it. Just make another set of rules for the other application under their own sub-domains.

Edit: The following configuration shows the Note Station, File Station and Video Station reverse proxy configuration that allows those “apps” to be accessible from the external IP. Note that this means that port 80 (plain HTTP, if used) and port 443 (HTTPS) must be forwarded on the router configuration to the Synology internal IP:

Selection_008

In my case only HTTPS is used, so I’ve not forwarded the port 80 from the external interface of the router to the Synology.

MQTT Mosquitto broker with SSL/TLS transport security

Just a quick note in setting up transport layer security on the MQTT Mosquitto broker for both supported protocols: MQTT and WebSockets.

There are several posts on the web regarding this, namely:

SSL Client certs to secure mqtt and Mosquitto websocket support

Those posts explain more or less what is needed to be done to have TLS/SSL transport security. These are just my notes:

Generating the server certificates:
This can be quite easily accomplished by using the following script: https://github.com/owntracks/tools/blob/master/TLS/generate-CA.sh.
This script will generate a self signed certificate to be used by Mosquito for providing TLS for the MQTT and WebSocket protocol. All that is needed to run the script is to have openssl installed on your Linux machine.

If the script is called without parameters, it will generate a self signed certificate for the hostname where the script is running. Otherwise a we can pass a hostname as the first parameter to the script.

After running the script, the following files are generated:

  1. ca.crt – The CA (Certificate Authority, who published the host certificate) public certificate.
  2. hostname.crt – The hostname, that will run the mosquitto broker, public certificate.
  3. hostname.key – The hostname private key.

After having these files, we need to configure the Mosquitto Broker to use them.

Mosquitto configuration:
To configure the Mosquito broker we need first to copy the certificates and key files to a known directory. We will create a certs directory under /etc/mosquitto:

sudo -s
mkdir -p /etc/mosquitto/certs
cp ca.crt /etc/mosquitto/certs
cp hostname.* /etc/mosquitto/certs

After this we can modify the mosquitto configuration file. One important thing to keep in mind is that lines must be following each other without blank lines after the listener directive.

So:

# Plain MQTT protocol
listener 1883

# End of plain MQTT configuration

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/ca.crt
certfile /etc/mosquitto/hostname.crt
keyfile /etc/mosquitto/hostname.key

# End of MQTT over TLS/SLL configuration

# Plain WebSockets configuration
listener 9001
protocol websockets

# End of plain Websockets configuration

# WebSockets over TLS/SSL
listener 9883
protocol websockets
cafile /etc/mosquitto/ca.crt
certfile /etc/mosquitto/hostname.crt
keyfile /etc/mosquitto/hostname.key

We will make one more change, but restart mosquitto broker now and do some testing.

Testing MQTT TLS/SSL configuration
We can use Mqtt-Spy to subscribe to our defined test topic: test. We can use plain MQTT or use MQTT over TLS/SSL:

MQTT Spy simple TLS configuration

MQTT Spy simple TLS configuration

We can use then the MQTT spy tool to publish or subscribe MQTT topics.

By command line, the mosquitto_sub and mosquitto_pub only worked if the port number for MQTTS is provided, otherwise it gives a TLS error:

mosquitto_pub --cafile /etc/mosquitto/ca.crt -h localhost -t "test" -m "message" -p 8883

mosquitto_sub -t \$SYS/broker/bytes/\# -v --cafile /etc/mosquitto/ca.crt -p 8883

This should work without any issues.

Testing MQTT websockets over TLS/SSL configuration
The issue with this testing is that we are using a self signed certificate, so only useful for local, restricted, testing.
Before we can use the MQTT websockets with TLS/SSL enabled we need to use the browser and visit the following URL:

  https://MQTT_BROKER_IP_OR_HOSTNAME:9883/mqtt

Note that we are using HTTPS. When connecting to the above URL, the browser should complain about the insecure connection, due to the self signed certificate, and we need to add an exception and always accept that certificate. After that the error should be something like connection reset or failed to load page. This is normal, since the browser won’t upgrade the connection to a web socket.
We can now use the Hive MQTT Websockets Client to test our connection, and it should work fine (Note the connected green icon and SSL is selected):
Hive MQTT WebSocket client

Forcing TLSv1.2
All this work of enabling TLS/SSL on the Mosquitto Broker is needed, since most IoT clouds that have MQTT interface need that the connection is over TLS/SSL. More specifically AWS IoT cloud needs the connection to be protected by TLS/SSL, but that connection must be only on version 1.2 of the TLS protocol. AWS IoT cloud also requires client authentication through client certificates, but we are not dealing with this part on this post.

So we are now configuring our Mosquitto broker to only accept TLSv1.2 connections. To do that we modify the mosquitto.conf file and add the following line:

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/ca.crt
certfile /etc/mosquitto/hostname.crt
keyfile /etc/mosquitto/hostname.key
tls_version tlsv1.2

# WebSockets over TLS/SSL
listener 9883
protocol websockets
cafile /etc/mosquitto/ca.crt
certfile /etc/mosquitto/hostname.crt
keyfile /etc/mosquitto/hostname.key
tls_version tlsv1.2

and restart the broker.

Testing TLS V1.2
We can specify the TLS version on the mosquitto command line utils:

[pcortex@pcortex:~]$ mosquitto_pub --cafile ./ca.crt --tls-version tlsv1.2 -h localhost -t "test" -m "mes" -p 8883 -d
Client mosqpub/26994-pcortex sending CONNECT
Client mosqpub/26994-pcortex received CONNACK
Client mosqpub/26994-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (3 bytes))
Client mosqpub/26994-pcortex sending DISCONNECT
[pcortex@pcortex:~]$ mosquitto_pub --cafile ./ca.crt --tls-version tlsv1.1 -h localhost -t "test" -m "m3224" -p 8883 
Error: A TLS error occurred.

As we can see the lower versions of the TLS protocol are now not accepted.
The Websockets client should work without any issues.

Conclusion:
This configuration only solves the transport security, not the authentication security. The later can be accomplished by using the username/password process or using client certificates, which is the process that Amazon AWS IoT cloud uses. But those are topics for other posts.