Cloud based deployment for IOT devices

Following up on my previous post Cloud based CI with Platformio, after we have the build output from the Continuous Integration process, we are able now to deploy to our devices.

This last deploy phase of the cycle Develop, CI, Deliver using Cloud infrastructure, only makes sense to devices that are powerful enough to have permanent or periodic network connectivity and have no problems or limitations with power usage, bandwidth, are in range and are able to remotely be updated.

In reality this means that most low power devices, devices that use LPWAN technologies like LoraWan or SigFox, devices that are sleeping most of the time and are battery powered are not able to be easily updated. For these cases the only solution is really out of band management by upgrading locally the device.

So the scope of this post is just to simply build a cloud based process to allow ESP8266 devices to get update firmware from the CI output. On it’s simplest form all we need is to create a web server, make the firmware available at the server and provide the URL for OTA updates to the ESP8266 that use the HTTP updater.

One can already use from the squix blog the PHP file to be deployed on PHP enabled web server that delivers the latest builds for devices requesting over the air updates.

Openshift PaaS Cloud Platform

The simplest way of making the Squix PHP page available on the cloud is to use the great Platform as a Service Openshift by RedHat. The free tier allows to have three applications (gears) available and the sign up is free. At sign up time we need to name our own domain suffix so that, for example I choose primal I’ll have URL’s such as application-primal.rhcloud.com.

Openshift offers a series of pre-configured applications ready to be deployed such NodeJs, Java, Python and PHP.

Openshift preconfigured platforms

So after sign up, all we need is to create a new application based on the PHP 5.4 template, give it an URL (it can be the default PHP), and that’s it: we have our PHP enabled web server.

Deploying code to Openshift

To deploy code to Openshift we use the Git tool for manipulating our application repository on the PaaS cloud platform.

So we must first clone our repository locally, modify it and then upload the changes.

For obtaining the repository URL and connection details, we must first setup our local machine with the rhc command line tool and upload our public SSH key to the Openshift servers:

 [pcortex@pcortex:~]$ gem install rhc

If the gem tool is not available, first install Ruby (sudo pacman -S ruby).

We then setup the rhc tool with the command rhc setup. Complete details here.

The command rhc apps should list now our Openshift applications:

[pcortex@pcortex:~]$ rhc apps
nodejs @ http://nodejs-primal.rhcloud.com/ (uuid: 9a72d50252d09a72d5)
-----------------------------------------------------------------------------
  Domain:     primal
  Created:    Aug 26  3:43 PM
  Gears:      1 (defaults to small)
  Git URL:    ssh://9a72d50252d09a72d5@nodejs-primal.rhcloud.com/~/git/nodejs.git/
  SSH:        9a72d50252d09a72d5@nodejs-primal.rhcloud.com
  Deployment: auto (on git push)

  nodejs-0.10 (Node.js 0.10) 
----------------------------             
    Gears: 1 small 
                    
php @ http://php-primal.rhcloud.com/ (uuid: c0c157c41271b559e66) 
-----------------------------------------------------------------------                    
  Domain:     primal          
  Created:    12:16 PM  
  Gears:      1 (defaults to small) 
  Git URL:    ssh://c0c157c41271b559e66@php-primal.rhcloud.com/~/git/php.git/                
  SSH:        c0c157c41271b559e66@php-primal.rhcloud.com 
  Deployment: auto (on git push) 

  php-5.4 (PHP 5.4)
  -----------------
    Gears: 1 small

You have access to 2 applications.

We pull now the remote repository to our machine:

[pcortex@pcortex:~]$ mkdir Openshift
[pcortex@pcortex:~]$ cd Openshift
[pcortex@pcortex:Openshift]$ git clone ssh://c0c157c41271b559e66@php-primal.rhcloud.com/~/git/php.git/
[pcortex@pcortex:Openshift]$ cd php
[pcortex@pcortex:php]$ wget https://raw.githubusercontent.com/squix78/esp8266-ci-ota/master/server/firmware.php 

We should now change the PHP file so it uses our repository to bring up our firmware:

 <?php
    $githubApiUrl = "https://api.github.com/repos/squix78/esp8266-ci-ota/releases/latest";
    $ch = curl_init();

And then it’s just to commit the change to Openshift:

[pcortex@pcortex:php]$ git add firmware.php
[pcortex@pcortex:php]$ git commit -m "Added firmware.php file"
[pcortex@pcortex:php]$ git push
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 924 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Stopping PHP 5.4 cartridge (Apache+mod_php)
remote: Waiting for stop to finish
remote: Waiting for stop to finish
remote: Building git ref 'master', commit a72403a
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is 8fdecb3f
remote: Activating deployment
remote: Starting PHP 5.4 cartridge (Apache+mod_php)
remote: Application directory "/" selected as DocumentRoot
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://php-primal.rhcloud.com/~/git/php.git/
   321e48b..a72403a  master -> master

And that’s it: the link for HTTP OTA is available at http://php-primal.rhcloud.com/firmware.php

Final notes:

With the above firmware.php file we can deliver a single firmware file to any device that calls the page.

But a better solution is needed if we want to:

– Deliver multiple firmware files to different devices
– Deliver different versions of firmware files, for example be able to lock a specific version to some devices
– Know which devices have updated
– Know which version of firmware the devices are running

and of course, add some security.

ESP8266 – Logging data in a backend – AES and Crypto-JS

After building, on the previous posts, the Node-Red based backend to support E2EE (End to End Encryption) so we can log data into a central server/database, from our devices securely, without using HTTPS, we need now to build the firmware for the ESP8266 that allows it to call our E2EE backend.

The firmware for the ESP8266 must gather the data that it wants to send, get or generate the current sequence number for the data (to avoid replay attacks), encrypt the data and send it to the backend.
On the backend we are using the Java script library for cryptographic functions Crypto-js, and specifically we are encrypting data with the encryption algorithm AES. So all we need is to encrypt our data with AES on the ESP8266, send it to the Node-Red Crypto-js backend, decrypt it and store it, easy right?

Not quite, let’s see why:

Crypto-js and AES:
We can see that on my Node-Red function code and testing programs I’m using something similar to the following code example:

var CryptoJS = require("crypto-js");
var message  = "Message to encrypt";
var AESKey   = '2B7E151628AED2A6ABF7158809CF4F3C';

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text in Base64: " ,  ciphertext.toString(CryptoJS.enc.base64) );

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Several points regarding the above code need clarification:

The code variable AESKey the way it is used on the above example encrypt and decrypt functions isn’t really a key but a passphrase from where the real key and an initialization vector or salt value is generated (I’m using the names interchangeably, but they are not exactly the same thing except they are public viewable data that must change over time/calls).
The use for the generated key is self explanatory, but the initialization vector (IV) or salt value is used to allow that the encrypted data output to be not the same for the same initial message. While the key is kept constant and secret to both parties, the IV/salt changes between calls, which means that the above code, when run multiple times, will never produce the same output for the same initial message.

Still referring to the above code, the algorithm that generates the key from the passphrase is the PBKDF2 algorithm. More info at Crypto-js documentation. At the end of the encryption the output is a OpenSSL salted format that means that the output begins by the signature id: Salted_, followed by an eight byte salt value, and after the salt, the encrypted data.

So if we want use the API has above on the node-js/crypto-js side, we need to implement on the ESP8266 side both the AES and PBKDF2 algorithms.

I decided not to do that, first because finding a C/C++ implementation of the PBKDF2 algorithm that could be portable and worked on the ESP822 proved difficult, and second the work for porting it to the ESP8266 won’t be needed if I use a KEY/IV directly, and so I decided to use the more standard way of providing an AES key and an initialization vector for encrypting and decrypting data.

In the case of Node-JS and Crypto-JS when using an explicit key and IV the code looks like:

var CryptoJS = require("crypto-js");
var request = require('request');

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":145 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);

var iv = CryptoJS.enc.Hex.parse('0000000000000000');
var key= CryptoJS.enc.Hex.parse(AESKey);

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, key , { iv: iv } );

//console.log("Cypher: ", ciphertext );
console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");

console.log("=============================================================================");
console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Now, with above code, where the IV is always initialized to the same value, in this case ‘0000000000000000’, we can see when running the above code several times that the output is always the same since the IV is kept constant. Also the encrypted output is now just the raw encrypted data and not the Openssl format.

So to make the above code secure we must randomize the IV value for producing an output that is always different, even from several program runs when encrypting the same source data.

As a final note, if we count the number of HEX characters on the Key string, we find out that they are 16 bytes, which gives a total of 128 key bits. So the above example is using AES128 encryption, and with default Crypto-js block mode and padding algorithms which are CBC (Chain block mode) and pkcs7.

Interfacing Crypto-js and the ESP8266:
Since we are using AES for encrypting data and decrypting data, we need first to have an AES library for the ESP8266. The AES library that I’m using is this one Spaniakos AES library for Arduino and RPi. This library uses AES128, CBC and pkcs7 padding, so it ticks all boxes for compatibility with Crypto-js…

I just added the code from the above library to my Sming project and also added this Base64 library so that I can encode to and from Base64.

The only remaining issue was to securely generate a truly random initialization vector. And while at first I’ve used some available libraries to generate pseudo-random numbers to real random numbers, I’ve found out that the ESP8266 seems to have already a random number generator that is undocumented: Random number generator

So to generate a random IV value is as easy as:

uint8_t getrnd() {
    uint8_t really_random = *(volatile uint8_t *)0x3FF20E44;
    return really_random;
}

// Generate a random initialization vector
void gen_iv(byte  *iv) {
    for (int i = 0 ; i < N_BLOCK ; i++ ) {
        iv[i]= (byte) getrnd();
    }

So our ESP8266 code is as follow:

Global variables declarations:
The N_Block defines the encryption block size, that for AES128 is 16 bytes.

#include "AES.h"
#include "base64.h"

// The AES library object.
AES aes;

// Our AES key. Note that is the same that is used on the Node-Js side but as hex bytes.
byte key[] = { 0x2B, 0x7E, 0x15, 0x16, 0x28, 0xAE, 0xD2, 0xA6, 0xAB, 0xF7, 0x15, 0x88, 0x09, 0xCF, 0x4F, 0x3C };

// The unitialized Initialization vector
byte my_iv[N_BLOCK] = { 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};

// Our message to encrypt. Static for this example.
String msg = "{\"data\":{\"value\":300}, \"SEQN\":700 , \"msg\":\"IT WORKS!!\" }";


The example function ESP8266 Sming code is:

void testAES128()  {

    char b64data[200];
    byte cipher[1000];
    byte iv [N_BLOCK] ;
    
    Serial.println("Let's encrypt:");
    
    aes.set_key( key , sizeof(key));  // Get the globally defined key
    gen_iv( my_iv );                  // Generate a random IV
    
    // Print the IV
    base64_encode( b64data, (char *)my_iv, N_BLOCK);
    Serial.println(" IV b64: " + String(b64data));
       
    Serial.println(" Mensagem: " + msg );
 
    int b64len = base64_encode(b64data, (char *)msg.c_str(),msg.length());
    Serial.println (" Message in B64: " + String(b64data) );
    Serial.println (" The lenght is:  " + String(b64len) );
    
    // For sanity check purpose
    //base64_decode( decoded , b64data , b64len );
    //Serial.println("Decoded: " + String(decoded));
    
    // Encrypt! With AES128, our key and IV, CBC and pkcs7 padding    
    aes.do_aes_encrypt((byte *)b64data, b64len , cipher, key, 128, my_iv);
    
    Serial.println("Encryption done!");
    
    Serial.println("Cipher size: " + String(aes.get_size()));
    
    base64_encode(b64data, (char *)cipher, aes.get_size() );
    Serial.println ("Encrypted data in base64: " + String(b64data) );
      
    Serial.println("Done...");
}

When the above code/function is executed on the ESP8266 it outputs the following:

Let's encrypt:
 IV b64: cAFviaDMHejlteGn9/4eQQ==
 Mensagem: {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }
 Message in B64: eyJkYXRhIjp7InZhbHVlIjozMDB9LCAiU0VRTiI6NzAwICwgIm1zZyI6IklUIFdPUktTISIgfQ==
 The lenght is:  76
Encryption done!
Cipher size: 80
Encrypted data in base64: /1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=
Done...

And now we can feed the above Base64 IV and encrypted data to our decoding program in Node-Js using Crypto-JS:

var CryptoJS = require("crypto-js");
var request = require('request');

var esp8266_msg = '/1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=';
var esp8266_iv  = 'cAFviaDMHejlteGn9/4eQQ==';

// The AES encryption/decryption key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

var plain_iv =  new Buffer( esp8266_iv , 'base64').toString('hex');
var iv = CryptoJS.enc.Hex.parse( plain_iv );
var key= CryptoJS.enc.Hex.parse( AESKey );

console.log("Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt( esp8266_msg, key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Base64);
var decoded_b64msg =  new Buffer(plaintext , 'base64').toString('ascii');
var decoded_msg =     new Buffer( decoded_b64msg , 'base64').toString('ascii');

console.log("Decrypted message: ", decoded_msg);

and the output is:

Decrypted message:  {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }

So, as the message shows, it WORKS!. AES encryption on the ESP8266 to a Node-JS Crypto-JS based code where decryption occurs.

Final notes:
So all is needed now is to build on the ESP8266 side the message with the encrypted data and the IV, and send through plain HTTP a JSON object to the Node-Red back end.

{
  "msg":"/1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=",
  "iv":"cAFviaDMHejlteGn9/4eQQ=="
}

On the Node-Red back end the decryption can now be done easily as was shown on the above testing code.

ESP8266 and the Micropython firmware

One of the alternative firmwares available for the ESP8266 is MicroPython Python interpreter. I’ve found by chance a great tutorial at Adafruit for building the Micropython firmware and I thought to give it a try.

Building the firmware:
The Adafruit tutorial uses a Vagrant based virtual machine to build the firmware, but since I’m already running Linux (Arch Linux to be more specific) and already have the Falcon open ESP8266 sdk installed (see here) and the esptool.py also available since I’m using the Sming firmware, I’ve just downloaded only the latest Micropython source code from the Github repository to a local directory.

cd ~
git clone https://github.com/micropython/micropython
cd ~/micropython
git submodule update --init
cd ~/micropython/esp8266
export PATH=/opt/esp-open-sdk/xtensa-lx106-elf/bin:$PATH
make axtls
make

So far nothing different from the Adafruit tutorial except that I’m not using the vagrant VM. Also make sure that you first execute the command make axtls otherwise the main make command will compiling about not finding a version.h file. Also make sure that the export command that adds the path to the Xtensa compiler points to the right location.

After compiling, which was fast, I’ve just flashed the firmware on my Wemos mini D1 board. Again I had trouble flashing this board with other speeds than the default 115200 bps.

So:

cd ~/micropython/esp8266/build
esptool.py -p /dev/ttyUSB0 --baud 115200 write_flash 0 firmware-combined.bin

And after flashing, we can connect through the serial terminal, and pressing CTRL-D we should be greeted with the following message:

PYB: soft reboot
could not open file 'main.py' for reading
MicroPython v1.8-157-g480159c on 2016-05-29; ESP module with ESP8266
Type "help()" for more information.

Some basic tests:
When doing some tests I’ve found out that most information is outdated regarding the version of Micropython that I’ve flashed. For example:

>>> import pyb
Traceback (most recent call last):
  File "", line 1, in 
ImportError: no module named 'pyb'
>>> 

The common refered module pyb doesn’t exist, because it’s now machine:

>>> import machine
>>> dir(machine)
['__name__', 'mem8', 'mem16', 'mem32', 'freq', 'reset', 'reset_cause', 'unique_id', 'deepsleep', 'disable_irq', 'enable_irq', 'RTC', 'Timer', 'Pin', 'PWM', 'ADC', 'UART', 'I2C', 'SPI', 'DEEPSLEEP', 'PWR_ON_RESET', 'HARD_RESET', 'DEEPSLEEP_RESET']
>>>

So the following code:

>>> import pyb
>>> pin = pyb.Pin(14, pyb.Pin.OUT)  # Set pin 14 as an output.
>>> for i in range(10):
...    pin.value(0)     # Set pin low (or use pin.low())
...    pyb.delay(1000)  # Delay for 1000ms (1 second)
...    pin.value(1)     # Set pin high (or use pin.high())
...    pyb.delay(1000)

should be now:

>>> import machine
>>> pin = machine.Pin(14, machine.Pin.OUT)  # Set pin 14 as an output.
>>> for i in range(10):
...    pin.value(0)     # Set pin low (or use pin.low())
...    pyb.delay(1000)  # Delay for 1000ms (1 second)
...    pin.value(1)     # Set pin high (or use pin.high())
...    pyb.delay(1000)

Other interesting stuff is for example, the esp module:

>>> import esp
>>> dir(esp)
['__name__', 'osdebug', 'sleep_type', 'deepsleep', 'flash_id', 'flash_read', 'flash_write', 'flash_erase', 'flash_size', 'neopixel_write', 'apa102_write', 'dht_readinto', 'freemem', 'meminfo', 'info', 'malloc', 'free', 'esf_free_bufs', 'SLEEP_NONE', 'SLEEP_LIGHT', 'SLEEP_MODEM', 'STA_MODE', 'AP_MODE', 'STA_AP_MODE']
>>> esp.freemem()
17672
>>> esp.meminfo()
data  : 0x3ffe8000 ~ 0x3ffe8410, len: 1040
rodata: 0x3ffe8410 ~ 0x3ffe9038, len: 3112
bss   : 0x3ffe9038 ~ 0x3fff6bd0, len: 56216
heap  : 0x3fff6bd0 ~ 0x3fffc000, len: 21552

There are examples on GitHub to use the deepsleep functions:

Regarding the Wifi connectivity, by default when starting up the ESP8266 sets up a Wifi access point with the name Micropython-XXXXX where XXXX are some digits from the MAC address. Following the documentation the AP is protected with the password micropythoN, and sure enough the connection works. Still I haven’t tested it enough, for example, accessing the Python interpreter over Wifi, instead of through the serial port.

Anyway, one final test is to use Python to connect to make the ESP8266 to connect to my network. The instructions are simple, just write help(), and the micropython shows how to do it:

import network
>>> help()
Welcome to MicroPython!

For online docs please visit http://docs.micropython.org/en/latest/esp8266/ .
For diagnostic information to include in bug reports execute 'import port_diag'.

Basic WiFi configuration:

import network
sta_if = network.WLAN(network.STA_IF)
sta_if.active(True)
sta_if.scan()                             # Scan for available access points
sta_if.connect("", "") # Connect to an AP

and we can see if it connected successfully:

>>> sta_if.isconnected() 
True

and what IP configuration was set:

>>> sta_if.ifconfig()
('10.42.0.173', '255.255.255.0', '10.42.0.1', '10.42.0.1')

Also I was unable to access the Python interpreter through the access point connection. Supposedly there should be a listener running on port 8266 that allows access over WIFI, but I my tests found that the port 8266 was closed. Probably I need to initialize something first…
Anyway, there is a tool webrepl that allows to use the browser through websockets to connect to the ESP8266 and access the Python prompt and also to copy files to the ESP8266, namely the main.py startup file.

To finish. during my tests I had no crashes or surprise reboots. Using Python also has the advantage, in my opinion, that is more mainstream than Lua, since we leverage desktop programming with device programming. Also the useful tool ESPLorer already supports Micropython, it means that probably it is a better alternative for quick hacks using the ESP8266 instead of Nodemcu running LUA.

ESP8266 – Logging data in a backend with end to end encryption – Storing Data

As we can see in my previous post, for securely store data from a device, we should make the data secure during transport and tamper proof. This can be achieved by using encryption, but the use of encryption only does not solve some attacks, like replay attacks.

So a scheme where data is encrypted and associated with a sequence number is used, to have transport security and replay attack protection, and on the previous post we’ve implemented the base framework that allows the support of our requirements.

So the device generates, for example, the following message:

 { "data":{"value": 300} , "SEQN": 120 }

meaning that it wants to send to the backend the JSON Object {“value”: 300}, and the current Sequence number is 120. On the next message the sequence number should be 121, but that is supposing that algorithm for the sequence number is to increase it one by one per message.

Giving the message, we encrypt it now with our device key, and build another JSON object, the one that will be sent to the Node-red based backend:

 { "msg":"U2FsdGVkX1..."}

Since we still don’t have our device ready to send messages, we will build a simple NodeJs program to generate the messages:

mkdir ~/gen
cd ~/gen
npm init   (Just accept the defaults)
npm install crypto-js --save
npm install request --save

And the code to generate our message is as follow:

var CryptoJS = require("crypto-js");
var request = require('request');

// API endpoint.
var apiEP = 'http://localhost:1880/data/';
var deviceID = '12FA';

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":121 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);
console.log("Message: " , message );

// Encode the string to base64. Not really needed.
message = new Buffer(message).toString("base64");
console.log("Message B64: " , message );

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");
console.log("Let's call the Node-Red API end point: ");

var URL = apiEP + deviceID;
var rawdata = ciphertext.toString(CryptoJS.enc.base64);

console.log(" Calling end point: " , URL);
console.log(" RawData: " , rawdata );

// Let's call the REST API end point.
request( {
    url: URL,
    method: "POST",
    json: true,
    body: { "msg": rawdata}
    } ,
    function (error, response, body) {
        if (!error && response.statusCode == 200) {
            console.log(" ");
            console.log("REST API Output: ");
            console.log(body)
        }
    }
);
console.log("=============================================================================");
console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);
console.log(" ");

console.log("Decrypted message: " , new Buffer(plaintext , 'base64').toString('ascii'));

We can modify the message by modifying the msgObjs variable. We run this program by executing node index.js and at the end the Node-Red answer should be displayed under REST API Output.

If everything is setup correctly the REST API should return:

REST API Output: 
{ status: 'OK' }

But calling a second time without modifying the sequence number should fail:

REST API Output: 
{ status: 'NOT OK' }

The Node-Red flow:

The Node-Red flow, receives the REST POST request for the device ID, obtains from the database the private key associated to the device, decrypts the payload, and checks the message sequence number vs the database sequence number. By design it allows the device to send messages above the current sequence number to allow message gaps (messages that where lost). We can, if needed, process this gaps so we can have an idea of how many messages that we are loosing.

If the decryption is successful and the sequence number is ok, we increment the sequence number to the next value, and store the data:

Node-Red Data Storage flow

Node-Red Data Storage flow

The code is as follow:

[{"id":"501864bd.eea2ec","type":"sqlitedb","z":"ee002ffe.ffd9e8","db":"/home/odroid/Databases/wsn.db"},{"id":"9e2ef.10b2b512","type":"http in","z":"ee002ffe.ffd9e8","name":"setSensorData","url":"/data/:id","method":"post","swaggerDoc":"","x":142.1666717529297,"y":812.0833587646484,"wires":[["fa205b8f.8de96"]]},{"id":"fa205b8f.8de96","type":"function","z":"ee002ffe.ffd9e8","name":"Build Query","func":"// Get the device id\nvar deviceId = msg.req.params.id;\n\n// Build the query. The SQLITE node requires the query in msg.topic\nmsg.topic=\"Select * from Devices where deviceID='\" + deviceId +\"'\";\n\n// Let's pass these parameters forward on its on variables:\nmsg.deviceid = deviceId;\nmsg.rawdata  = msg.payload;\n\nreturn msg;","outputs":1,"noerr":0,"x":199.1666717529297,"y":882.7499694824219,"wires":[["94004285.aa1418"]]},{"id":"94004285.aa1418","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Get Device AES Key","x":404.16668701171875,"y":817.0833435058594,"wires":[["7dd8f03c.ab2778"]]},{"id":"d0ed1bda.b15428","type":"http response","z":"ee002ffe.ffd9e8","name":"","x":1023.1666870117188,"y":956.8333435058594,"wires":[]},{"id":"7dd8f03c.ab2778","type":"function","z":"ee002ffe.ffd9e8","name":"Decrypt Request","func":"var cryptojs = context.global.cryptojs;\ntry {\n    // Get the key, Sequence number and the raw encrypted data\n    var AESKey = msg.payload[0].deviceKey;\n    msg.dbSQN = msg.payload[0].deviceSQN;   // Obtain also the currrent SQN on the database\n    \n    var rawdata= msg.rawdata.msg;\n    \n    node.log(  msg.devSQN );\n    // Decrypt the payload data with the device key. \n    // It returns a string sequence of bytes.\n    var bytes = cryptojs.AES.decrypt( rawdata, AESKey );\n    \n    // Convert bytes to an UTF8 plain string\n    var plaintext = bytes.toString(cryptojs.enc.Utf8);\n    node.log( plaintext );\n    // Convert from base64 to string\n    msg.payload  = new Buffer(plaintext , 'base64').toString('ascii');\n\n    return [ null , msg ];  // Exit the function at output 2.\n    \n} catch (err) {\n    msg.payload = { \"status\":\"NOT OK\"};\n    msg.statusCode = 500;     // Set internal server error\n    node.log(\"Invalid deviceID request for GET SQN Operation\");\n    return [ msg , null ];  // Exit the function at output 1\n}\nreturn msg;","outputs":"2","noerr":0,"x":668.1666870117188,"y":817.7499694824219,"wires":[["d0ed1bda.b15428"],["c7be8b09.893b9"]]},{"id":"c7be8b09.893b9","type":"function","z":"ee002ffe.ffd9e8","name":"Verify MSG SQN","func":"// At this point we should have:\n// msg.payload with the sequence number in JSON: { SEQN: 200} and the rest of the message\n// msg.deviceid with the device id\ntry {\n    var msgObj = JSON.parse(msg.payload);\n    var msgSeqn = msgObj.SEQN;   // The sequence number that the device as sent\n    var msgData = msgObj.data;   // The data sent\n  \n    if ( msg.dbSQN <= msgSeqn )  { // Valid Sequence number\n      node.log( \"Sequence is VALID!!!!!\");\n      msg.payload = { \"status\":\"OK\"};\n      msg.data = msgData;\n      msg.devSQN= msgSeqn;\n      return [ msg, msg];\n    } else {\n        node.log( \"Sequence is invalid!!!!\");\n        msg.payload = { \"status\":\"NOT OK\"};\n        return [ null , msg ];\n    }\n} catch( e ) {\n    node.log(\"Error verifying SEQN...\") \n    msg.payload = { \"status\":\"Internal Error\"};\n    return [ null , msg ];\n}","outputs":"2","noerr":0,"x":164.1666717529297,"y":996.7499694824219,"wires":[["c8ce0d1c.589058","533d8275.e04474"],["d0ed1bda.b15428"]]},{"id":"c8ce0d1c.589058","type":"function","z":"ee002ffe.ffd9e8","name":"Update Sequence ","func":"\n    var dbSQN = msg.devSQN + 1;   // It allows message gaps.\n\n    msg.topic=\"Update Devices Set deviceSQN= \" + dbSQN + \" Where deviceID='\" + msg.deviceid +\"'\";\n \n    //node.log(\"SQL: \" + msg.topic );\nreturn msg;","outputs":1,"noerr":0,"x":584.1666870117188,"y":1034.5833740234375,"wires":[["bef21dc4.4642a"]]},{"id":"bef21dc4.4642a","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Set SQN","x":780.1666870117188,"y":1034.4166259765625,"wires":[[]]},{"id":"533d8275.e04474","type":"function","z":"ee002ffe.ffd9e8","name":"Update Data","func":"  // Let's extract the data.\n  // This step should be modified as needed.\n  var data = msg.data.value;\n  msg.topic=\"Insert into Data Values ( '\" + msg.deviceid + \"', CURRENT_TIMESTAMP , \" + data + \")\";\n \n  //node.log (\"Data: \" + data );\nreturn msg;","outputs":1,"noerr":0,"x":579.1666870117188,"y":922.5833435058594,"wires":[["770abb44.6f54d4"]]},{"id":"770abb44.6f54d4","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Save Data","x":782.1666870117188,"y":922.4165954589844,"wires":[[]]}]

And so the only thing missing is the ESP8266 code to call and setting data using the encrypted transport. For implementing that we will be using the Sming framework that will use and call the above defined API on this post and previous.

ESP8266 Wemos D1 mini

A quick note regarding the ESP8266 based board named Wemos D1 mini.

I bought one of these boards, from Aliexpress for under 4€. It took more or less a month to arrive. I’ve still haven’t too much time to do something with it, but here are my first impressions:

The Wemos D1 mini is, in my opinion a better buy than the ESP-01 and ESP-12, for many things:

– It has all the supporting electronics for power and serial communication:
– Included USB to serial converter CH340G, but with no TX and RX activity leds.
– It can be powered by the mini USB connection, or through a 5V pin, since it has a 5V to 3.3V linear converter
– It also has a 3.3V input/output pin.
– It comes with some modern version of NodeMcu, so it’s ready to be used with the ESPlorer application, just connect to USB and we are ready.
– It includes stack able pins that allows several ways of creating our devices.

But most important is that it can be flashed with alternative firmwares, like Sming, and that’s what I did. Sming applications that I have, like the Wifi Scanner worked just fine and detected several wifi networks on the vicinity.

Also, the Wemos D1 mini has a reset button, but there is no need to use it to flash the device. The RTS and DTR lines of the serial connection are properly mapped to the GPIO0 and RST pins of the ESP-8266 so flashing is automatic. No need to press reset or manipulate pins. I was able to flash the device with several different bit rates like 460800 and 921600. The esptool.py can have some issues detecting the board, from some of the tests that I did, but at the end it flashes the board with no issue.

So, being the double of the price of the ESP-01 and a bit less price difference from the ESP-12, but with the included power and USB-Serial electronics, that are needed to be budgeted and implemented with the other boards, and all on a nice small package, it is by far the best way to enter the ESP-8266 world.

ESP8266 – Sming I2C

This brief post is about how to use the I2C bus/protocol with the Sming framework running on the ESP8266.
Since one of the Sming framework objectives is to be as close as possible to Arduino compatibility, it comes as no surprise, that I2C usage on the Sming framework is almost, if not identical, to the Wire Library on the Arduino.

Starting up:
Before using the I2C functions we need to initialize the I2C bus, otherwise the ESP8266 will crash when using it.
There is no need to include any .h file to access the global Wire object, so just initialize it with the Wire.begin() function. By default the bus will be on the pins GPIO0 and GPIO2:

  1. GPIO0 – SCL (Clock)
  2. GPIO2 – SDA (Data)

But those default pins can be changed before initializing the I2C bus with the function Wire.pins( SCL , SDA).

So a possible initialization sequence could be something like:

void startMain()  {  // Called from the Wifi connected callback
 ...
 ...
 Wire.pins ( 0 , 2 ); // SCL, SCA -> 0 and 2 are the defaults. This is optional.
 Wire.begin();        // Initialize the bus.  This is mandatory.
 ...
 ...

And now we can use the Wire object to access the I2C bus.

Reading and writing on the I2C bus:
After the bus is initialized we can read and write from/to I2C connected devices:

  Wire.beginTransmission(I2C_addr);       // An I2C transaction always starts with this.
  Wire.requestFrom( I2C_addr, length);    // Request for reading.

  while(Wire.available())                 // Loop as data is available
  {
	  for(uint8_t i = 0; i < length; i++)
	  {
		  buffer[i] = Wire.read();// Collect data.
	  }
  }
  Wire.endTransmission();                 // I2C transaction always ends with endTransmission()
}

On the above case the code will read lenght bytes from the I2C I2C_addr address. This will work for example for an I2C device that has no command register, so no need to define what we want to read. An example of that is the PCF7485 I2C expander.

 // The PCF7485 only reads one byte:
 byte readPCF7485( int I2C_addr ) {
     byte b;

     Wire.beginTransmission(I2C_addr);
     Wire.requestFrom( I2C_addr, 1);
     b = Wire.read();
     Wire.endTransmission();

     return b;
 }

But for devices that have a command register and return several bytes, like the BMP180, something like this is what is used:

// Taken from the BMP180 lib and adapted for use as an example.
void readBMP180(int BMP180_Address, int sub_address, int length, uint8_t buffer[])
{
  Wire.beginTransmission(BMP180_Address);
  Wire.write(sub_address);
  Wire.endTransmission();
  
  Wire.beginTransmission(BMP180_Address);
  Wire.requestFrom(BMP180_Address, length);

  while(Wire.available())
  {
	  for(uint8_t i = 0; i < length; i++)
	  {
		  buffer[i] = Wire.read();
	  }
  }
  Wire.endTransmission();
}

So the above code samples show how to read and write data to the I2C bus. A simple I2C write code looks like:

 void writePCF8574( int I2C_addr , byte data ) {
    Wire.beginTransmission(BMP180_Address);
    Wire.write( data );
    Wire.endTransmission();
 }

I2C bus scanner:
Available on the Sming code base there is an I2C bus scanner. This function will scan all possible I2C address to see if any I2C connected device is available. For reference the code is as follows:

void scanI2CBus()
{
  byte error, address;
  int nDevices;

  Serial.println("Scanning...");

  nDevices = 0;
  for(address = 1; address < 127; address++ )
  {
	// The i2c_scanner uses the return value of
	// the Write.endTransmisstion to see if
	// a device did acknowledge to the address.
	Wire.beginTransmission(address);
	error = Wire.endTransmission();

	WDT.alive(); // Second option: notify Watch Dog what you are alive (feed it)

	if (error == 0)
	{
	  Serial.print("I2C device found at address 0x");
	  if (address<16)
		Serial.print("0");
	  Serial.print(address,HEX);
	  Serial.println("  !");

	  nDevices++;
	}
	else if (error==4)
	{
	  Serial.print("Unknow error at address 0x");
	  if (address<16)
		Serial.print("0");
	  Serial.println(address,HEX);
	}
  }
  if (nDevices == 0)
	Serial.println("No I2C devices found\n");
  else
	Serial.println("done\n");
}

So when we are prototyping we can at startup call the scanI2cBus() to get a list of available devices on the terminal console.

Final thoughts:
The Wire object is global and it is compatible with the Arduino code and the many libraries that are available, for example the BM180 library that is available on the Sming framework.

But Sming framework also offers a TwoWire class for I2C communication, that possibly allows to have multiple I2C buses, if that makes sense, to be available. But if we use this class instead of the Wire global object, most of the libraries don’t work because of the Wire object dependency.

Sming ESP8266 – Driving a 1.8 TFT ST7735 chip based display

The ST7735 display controller is one of the available display drivers found driving the 1.8 TFT 128×160 resolution display. This driver/TFT combo is found from several sources like Adafruit and eBay, and has software drivers for Arduino and other platforms.
Still these displays, despite the same name (1.8 TFT 128×160), can come with different controllers from the ST7735 chip, like the Samsung S6D02A1 normally found on displays labelled QDTech, and apparently, there are also versions based on the ILI9163C chip.

Anyway I only have one of these displays and it’s based on the ST7735 chip and so here’s how I’ve connected and used a 1.8 TFT with 128×160 resolution the ESP8266 based ESP-12 board using the Sming Framework. The board needs to be an ESP-12 or other that exposes the ESP8266 SPI hardware pins, and so the ESP-01 is out of the question.

Why Sming framework?
The main reason is that I can use Netbeans on my ESP8266 software build process without touching the Arduino IDE… Reason enough, I think 🙂

The Adafruit ST7735 library
The latest Adafruit ST7735 Library supports the Arduino Boards and Teensy boards, and at compile time, specific code segments are enabled/disabled based on the target architecture. For example, when compiling to an Arduino the __AVR__ compiler time variable is defined. As far as I’m aware, this variable is also set when using the Arduino IDE when targeting the ESP8266 board, so no issues here.

The Sming framework defines the following compile time variable: __ESP8266_EX__, and we can just hack the library to support the Sming framework, or adapt it to be used on all or platforms. Anyway, my version is available here, and as far as my tests go, it works fine: Sming Adafruit ST7735 ESP8266.

The hardware connections:
The Library for the Sming framework uses the hardware SPI pins, so the connections should be the following:

Fixed pins SPI pins (can’t be changed)

  1. D13 – Serial data out -> MOSI/SDA
  2. D14 – Clock

Pins that can be changed and defined on the library constructor:

  1. D4 – CS -> Chip Select
  2. D5 – DC -> Data/Command

Also there is the reset pin from the display, that we can connect to an ESP8266 pin, like RST, for example, but I’ve just connected the pin to VCC (3.3v).

Power

The display is powered up by 3.3V. To do that I needed to put a bit of solder on the back pads near the power regulator, that regulates VCC from 5V to 3.3V so that the regulator is out of the circuit. This is of course because the ESP8266 works at 3.3V, so no need for level conversion and other power source.

Software:

Based on this, and using my ST7735 library on Github, my initialization code looks like:

#include <SmingCore/SmingCore.h>
#include <Adafruit_ST7735.h>
#define TFT_CS    4
#define TFT_DC    5

Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, 0);

...
// And on a main function after boot:
void startMain() {

    Serial.println("----TFT_INIT-----");
    tft.initR(INITR_BLACKTAB);
    Serial.println("TFT Initialized");

    tft.setRotation(1);
    tft.fillScreen(ST7735_BLACK);  
       
    tft.drawLine(160, 12, 0, 12, ST7735_YELLOW);    
    cycle.initializeMs(2000, doScreen).start();
}

And that’s it.

Note: Notice that the ST7735 available examples, use extensively the delay function, and that will make the ESP8266 to reboot.