LPWAN – Starting up with LoraWAN and The Things Network

LPWAN Networks – A simple introduction
Low Power Wide Area Communications (LPWAN) classifies a group of communication protocols featuring low power usage and a high communication range. For Internet of Things communications, where battery powered devices and constrained devices (weak CPU, low RAM/ROM) are the norm, LPWAN use as the communication protocol for IoT makes sense, since it makes possible to have standalone devices with batteries that last years, instead of hours or days, and might be hundreds of meters to Kms away from a base station/gateway.

But LPWAN protocols, in contrast have low communication bandwidth, since from power, range and bandwidth, we can only have two of those, and while this might be a problem for certain applications, IoT devices don’t require high bandwidth and since most have sparse communication requirements, and when they do communicate they don’t need to transmit a lot of data.

Starting up with LoraWan and The Things Network
One of the possible ways of starting to use LPWAN networks in our IoT devices, is to use a LPWAN protocol named LoraWan, which is based on the Lora physical protocol, and if available at our area, the crowd sourced LPWAN network The Things Network as the backend network for receiving data.

Lora uses the 868Mhz ISM radio band and it uses a form of signal transmission called CSS. The ISM radio band can be used freely by everybody, but under certain rules that depends of the country where the devices are located. So to use that band, rules like maximum emission power and duty cycle (usage of the available spectrum) are defined, and in Europe, the maximum power is 20mW and 1% duty cycle per day. Additionally the back end operator can add other restrictions.

Over Lora, the Lorawan protocol is an open standard and source code for implementing it is available and already ported to several devices, including the Arduino, Raspberry PI and the ESP8266, among others.

The software for implementing the LoraWan protocol at the end devices (nodes) is IBM’s LMIC library, available at Github and on Platformio libs:

[ ID  ] Name             Compatibility         "Authors": Description
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[ 852 ] IBM LMIC framework arduino, atmelavr, atmelsam, espressif8266, intel_arc32, microchippic32, nordicnrf51, teensy, timsp430 "IBM, Matthijs Kooijman": Arduino port of the LMIC (LoraWAN-in-C, formerly LoraMAC-in-C) framework provided by IBM. | @1.5.0+arduino-1

Based on this library we can build code and nodes that are able to transmit data through LoraWan to a network backend. Specifically for the Arduino, the Atmega328 is the bare minimum to begin with, since the library, due to use of AES encryption functions occupies a lot of memory.

The backend can be provided by really anyone, including Telco operators, or private and crowd source operators like The Things Network (TTN). TTN provides the backend and management services, but depends on crowd sourced gateways, and might not be available at our area. Still it is possible, for testing, to build our own gateways, our buy them, and connect them to the Things Network. TTN doesn’t require any access fees (yet).

So with LoraWan an the Things Network, we can build our own nodes and gateways since code and hardware can be easily obtained, connect them and use it to build applications.

Regarding LoraWan we can also read this great introduction from Design Spark.

Lora hardware:

Anyway the easiest way for starting up using Lora and Lorawan, is to buy an Dragino Lora Shield and connect it to an Arduino Uno or Mega.

Dragino Lora Shield

Dragino Lora Shield

This is a great shield to startup since doesn’t need any soldering or complex configuration, just plug it on an Arduino shield and use the LMIC library and some code samples. Also the Dragino Shield can be used to build a simple gateway by connecting it to a Raspberry PI or ESP8266 to build what is called a single channel gateway, that allows to test gateway functionality, but it isn’t quite compatible with the LoraWan specifications. Anyway it gets the job done if there is no gateway nearby. Just make sure that you by version 1.3 or 1.4 of the shield. Mine also came with an SMA antenna.

Other alternatives to start using Lorawan are available at eBay/Aliexpress and other renowned name shops, namely Lora radio transceivers at around 8/16€, for example the HopeRF RFM95. Those we can also use them with Arduino or ESP8266 to build our nodes or single channel gateways.

Just make sure that the frequency for these modules and shields must match the allowed radio transmission frequency in your area. In my case, in Europe is 868Mhz, but for example at USA is 900Mhz.

Dragino Lora Shield Jumpers
The Shield has some jumpers and only looking at the schematic and cross referenced them with the RFM95 module (used by the shield as the Lora transceiver) I could see what they where for:

– There are two set of jumpers:

  • One defines the pins for SPI communication: SV# Jumpers;
  • The other set defines which data pins of the RFM95 module are exposed to the shield allowing them to be connected to the Arduino: JP# Jumpers.

The JP# jumpers connect the following if closed:

  • JP1 – RFM95 DI1 – Arduino pin D6
  • JP2 – RFM95 DI2 – Arduino pin D7
  • JP3 – RFM95 DI5 – Arduino pin D8

The RFM95 pin DI0 is permanently connected to Arduino pin D2.
The RFM95 pin RST (Reset) is permanently connected to Arduino pin D9.

Their function for Lora might be the following (The function depends how the RFM95 module is used)

  • RFM95 DI0 – Indicates if data was received and is ready to be read through the SPI bus. Indicates end of transmission for data that was previously sent: RXReady/TXReady
  • RFM95 DI1 – Receive timeout for Lora mode.
  • RFM95 DI2 – Receive timeout for FSK mode.
  • RFM95 DI5 – Used by Semtech Library. LMIC library doesn’t use it.

The SV# jumpers connect:

  • SV2 – SPI Clock line. Default on pin D13, otherwise on Arduino SPI CLK on the ICSP header.
  • SV3 – SPI Data line In (MOSI). Default on pin D11, otherwise on Arduino MOSI pin on the ICSP header.
  • SV4 – SPI Data line Out (MISO). Default on pin D12, otherwise on Arduino MISO pin on the ICSP header.

The SPI Chip Select line is always at pin D10.

So now we know that D10, D9 and D2 are used permanently connected and used by the shield, and the others can be connected or disconnected if needed or not.

LMIC software with Dragino Lora Shield:

To start using the Dragino Lora Shield so it connects to a LPWAN network, we can start using the following example: TTN node by ABP. ABP means Activation by Personalization, which means that all data for joining the network will be explicitly defined on the code. Other alternative would be OTAA: Over the air activation, where the gateway sends the configuration data to the node. Since we don’t know if we have a gateway in range, let’s start first with ABP mode.

The above code uses the LMIC library for implementing the LoraWan stack.
Since LMIC library doesn’t use DI05, we can remove the JP3 jumper, and free this IO line for other things, like another shield.

To use the LMIC library we must define first the pins that are used according to the shield configuration:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, 7}             //DIO pin mapping for DIO0, DIO1 and DIO2  
};

But DIO2 pin is only used for FSK modulation, and so if only using LoraWan we can also open up the JP2 jumper, and our LMIC pin configuration can be as follows:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, LMIC_UNUSED_PIN} //DIO pin mapping for DIO0, DIO1. DIO2 is not used  
};

Connecting to TTN
Connecting to The Things Network (TTN) depends of course of an available TTN gateway at the nodes range. Still we need to configure some parameters to allow the node to connect.

On this example the connection is done through Activation by Personalization. This means that we should put on our code the Network Session key and Application Session key on the code. To obtain these values we need to register on the TTN site, TTN Dashboard and add an ABP device..

From this site, then we can get the three configuration parameters that we need:

  • Device ID
  • Network Session key
  • Application Session key

Note by pressing the icon we can get the values in a form ready to paste it on the code:

/*************************************
 * TODO: Change the following keys
 * NwkSKey: network session key, AppSKey: application session key, and DevAddr: end-device address
 *************************************/
static u1_t NWKSKEY[16] = { 0xAA, 0x0F, 0x29, 0xD3, 0x9D, 0x7A, 0xAE, 0x3B, 0x54, 0xCF, 0xDF, 0x2F, 0x2A, 0x23, 0x55, 0xB5 };
static u1_t APPSKEY[16] = { 0x2F, 0x85, 0x43, 0x5B, 0x34, 0x9C, 0x80, 0xC6, 0xA8, 0xFA, 0x27, 0x49, 0x5A, 0x36, 0x82, 0x21 };
static u4_t DEVADDR = 0x95337738;

And that’s it, when running the sketch, and if in range of a gateway, the sent message should appear at the Dashboard:

Messages received for the device

Messages received for the device

Final thoughts:
In my area there are at least, supposedly, two active TTN gateways and I’m around 2Km’s away from them in a dense urban area.
But when running the sketch the first times, I had no results what so ever.

One of the configurations options for LoraWan is what is called a Spread Factor that, in a simplest way, exchanges range with on-air time for transmission, being SF7 the fastest speed and shorter range and SF12 the slowest speed and higher range. The sketch default value for the sprea factor was SF7, and changing it to SF12:

 ...
    // TTN uses SF9 for its RX2 window.
    LMIC.dn2Dr = DR_SF9;

    // Set data rate and transmit power for uplink (note: txpow seems to be ignored by the library)
    LMIC_setDrTxpow( DR_SF12 ,14);

    // Start job
    do_send(&sendjob);

With SF12 the message started be received and the sequence numbers of the messages received where continuous, so no lost messages.

Dropping to SF11, also worked fine, and the message sequence received and shown on the TTN Dashboard where still continuous.

At SF10, some of the messages where lost, almost 75% of them. In this case, arranging the antenna position (to exactly vertical) and placement did alter the reception successes.

Processing received data
After the data is at the TTN backend there are several ways of getting it. For reference on the TTN site there are instructions in how to access the data.

Crash course on Lora
For further information the next one hour video is a great starting point:

The slides for the presentation are available here:

https://www.dropbox.com/s/87htha4dq5b32wa/ttn-lora-crash-course.pdf?dl=0

MQTT Mosquitto broker – Client Authentication and Client Certificates

After seeing how to set up transport layer security for the Mosquitto MQTT broker by using the Transport layer security on this post, we need to see how to setup client authentication (only authorized clients can connect to the broker) either by using the common user/password based authentication method or using client certificates.

Authentication by user and password:
First let’s enable authentication to the broker by setting up user and password authentication. For enabling this kind of authentication we need to modify the broker configuration (/etc/mosquitto/mosquitto.conf in my case) file and change the following entries:

allow_anonymous false
password_file /etc/mosquitto/passwd_mqtt

Before restarting the MQTT broker we need to add some users to the passwd_mqtt file with the command mosquitto_passwd:

mosquitto_passwd -c passwd_mqtt user1

The -c parameter is for creating the initial password file, if doesn’t exist, otherwise it will overwrite it!. For adding new users or update the passwords just run the command without the -c parameter.

After restarting the broker will should not be able to logon anonymously:

# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d
Client mosqpub/25688-pcortex sending CONNECT
Client mosqpub/25688-pcortex received CONNACK
Connection Refused: not authorised.
Error: The connection was refused.

But providing the user name and password:

# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d -u user1 -P secret
Client mosqpub/25709-pcortex sending CONNECT
Client mosqpub/25709-pcortex received CONNACK
Client mosqpub/25709-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (7 bytes))
Client mosqpub/25709-pcortex sending DISCONNECT 

For subscription, the same has to be done, again by providing an user and password:

# mosquitto_sub -t \$SYS/broker/bytes/\# -v --cafile /etc/mosquitto/certs/ca.crt -p 8883 -u user1 -P secret
$SYS/broker/bytes/received 217
$SYS/broker/bytes/sent 20

Authentication by using client certificates
Using client certificates, signed by a certificate authority, assures the client identity. The certificate authority used must be the same used by the server certificates and is only supported over TLS/SSL.
For using client certificates for authentication, we need to change the listener configuration for TLS/SSL by adding the following directives:

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/hostname.crt
keyfile /etc/mosquitto/certs/hostname.key
require_certificate true
use_identity_as_username true

The require_certificate directive with the value true means that clients must now provide a client certificate to connect.
The use_identity_as_username means that the user name of the connecting user is taken from the CN (Common Name) property of the certificate, otherwise we still need to provide an user and password.

With the above new configuration, we now can’t access the broker with user/password token:

mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d -u user1 -P secret
Client mosqpub/26549-pcortex sending CONNECT
Error: A TLS error occurred.

the result on the log file is:

1478537500: New connection from 127.0.0.1 on port 8883.
1478537500: OpenSSL Error: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
1478537500: Socket error on client , disconnecting.

For accessing now the broker we must provide a client certificate and a private key. The client certificate must be generated from the same CA (Certificate authority) that created the server certificate, otherwise, the client certificate can’t be validated, and the connection fails:

1478537629: New connection from 127.0.0.1 on port 8883.
1478537629: OpenSSL Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
1478537629: Socket error on client , disconnecting.

So on the same directory where the ca.crt and ca.key are residing execute the script to create the client certificate. For example for creating a client certificate for user1, we need to execute the following command:

# ./gen_CA.sh client user1

And we should have three files, two of them the user1.crt, the user certificate, and the user1.key, the user1 private key.

We can now logon to the broker:

# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d  --cert user1.crt --key user1.key 
Client mosqpub/30264-pcortex sending CONNECT
Client mosqpub/30264-pcortex received CONNACK
Client mosqpub/30264-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (7 bytes))
Client mosqpub/30264-pcortex sending DISCONNECT

If we check the log, we have the following:

1478539830: New client connected from 127.0.0.1 as mosqpub/27159-pcortex (c1, k60, u'user1').
1478539830: Client mosqpub/27159-pcortex disconnected.
1478601507: New connection from 127.0.0.1 on port 8883.

We can see that the broker extracted the username from the certificate property CN. If we don’t use the use_identity_as_username as true we need to provide the username and password and the client certificate. In this case, the certificate is validated, and the user used to logon might not be the same as the one defined on the CN certificate property. So without this directive we either need to allow anonymous logon again or define user and passwords.

Additional thoughts:
With user and password authentication we can revoke access to an user, by deleting it from the password file or changing the password.
But what about the authentication based on client certificates? As long the certificate is valid, the user can logon at will, since the broker will always accept it until the end of the certificate validation date.
The solution for this is to revocate the client certificate so when it is used, the broker rejects it. For this functionality, most Certificate Authorities provide the revocation list by providing either by CRL (Certificate Revocation list) file, or by OSCP (Online Status Certificate Protocol), and the server checks the client certificate on this list before allowing access. Mosquitto broker only works with CRL files.

We need to modify the listener configuration to verify the client certificates if it was revocated or not:

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/halcyon.crt
keyfile /etc/mosquitto/certs/halcyon.key
require_certificate true
use_identity_as_username true
crlfile /etc/mosquitto/certs/ca.crl

The issue now is that the script that we are using for generating certificates doesn’t generate CRLs, and so we can’t revocate certificates by using this script.

We can do it by hand, but that wouldn’t make our life much easier, since revocating certificates means that the CA server must now track all certificates and their status.
One possible solution is to use scripts that allow to create (a not very secure) CA: easy-ca. These scripts will replace the genCA.sh script and allow the revocation of certificates.

Getting the easy-CA scripts:

git clone https://github.com/fcgdam/easy-ca.git

I’m omitting a lot of output that these scripts do, namely asking for the Organization name, CA passwords and so on.

Somewhere where we find suitable we create our CA:

Generating the CA:

# cd ~ 
# ~/easy-ca/create-root-ca -d CA
# cd CA

Then at the CA directory we created, we can now create our broker certificate and user certificate. Make sure that the names make sense.

Generating the broker certificate:

# bin/create-ssl -s brokername
# bin/create-client -c user1

At the end we have the following files:

ca/ca.crt  <- CA certificate
crl/ca.crl  <- CA CRL list

certs/brokername.server.crt  <- Server certificate
certs/user1.server.crt <- Client certificate

private/brokername.server.key
private/user1.server.key

We now have all the needed files for setting up our broker with support for CRL.

# sudo -s
# cp ~/CA/ca/ca.crt /etc/mosquitto/certs
# cp ~/CA/crl/ca.crl /etc/mosquitto/certs
# cp ~/CA/certs/brokername.server.crt /etc/mosquitto/certs/brokername.crt
# cp ~/CA/private/brokername.server.key /etc/mosquitto/certs/brokername.key

And restarting the broker we have the new certificates in place.
We can test now by using our CA user certificate:

# cd ~/CA
# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d  --cert certs/user1.client.crt --key private/user1.client.key 
Client mosqpub/7424-pcortex sending CONNECT
Client mosqpub/7424-pcortex received CONNACK
Client mosqpub/7424-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (7 bytes))
Client mosqpub/7424-pcortex sending DISCONNECT

It works. Now let’s revoke the user certificate:

# bin/revoke-cert -c certs/user1.client.crt 

Revoking certificate 'certs/user1.client.crt'

Reason for revocation: 

1. unspecified
2. keyCompromise
3. CACompromise
4. affiliationChanged
5. superseded
6. cessationOfOperation
7. certificateHold

Enter 1-7 [1]: 2
You are about to revoke this certificate with reason 'keyCompromise'.
Are you SURE you wish to continue? [y/N]: y
Using configuration from conf/ca.conf
Revoking Certificate 02.
Data Base Updated
Using configuration from conf/ca.conf

Server certificate revoked.

We need to copy the new CRL to the directory where mosquitto expects the CRL:

# sudo cp crl/ca.crl /etc/mosquitto/certs

And after restarting the broker we have:

mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d  --cert certs/user1.client.crt --key private/user1.client.key  
Client mosqpub/7640-pcortex sending CONNECT
Error: A TLS error occurred.

And on the logs:

1478625909: OpenSSL Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
1478625909: Socket error on client , disconnecting.
1478625910: New connection from 127.0.0.1 on port 8883.

Cloud based deployment for IOT devices

Following up on my previous post Cloud based CI with Platformio, after we have the build output from the Continuous Integration process, we are able now to deploy to our devices.

This last deploy phase of the cycle Develop, CI, Deliver using Cloud infrastructure, only makes sense to devices that are powerful enough to have permanent or periodic network connectivity and have no problems or limitations with power usage, bandwidth, are in range and are able to remotely be updated.

In reality this means that most low power devices, devices that use LPWAN technologies like LoraWan or SigFox, devices that are sleeping most of the time and are battery powered are not able to be easily updated. For these cases the only solution is really out of band management by upgrading locally the device.

So the scope of this post is just to simply build a cloud based process to allow ESP8266 devices to get update firmware from the CI output. On it’s simplest form all we need is to create a web server, make the firmware available at the server and provide the URL for OTA updates to the ESP8266 that use the HTTP updater.

One can already use from the squix blog the PHP file to be deployed on PHP enabled web server that delivers the latest builds for devices requesting over the air updates.

Openshift PaaS Cloud Platform

The simplest way of making the Squix PHP page available on the cloud is to use the great Platform as a Service Openshift by RedHat. The free tier allows to have three applications (gears) available and the sign up is free. At sign up time we need to name our own domain suffix so that, for example I choose primal I’ll have URL’s such as application-primal.rhcloud.com.

Openshift offers a series of pre-configured applications ready to be deployed such NodeJs, Java, Python and PHP.

Openshift preconfigured platforms

So after sign up, all we need is to create a new application based on the PHP 5.4 template, give it an URL (it can be the default PHP), and that’s it: we have our PHP enabled web server.

Deploying code to Openshift

To deploy code to Openshift we use the Git tool for manipulating our application repository on the PaaS cloud platform.

So we must first clone our repository locally, modify it and then upload the changes.

For obtaining the repository URL and connection details, we must first setup our local machine with the rhc command line tool and upload our public SSH key to the Openshift servers:

 [pcortex@pcortex:~]$ gem install rhc

If the gem tool is not available, first install Ruby (sudo pacman -S ruby).

We then setup the rhc tool with the command rhc setup. Complete details here.

The command rhc apps should list now our Openshift applications:

[pcortex@pcortex:~]$ rhc apps
nodejs @ http://nodejs-primal.rhcloud.com/ (uuid: 9a72d50252d09a72d5)
-----------------------------------------------------------------------------
  Domain:     primal
  Created:    Aug 26  3:43 PM
  Gears:      1 (defaults to small)
  Git URL:    ssh://9a72d50252d09a72d5@nodejs-primal.rhcloud.com/~/git/nodejs.git/
  SSH:        9a72d50252d09a72d5@nodejs-primal.rhcloud.com
  Deployment: auto (on git push)

  nodejs-0.10 (Node.js 0.10) 
----------------------------             
    Gears: 1 small 
                    
php @ http://php-primal.rhcloud.com/ (uuid: c0c157c41271b559e66) 
-----------------------------------------------------------------------                    
  Domain:     primal          
  Created:    12:16 PM  
  Gears:      1 (defaults to small) 
  Git URL:    ssh://c0c157c41271b559e66@php-primal.rhcloud.com/~/git/php.git/                
  SSH:        c0c157c41271b559e66@php-primal.rhcloud.com 
  Deployment: auto (on git push) 

  php-5.4 (PHP 5.4)
  -----------------
    Gears: 1 small

You have access to 2 applications.

We pull now the remote repository to our machine:

[pcortex@pcortex:~]$ mkdir Openshift
[pcortex@pcortex:~]$ cd Openshift
[pcortex@pcortex:Openshift]$ git clone ssh://c0c157c41271b559e66@php-primal.rhcloud.com/~/git/php.git/
[pcortex@pcortex:Openshift]$ cd php
[pcortex@pcortex:php]$ wget https://raw.githubusercontent.com/squix78/esp8266-ci-ota/master/server/firmware.php 

We should now change the PHP file so it uses our repository to bring up our firmware:

 <?php
    $githubApiUrl = "https://api.github.com/repos/squix78/esp8266-ci-ota/releases/latest";
    $ch = curl_init();

And then it’s just to commit the change to Openshift:

[pcortex@pcortex:php]$ git add firmware.php
[pcortex@pcortex:php]$ git commit -m "Added firmware.php file"
[pcortex@pcortex:php]$ git push
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 924 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Stopping PHP 5.4 cartridge (Apache+mod_php)
remote: Waiting for stop to finish
remote: Waiting for stop to finish
remote: Building git ref 'master', commit a72403a
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is 8fdecb3f
remote: Activating deployment
remote: Starting PHP 5.4 cartridge (Apache+mod_php)
remote: Application directory "/" selected as DocumentRoot
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://php-primal.rhcloud.com/~/git/php.git/
   321e48b..a72403a  master -> master

And that’s it: the link for HTTP OTA is available at http://php-primal.rhcloud.com/firmware.php

Final notes:

With the above firmware.php file we can deliver a single firmware file to any device that calls the page.

But a better solution is needed if we want to:

– Deliver multiple firmware files to different devices
– Deliver different versions of firmware files, for example be able to lock a specific version to some devices
– Know which devices have updated
– Know which version of firmware the devices are running

and of course, add some security.

Cloud based continuous integration and delivery for IOT using PlatformIO

After finding out the PlatformIO for IoT development I started to read some of the Platformio documentation and also what other users have written about it.

One of the most interesting features of Platformio is that it supports to be used on a https://en.wikipedia.org/wiki/Continuous_integration process for any PlatformIO based project. This is important for using automated build systems for CI (Continuous Integration), and so, allows early detection of possible build problems. CI makes sense when several contributors/team are working on the same code repository, and we need to make sure that the project is able to build with all the team/contributors code inputs/changes. At the end, deliverables can be pushed to their destination:

CI

What Platformio CI enables is that for our IOT projects we can have automatic builds after code commits on the code repository (for example, GitHub). When the automatically build is triggered, PlatformIO is able to pull all dependencies and frameworks to build our project on the automated build system.

After the automatically build is triggered and the build is successful we can then deliver the output.

One of the most interesting examples of this workflow is the following post http://blog.squix.org/2016/06/esp8266-continuous-delivery-pipeline-push-to-production.html that shows the process of developing, committing code to the repository, triggering automatic builds and, at major releases, deploy firmware updates over the air (OTA) to the ESP8266. All of this using Platformio and 100% cloud infrastructure for IOT deployment.

Starting up with Platformio and TravisCI

Platformio supports several CI systems, and one of them is Travis.CI that integrates out of the box with GitHub. To enable Travis.CI on your GitHub projects/repository, just sign in on GitHub, and on (another browser tab) goto the TravisCI site and press the Sign in with GitHub button, accept the authorization requests, and select the repositories to enable for CI by going to your user profile.

Enabled Repository

After enabling the repositories, every commit will trigger an automatic build on the Travis CI platform. As a build status indicator we can add a IMG tag to the README.md file so we can have the build status without going to the Travis site, for example: https://travis-ci.org/fcgdam/RFGW_SensorNodes.svg?branch=master.

Setting up the build
Travis.CI will start the build process according to instructions defined on the hidden repository file .travis.yml that is unique and specific for each repository.

This Travis Documentation link explains in detail the logic and options behind the travis.yml file and build process.

Fortunately PlatformIO when initializing a project creates a sample travis.yml file.
Based on that sample, here is one of mine for compiling two sample Arduino projects on the same code repository:

language: python
python:
     - "2.7"

sudo: false
cache:
     directories:
         - "~/.platformio"

install:
     - pip install -U platformio

script:
     - cd RFGW && platformio run -e nanoatmega328
     - cd ../ATTINNY_MailMonitor_TX && platformio run

The tags language:, sudo: and cache: are not changed in this case.

The install: and script: tags are customized so that our project can be built.

On the install: tag, the first command is always the installation of the platformio tools, followed, if necessary, by installation of other dependencies. For example if our project depends on libraries from the Platformio library registry we can do the following:

install:
     - pip install -U platformio
     - platformio lib install 64

This, before building, will install first platformio, and then it will install the ArduinoJson (Id 64) library. We can add as much commands as we want prefixed by the dash character.
Also this is one way of doing it, but this means that we need to change .travis.yml file every time we add/remove libraries.

Another way is to add the library dependency on the project file platformio.ini like this:

[env:nanoatmega328]
platform = atmelavr
framework = arduino
board = nanoatmega328
lib_install= 64

And in this case all the dependencies are associated within the project file. But in this case the build commands are different.

So one example with multiple libraries could be as follow:

install:
     - pip install -U platformio
     - platformio lib install 64
     - platformio lib install 652
script:
     - platformio run

And this is the same as above:

install:
     - pip install -U platformio
script:
     - platformio run -e nanoatmega328

where nanoatmega328 is the environment configuration on platformio.ini file:

[env:nanoatmega328]
platform = atmelavr
framework = arduino
board = nanoatmega328
lib_install= 64,652

To end this topic, notice that we can have several builds on the same repository, just add several command lines to the script: tag:

script:
     - cd RFGW && platformio run -e nanoatmega328
     - cd ../ATTINNY_MailMonitor_TX && platformio run

I’m always using relative paths related to the project root in the above example.

Further information can be found on the Platformio Travis integration guide and on the Travis.CI site.

Continuous delivery

Since every commit to our repository triggers the Travis build process, we need now to distinguish between working commits and release commits so that on this last type, the build output is made available to be deployed to end devices/platforms for OTA updates (or not).

This can be easily achieved by using git tags and conditional deploy process that only works when a git tag is defined.

With this scheme the normal cycle of git add, commit and push will create a working commit that triggers as usual the CI build process but not the deployment phase of copying the build output (binaries, firmware) to the GitHub releases tab.

Creating a tag and a release can be done either by command line or by using the Github web interface, being this the easiest way of doing it.

But there are some pre-requisites to allow this to happen:
– Generate an OAuth personal GitHub token so that Travis can copy the output to the Releases GitHub tab.
– Encrypt the OAuth token with the travis command line tool.
– Change the .travis.yml file so that it deploys the build output to the releases tab only at tagged commits.

The GitHub token is generated by going to your Github Profile, selecting settings and then Personal Access tokens.
Press Generate new token, enter your password and add permissions to access your repositories.
The permissions should be at least full repo access:

GitHub Personal token permission

Make sure that at the end you copy the OAuth token, otherwise you must generate another one from the beginning.

The Github token must be kept secret at all times, and since we need to have it on the travis.yml file which can be read by everyone, we must make sure that we encrypt it in such a way that only Travis.CI can use it.
This is achieved by using the travis command line tool on our machine so we need to:

[pcortex@pcortex:RFGW_SensorNodes|master]$ travis encrypt GH_TOKEN="7d54...df5977" --add 

The GH_TOKEN is the name that must hold the OAuth token so that Travis can use it securely.

With the above command, the .travis.yml file is modified and the following entry is added:

env:
  global:
    secure: WqroI5PtWWm94svvau5G3LFz4PMBU...fY=

We can now add the final configuration to the Travis.CI configuration file, so that at tag releases, the build output is automatically added.

deploy:
  provider: releases
  api_key: ${GH_TOKEN}
  file:
    - $TRAVIS_BUILD_DIR/.pioenvs/nanoatmega328/firmware.hex
    - $TRAVIS_BUILD_DIR/.pioenvs/digispark-tiny/firmware.elf
  skip_cleanup: true
  on:
    tags: true  

The provider: tag defines that we want to deploy to GitHub Releases, and the api_key: tag contains the secure Oauth token to access GitHub.

The file: tag define which files we want to deploy, and in this case we use the $TRAVIS_BUILD_DIR environment variable to locate our build directory root. The skip_cleanup: will avoid cleaning all build outputs.

The on: tag is the most important because it conditionally defines that the deploy process only happens at tagged release.

So after this configuration, if we commit without tagging, the build is made, but no deploy to the Releases happens:

Travis Build without tagging

If we want to trigger a tagged commit we can do it purely on the command line:

[pcortex@pcortex:RFGW_SensorNodes|master]$ git tag -a v0.3 -m "Version 0.3"
[pcortex@pcortex:RFGW_SensorNodes|master]$ git push origin v0.3
....
To https://github.com/fcgdam/RFGW_SensorNodes.git
 * [new tag]         v0.3 -> v0.3

And that’s it: Automatic build process and release:

Tagged build process output

And the final result:

Tagged build output

We have now a tagged release with source code and binaries automatically created and packaged.

Deployment

At this point we have the deliverables for a release, and we should now distributed/deploy it. This is by itself another process that can be done through Cloud services or locally, it really depends of the end architecture.

The most important issue here is related to security: making sure that the correct build is delivered, was not changed in any way and reaches the intended devices.

Platformio

Normally I don’t use or look solutions for problems that I don’t have. And for this reason alone, meant that http://platformio.org/ stayed under my radar for so long.

Whats my problem?
Since I’m building my mailbox monitoring solution, I’m using two different types of Arduino boards: a Arduino nano 328p based board for the RF gateway, and some Digispark AtTinny85 based boards for the sensors. The Digispark AtTinny85 boards are not completely energy efficient for battery power sensor usage, but they are good enough to be used as initial proof of concept.

To be able to program the Digispark board, I had to use the Arduino IDE, and through the IDE Boards Manager, add support for them, so that these new boards are available to be selected and programmed.

Now, this bring two problems:

– The first one is that after selecting on the IDE the board type, every window instance of the IDE assumes the same board. This means that I can’t have side by side one Arduino IDE window for the RF gateway based Atmega328p board, and other window for the AtTinny85 sensor board. I have to constantly change board types depending of what code I’m working for. A good solution (as the Platformio uses) should to associate the board type to the project, but that is not possible on the Arduino IDE.

– The second problem, is that the last Arduino IDE tools update broke the integration between the native Arduino boards and the Digispark based boards. I can have support for one of them or the other, but not both at the same time, otherwise I get errors. There are some discussions on the Arduino forums that acknowledge the issues that I’m having.

Still I could use one IDE/editor for one type of board, and the Arduino IDE for the Attinny boards, but is not very efficient. Anyway, the Arduino IDE is too much hassle when complexity starts to grow. I’m already using the Netbeans IDE for programming the ESP8266 and the KDE Kate editor for some Arduino basic programming, so that all I need was something that supported the Digispark AtTinny85 toolset.

And so, I have several problems, which means I need to look for a solution, and preferably one that unifies all the platforms.

Platformio and Platformio IDE

Platformio is an open source toolset that allows, using the same base tools, to target different target environments: Atmel/Arduino, Espressif ESP8266, ARM, and so on.

This means that from a unified toolset/IDE I can target different platforms, and one important thing, the target is defined by project and not by the tool or IDE, which solves my first problem.

Also Platformio, since it supports out of the box several targets, it probably also solves problem number two of having possible future clashes between different device platforms/architectures.

Platformio is a command line based tool, and associated with it there is an IDE where development can take place on a modern editor (Atom) that, among another things, supports code completion, serial port monitoring, embedded terminal, and so on…

The command line tool supports searching and installing support for the several boards available on the market, and also allows to search and install user contributed libraries.

Anyway, the Platformio docs can explain better the purpose and capabilities of these tools, but the greatest achievement of this is that allows an unified toolset to be used for different boards/targets.

Keep in mind that there are at least two tools:

– Platformio – This is a Python based command line tool that unifies the compiling, uploading, library management, and so on.
– Platformio IDE – This is a NodeJS, Atom Editor based IDE that integrates the Platform tools on the IDE.

While I had no issues, worth of mention, on Arch Linux, in using Platformio cli tools, the IDE has a lot of issues, not due to Platformio IDE, but due to Atom editor and supporting software (Electron). I’m still not able to use the IDE to it’s full potential, but as an editor that has code completion and project management it works fine, but so far for me, upload to the boards must be done through the command line platformio tools.

Installing Platformio on Arch Linux
So I’m running Arch Linux, which by definition is quite near bleeding edge… There are instructions for other platforms, and so it is my take on the installation on Arch:

The main platformio package is available on the AUR repository, so just install it with pacaur or yaourt:

 yaourt -S platformio

We should then have the command line tools:

root@pcortex:~# pio
Usage: pio [OPTIONS] COMMAND [ARGS]...

Options:
  --version          Show the version and exit.
  -f, --force        Force to accept any confirmation prompts.
  -c, --caller TEXT  Caller ID (service).
  -h, --help         Show this message and exit.

Commands:
  boards       Pre-configured Embedded Boards
  ci           Continuous Integration
  init         Initialize new PlatformIO based project
  lib          Library Manager
  platforms    Platforms and Packages Manager
  run          Process project environments
  serialports  List or Monitor Serial ports
  settings     Manage PlatformIO settings
  update       Update installed Platforms, Packages and Libraries
  upgrade      Upgrade PlatformIO to the latest version

To start a simple arduino project we can first install the Atmel AVR platform:

root@pcortex:~#  pio platforms install atmelavr
Installing toolchain-atmelavr package:
Downloading  [####################################]  100%             
Unpacking  [####################################]  100%             
The platform 'atmelavr' has been successfully installed!
The rest of packages will be installed automatically depending on your build environment.

We can search for available platforms with pio platforms search

 mkdir myproject
 cd myproject
 pio init --board uno

And that’s it. We can start to edit the src.main.cpp file, add libraries to the lib directory, execute pio run to compile, and pio run -t upload to upload to the board.

We can see further instructions here

And that basically it for the Command Line tools.

For the IDE:

Install from the main repository the clang and atom editor. Minicom is to have the Serial port monitoring from the IDE (or not):

Edit: Do not install atom editor from the main repository. Install atom-editor-bin from AUR instead. Many problems are solved with the AUR version. You may first install the editor from the main repositories so that all possible dependencies are pulled first, and then remove it with pacman -R atom apm and install the AUR version with yaourt -S atom-editor-bin

root@pcortex:~# pacman -S clang atom minicom
resolving dependencies...
looking for conflicting packages...

Packages (11) apm-1.12.6-1  electron-1.3.3-1  http-parser-2.7.1-1  libuv-1.9.1-1  minizip-1:1.1-1  nodejs-6.4.0-1  npm-3.10.6-1  re2-20160301-1  semver-5.3.0-1  atom-1.9.8-3  clang-3.8.1-1 minicom-2.7-2

This will bring also the node-js and electron platforms.

We can now start the Atom editor to add the package Platformio-IDE. Installing the package Platformio-IDE will also pull the Platformio-IDE-Terminal.

root@pcortex:~# atom

To clear the error (if it appears) that the atom editor can’t watch the .atom/config.cson file, execute also the following command:

sudo sysctl fs.inotify.max_user_watches=32768

In my case, after starting Atom, the main window appears, but nothing else works. For example, going to Edit->Preferences to try add the Platformio-IDE package does nothing. The same applies to other menu options. On the other hand, running atom as root, seems to work, but is not a solution.

Starting atom on the foreground (atom -f ) I can see the following error:

  TypeError: Path must be a string. Received undefined", source: path.js (7)

What I’ve found out is that if we open a file passed through the command line, close atom, and start it again without any parameter, it starts to work…

So, just do, for example:

 root@pcortex:~# atom somefile.txt

 Close atom, and start it again:

 root@pcortex:~# atom 

The menus should start to work and we should be to install the platformio-ide package through the IDE Graphical Package Manager. Just go to Edit->Settings->Install search for Platformio and add Platformio IDE. The Platformio IDE Terminal will also be installed automatically.

If, as in my case, we are behind a corporate proxy, we set the proxy environment variables on a terminal session, and start atom from there.

PlatformIO Instalation

After installation the Platformio menu and toolbar should appear.

One thing that I’ve found out was that the terminal window and serial port monitor wouldn’t work. In one of my machines the window just opens and stays blank with a blinking cursor. On other machine, an error appears saying that Platformio Terminal is not installed, which is not the case. In this last machine the error that appears with atom -f is:

 "Failed to require the main module of 'platformio-ide-terminal' because it requires an incompatible native module

On the first situation, the window only with the blinking cursor, pressing CTRL-SHIFT-I to open the debugger and viewing the console, an error like this is shown:

/usr/lib/atom/src/task.js:52 Cannot find module '../bin/linux/x64.m49.node' Error: Cannot find module '../bin/linux/x64.m49.node'
    at Function.Module._resolveFilename (module.js:440:15)
    at Function.Module._load (module.js:388:25)
    at Module.require (module.js:468:17)
    at require (internal/module.js:20:19)
    at Object. (/home/fdam/.atom/packages/platformio-ide-terminal/node_modules/pty.js/lib/pty.js:18:9)
    at Module._compile (module.js:541:32)
    at Object.value [as .js] (/usr/lib/atom/src/compile-cache.js:208:21)
    at Module.load (module.js:458:32)
....

What I’ve done to solve this:

– Goto ~/.atom/packages/plataformio-ide-terminal
– Delete completely the node_modules directory: rm -rf node_modules
– Install nslog: npm install nslog
– Edit the package json file, and change the nan version from 2.0.5 to >2.0.5

...
      {
        "name": "nan",
        "version": ">2.0.5",
        "path": "node_modules/nan/include_dirs.js"
      },
...

– Install the packages: npm install.
– It should error on the pty.js package. Do not worry (yet…)
– Goto node_modules/pty.js and edit the package.json file. Change the version of nan from 2.0.5 to >2.0.5

  "dependencies": {
    "extend": "~1.2.1",
    "nan": ">2.0.5"
  },

– Remove the node_modules directory (for the pty.js): rm -rf node_modules
– Check what is our electron version: electron -v
– In my case it is v1.3.3
– Paste the following lines on the terminal:

# Electron's version.
export npm_config_target=1.3.3
# The architecture of Electron, can be ia32 or x64.
export npm_config_arch=x64
# Download headers for Electron.
export npm_config_disturl=https://atom.io/download/atom-shell
# Tell node-pre-gyp that we are building for Electron.
export npm_config_runtime=electron
# Tell node-pre-gyp to build module from source code.
export npm_config_build_from_source=true
# Install all dependencies, and store cache to ~/.electron-gyp.
HOME=~/.electron-gyp npm install

Start again the atom editor. The terminal should work now. If not, atom might complain and show a red icon bug on the bottom right side. Just press it, and choose module rebuild, restart atom and it should be ok.

Conclusion

While the installation and usage of the command line tools is straight forward and it works out of the box, the Atom based IDE is another story. It has a steep installation curve, not Platformio fault, but due to the number of components involved. Also those issues might be due to my Linux distribution (Arch), but still, it might be a real show stopper for some users if this happens on other distributions. I’ve lost some serious hours debugging this🙂 to arrive to an almost fully functional IDE.

Anyway at the end, the platform and the IDE are fantastic. With code completion, platformio tools seamlessly integrated, including simultaneous serial port monitoring to different boards, support for different targets and so on, is really a great product.

Platformio is highly recommended as also the IDE, despite it’s rough edges.

Communication over 433Mhz links

I’ve bought, about a year ago, maybe more, some 433Mhz transmitter and receivers , so I could build a mailbox monitoring solution, loosely based on LOFI Project. The base idea was when someone put something in my mailbox I was notified. Anyway, the ESP8266 came along, and after some experiments, there is no way that the ESP8266 can be used for implementing the monitoring mailbox project I was thinking due to power consumption but mainly because the high distance across several floors and walls between the mailbox and my access point. So back to basics and to the original simpler idea of using plain 433Mhz link for transmitting data.

I’ve started to do some experiments using these devices:

eBay 433Mhz RX TX

All I can say about these is while the transmitter is ok and works fine across floors and walls (I can see the signal clearly using my SDR), the receiver is absolute garbage and useless for the intended purpose.

The receiver is only able to receive in line of sight with the emitter when there are no obstacles, and even in this scenario with both emitter an receiver with attached antennas, the maximum distance between the emitter and the receiver is 4/5 meters at maximum.

The solution is to use the much better and higher cost RXB8 receiver that has according to the datasheet -114dBm sensitivity, but it does work and is very good.

rxb8

The code for interfacing with this receiver is the same code that works with the cheaper receiver and is the Arduino Manchester encoding library.

Using the RXB8 with an attached 433Mhz antenna (I’m using an antena with 5dBi and a SMA connector), the results are simply superior, with the end result that the signal/messages when the emitter is located several floors down and across several walls, are received correctly and are able to be decoded by the Arduino.

Anyway, one of the interesting things on the LOFI project is/was the Hamming Error Correction implementation found here in the RobotRoom blog.

So I’ve forked the original mchr3k Arduino Manchester encoding library and added the Hamming Code Error correction support: Arduino Manchester encoding library with Hamming EC support.

The usage is quite simple, and can be seen on the examples Hamming_TX e Hamming_RX.

The two new functions are:

uint8_t Manchester::EC_encodeMessage( uint8_t numBytes, uint8_t *data, uint8_t *ecout)

where the input data buffer is provided by the data parameter with the buffer size also provided by the numBytes parameter. The output buffer ecout should at least be one third bigger than the original buffer since for every two input bytes an additional parity byte is added.

For receiving the buffer with the data and parity data is decoded and corrected if any errors are detected. Errors are corrected if possible.

uint8_t Manchester::EC_decodeMessage( uint8_t numBytes, uint8_t *ecin, uint8_t *bytesOut, uint8_t *dataout )

The input data is the size and the buffer with the data and parity, namely the numBytes and ecin input buffer pointer, and the output is returned on the dataout buffer, with the decoded size also returned on the bytesOut parameter.

Above these functions we can build some logical protocol for the received data, but that really depends on the way we want to use the library.

ESP8266 – Logging data in a backend – AES and Crypto-JS

After building, on the previous posts, the Node-Red based backend to support E2EE (End to End Encryption) so we can log data into a central server/database, from our devices securely, without using HTTPS, we need now to build the firmware for the ESP8266 that allows it to call our E2EE backend.

The firmware for the ESP8266 must gather the data that it wants to send, get or generate the current sequence number for the data (to avoid replay attacks), encrypt the data and send it to the backend.
On the backend we are using the Java script library for cryptographic functions Crypto-js, and specifically we are encrypting data with the encryption algorithm AES. So all we need is to encrypt our data with AES on the ESP8266, send it to the Node-Red Crypto-js backend, decrypt it and store it, easy right?

Not quite, let’s see why:

Crypto-js and AES:
We can see that on my Node-Red function code and testing programs I’m using something similar to the following code example:

var CryptoJS = require("crypto-js");
var message  = "Message to encrypt";
var AESKey   = '2B7E151628AED2A6ABF7158809CF4F3C';

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text in Base64: " ,  ciphertext.toString(CryptoJS.enc.base64) );

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Several points regarding the above code need clarification:

The code variable AESKey the way it is used on the above example encrypt and decrypt functions isn’t really a key but a passphrase from where the real key and an initialization vector or salt value is generated (I’m using the names interchangeably, but they are not exactly the same thing except they are public viewable data that must change over time/calls).
The use for the generated key is self explanatory, but the initialization vector (IV) or salt value is used to allow that the encrypted data output to be not the same for the same initial message. While the key is kept constant and secret to both parties, the IV/salt changes between calls, which means that the above code, when run multiple times, will never produce the same output for the same initial message.

Still referring to the above code, the algorithm that generates the key from the passphrase is the PBKDF2 algorithm. More info at Crypto-js documentation. At the end of the encryption the output is a OpenSSL salted format that means that the output begins by the signature id: Salted_, followed by an eight byte salt value, and after the salt, the encrypted data.

So if we want use the API has above on the node-js/crypto-js side, we need to implement on the ESP8266 side both the AES and PBKDF2 algorithms.

I decided not to do that, first because finding a C/C++ implementation of the PBKDF2 algorithm that could be portable and worked on the ESP822 proved difficult, and second the work for porting it to the ESP8266 won’t be needed if I use a KEY/IV directly, and so I decided to use the more standard way of providing an AES key and an initialization vector for encrypting and decrypting data.

In the case of Node-JS and Crypto-JS when using an explicit key and IV the code looks like:

var CryptoJS = require("crypto-js");
var request = require('request');

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":145 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);

var iv = CryptoJS.enc.Hex.parse('0000000000000000');
var key= CryptoJS.enc.Hex.parse(AESKey);

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, key , { iv: iv } );

//console.log("Cypher: ", ciphertext );
console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");

console.log("=============================================================================");
console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Now, with above code, where the IV is always initialized to the same value, in this case ‘0000000000000000’, we can see when running the above code several times that the output is always the same since the IV is kept constant. Also the encrypted output is now just the raw encrypted data and not the Openssl format.

So to make the above code secure we must randomize the IV value for producing an output that is always different, even from several program runs when encrypting the same source data.

As a final note, if we count the number of HEX characters on the Key string, we find out that they are 16 bytes, which gives a total of 128 key bits. So the above example is using AES128 encryption, and with default Crypto-js block mode and padding algorithms which are CBC (Chain block mode) and pkcs7.

Interfacing Crypto-js and the ESP8266:
Since we are using AES for encrypting data and decrypting data, we need first to have an AES library for the ESP8266. The AES library that I’m using is this one Spaniakos AES library for Arduino and RPi. This library uses AES128, CBC and pkcs7 padding, so it ticks all boxes for compatibility with Crypto-js…

I just added the code from the above library to my Sming project and also added this Base64 library so that I can encode to and from Base64.

The only remaining issue was to securely generate a truly random initialization vector. And while at first I’ve used some available libraries to generate pseudo-random numbers to real random numbers, I’ve found out that the ESP8266 seems to have already a random number generator that is undocumented: Random number generator

So to generate a random IV value is as easy as:

uint8_t getrnd() {
    uint8_t really_random = *(volatile uint8_t *)0x3FF20E44;
    return really_random;
}

// Generate a random initialization vector
void gen_iv(byte  *iv) {
    for (int i = 0 ; i < N_BLOCK ; i++ ) {
        iv[i]= (byte) getrnd();
    }

So our ESP8266 code is as follow:

Global variables declarations:
The N_Block defines the encryption block size, that for AES128 is 16 bytes.

#include "AES.h"
#include "base64.h"

// The AES library object.
AES aes;

// Our AES key. Note that is the same that is used on the Node-Js side but as hex bytes.
byte key[] = { 0x2B, 0x7E, 0x15, 0x16, 0x28, 0xAE, 0xD2, 0xA6, 0xAB, 0xF7, 0x15, 0x88, 0x09, 0xCF, 0x4F, 0x3C };

// The unitialized Initialization vector
byte my_iv[N_BLOCK] = { 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};

// Our message to encrypt. Static for this example.
String msg = "{\"data\":{\"value\":300}, \"SEQN\":700 , \"msg\":\"IT WORKS!!\" }";


The example function ESP8266 Sming code is:

void testAES128()  {

    char b64data[200];
    byte cipher[1000];
    byte iv [N_BLOCK] ;
    
    Serial.println("Let's encrypt:");
    
    aes.set_key( key , sizeof(key));  // Get the globally defined key
    gen_iv( my_iv );                  // Generate a random IV
    
    // Print the IV
    base64_encode( b64data, (char *)my_iv, N_BLOCK);
    Serial.println(" IV b64: " + String(b64data));
       
    Serial.println(" Mensagem: " + msg );
 
    int b64len = base64_encode(b64data, (char *)msg.c_str(),msg.length());
    Serial.println (" Message in B64: " + String(b64data) );
    Serial.println (" The lenght is:  " + String(b64len) );
    
    // For sanity check purpose
    //base64_decode( decoded , b64data , b64len );
    //Serial.println("Decoded: " + String(decoded));
    
    // Encrypt! With AES128, our key and IV, CBC and pkcs7 padding    
    aes.do_aes_encrypt((byte *)b64data, b64len , cipher, key, 128, my_iv);
    
    Serial.println("Encryption done!");
    
    Serial.println("Cipher size: " + String(aes.get_size()));
    
    base64_encode(b64data, (char *)cipher, aes.get_size() );
    Serial.println ("Encrypted data in base64: " + String(b64data) );
      
    Serial.println("Done...");
}

When the above code/function is executed on the ESP8266 it outputs the following:

Let's encrypt:
 IV b64: cAFviaDMHejlteGn9/4eQQ==
 Mensagem: {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }
 Message in B64: eyJkYXRhIjp7InZhbHVlIjozMDB9LCAiU0VRTiI6NzAwICwgIm1zZyI6IklUIFdPUktTISIgfQ==
 The lenght is:  76
Encryption done!
Cipher size: 80
Encrypted data in base64: /1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=
Done...

And now we can feed the above Base64 IV and encrypted data to our decoding program in Node-Js using Crypto-JS:

var CryptoJS = require("crypto-js");
var request = require('request');

var esp8266_msg = '/1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=';
var esp8266_iv  = 'cAFviaDMHejlteGn9/4eQQ==';

// The AES encryption/decryption key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

var plain_iv =  new Buffer( esp8266_iv , 'base64').toString('hex');
var iv = CryptoJS.enc.Hex.parse( plain_iv );
var key= CryptoJS.enc.Hex.parse( AESKey );

console.log("Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt( esp8266_msg, key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Base64);
var decoded_b64msg =  new Buffer(plaintext , 'base64').toString('ascii');
var decoded_msg =     new Buffer( decoded_b64msg , 'base64').toString('ascii');

console.log("Decrypted message: ", decoded_msg);

and the output is:

Decrypted message:  {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }

So, as the message shows, it WORKS!. AES encryption on the ESP8266 to a Node-JS Crypto-JS based code where decryption occurs.

Final notes:
So all is needed now is to build on the ESP8266 side the message with the encrypted data and the IV, and send through plain HTTP a JSON object to the Node-Red back end.

{
  "msg":"/1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=",
  "iv":"cAFviaDMHejlteGn9/4eQQ=="
}

On the Node-Red back end the decryption can now be done easily as was shown on the above testing code.