Communication over 433Mhz links

I’ve bought, about a year ago, maybe more, some 433Mhz transmitter and receivers , so I could build a mailbox monitoring solution, loosely based on LOFI Project. The base idea was when someone put something in my mailbox I was notified. Anyway, the ESP8266 came along, and after some experiments, there is no way that the ESP8266 can be used for implementing the monitoring mailbox project I was thinking due to power consumption but mainly because the high distance across several floors and walls between the mailbox and my access point. So back to basics and to the original simpler idea of using plain 433Mhz link for transmitting data.

I’ve started to do some experiments using these devices:

eBay 433Mhz RX TX

All I can say about these is while the transmitter is ok and works fine across floors and walls (I can see the signal clearly using my SDR), the receiver is absolute garbage and useless for the intended purpose.

The receiver is only able to receive in line of sight with the emitter when there are no obstacles, and even in this scenario with both emitter an receiver with attached antennas, the maximum distance between the emitter and the receiver is 4/5 meters at maximum.

The solution is to use the much better and higher cost RXB8 receiver that has according to the datasheet -114dBm sensitivity, but it does work and is very good.

rxb8

The code for interfacing with this receiver is the same code that works with the cheaper receiver and is the Arduino Manchester encoding library.

Using the RXB8 with an attached 433Mhz antenna (I’m using an antena with 5dBi and a SMA connector), the results are simply superior, with the end result that the signal/messages when the emitter is located several floors down and across several walls, are received correctly and are able to be decoded by the Arduino.

Anyway, one of the interesting things on the LOFI project is/was the Hamming Error Correction implementation found here in the RobotRoom blog.

So I’ve forked the original mchr3k Arduino Manchester encoding library and added the Hamming Code Error correction support: Arduino Manchester encoding library with Hamming EC support.

The usage is quite simple, and can be seen on the examples Hamming_TX e Hamming_RX.

The two new functions are:

uint8_t Manchester::EC_encodeMessage( uint8_t numBytes, uint8_t *data, uint8_t *ecout)

where the input data buffer is provided by the data parameter with the buffer size also provided by the numBytes parameter. The output buffer ecout should at least be one third bigger than the original buffer since for every two input bytes an additional parity byte is added.

For receiving the buffer with the data and parity data is decoded and corrected if any errors are detected. Errors are corrected if possible.

uint8_t Manchester::EC_decodeMessage( uint8_t numBytes, uint8_t *ecin, uint8_t *bytesOut, uint8_t *dataout )

The input data is the size and the buffer with the data and parity, namely the numBytes and ecin input buffer pointer, and the output is returned on the dataout buffer, with the decoded size also returned on the bytesOut parameter.

Above these functions we can build some logical protocol for the received data, but that really depends on the way we want to use the library.

ESP8266 – Logging data in a backend – AES and Crypto-JS

After building, on the previous posts, the Node-Red based backend to support E2EE (End to End Encryption) so we can log data into a central server/database, from our devices securely, without using HTTPS, we need now to build the firmware for the ESP8266 that allows it to call our E2EE backend.

The firmware for the ESP8266 must gather the data that it wants to send, get or generate the current sequence number for the data (to avoid replay attacks), encrypt the data and send it to the backend.
On the backend we are using the Java script library for cryptographic functions Crypto-js, and specifically we are encrypting data with the encryption algorithm AES. So all we need is to encrypt our data with AES on the ESP8266, send it to the Node-Red Crypto-js backend, decrypt it and store it, easy right?

Not quite, let’s see why:

Crypto-js and AES:
We can see that on my Node-Red function code and testing programs I’m using something similar to the following code example:

var CryptoJS = require("crypto-js");
var message  = "Message to encrypt";
var AESKey   = '2B7E151628AED2A6ABF7158809CF4F3C';

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text in Base64: " ,  ciphertext.toString(CryptoJS.enc.base64) );

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Several points regarding the above code need clarification:

The code variable AESKey the way it is used on the above example encrypt and decrypt functions isn’t really a key but a passphrase from where the real key and an initialization vector or salt value is generated (I’m using the names interchangeably, but they are not exactly the same thing except they are public viewable data that must change over time/calls).
The use for the generated key is self explanatory, but the initialization vector (IV) or salt value is used to allow that the encrypted data output to be not the same for the same initial message. While the key is kept constant and secret to both parties, the IV/salt changes between calls, which means that the above code, when run multiple times, will never produce the same output for the same initial message.

Still referring to the above code, the algorithm that generates the key from the passphrase is the PBKDF2 algorithm. More info at Crypto-js documentation. At the end of the encryption the output is a OpenSSL salted format that means that the output begins by the signature id: Salted_, followed by an eight byte salt value, and after the salt, the encrypted data.

So if we want use the API has above on the node-js/crypto-js side, we need to implement on the ESP8266 side both the AES and PBKDF2 algorithms.

I decided not to do that, first because finding a C/C++ implementation of the PBKDF2 algorithm that could be portable and worked on the ESP822 proved difficult, and second the work for porting it to the ESP8266 won’t be needed if I use a KEY/IV directly, and so I decided to use the more standard way of providing an AES key and an initialization vector for encrypting and decrypting data.

In the case of Node-JS and Crypto-JS when using an explicit key and IV the code looks like:

var CryptoJS = require("crypto-js");
var request = require('request');

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":145 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);

var iv = CryptoJS.enc.Hex.parse('0000000000000000');
var key= CryptoJS.enc.Hex.parse(AESKey);

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, key , { iv: iv } );

//console.log("Cypher: ", ciphertext );
console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");

console.log("=============================================================================");
console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);

Now, with above code, where the IV is always initialized to the same value, in this case ‘0000000000000000’, we can see when running the above code several times that the output is always the same since the IV is kept constant. Also the encrypted output is now just the raw encrypted data and not the Openssl format.

So to make the above code secure we must randomize the IV value for producing an output that is always different, even from several program runs when encrypting the same source data.

As a final note, if we count the number of HEX characters on the Key string, we find out that they are 16 bytes, which gives a total of 128 key bits. So the above example is using AES128 encryption, and with default Crypto-js block mode and padding algorithms which are CBC (Chain block mode) and pkcs7.

Interfacing Crypto-js and the ESP8266:
Since we are using AES for encrypting data and decrypting data, we need first to have an AES library for the ESP8266. The AES library that I’m using is this one Spaniakos AES library for Arduino and RPi. This library uses AES128, CBC and pkcs7 padding, so it ticks all boxes for compatibility with Crypto-js…

I just added the code from the above library to my Sming project and also added this Base64 library so that I can encode to and from Base64.

The only remaining issue was to securely generate a truly random initialization vector. And while at first I’ve used some available libraries to generate pseudo-random numbers to real random numbers, I’ve found out that the ESP8266 seems to have already a random number generator that is undocumented: Random number generator

So to generate a random IV value is as easy as:

uint8_t getrnd() {
    uint8_t really_random = *(volatile uint8_t *)0x3FF20E44;
    return really_random;
}

// Generate a random initialization vector
void gen_iv(byte  *iv) {
    for (int i = 0 ; i < N_BLOCK ; i++ ) {
        iv[i]= (byte) getrnd();
    }

So our ESP8266 code is as follow:

Global variables declarations:
The N_Block defines the encryption block size, that for AES128 is 16 bytes.

#include "AES.h"
#include "base64.h"

// The AES library object.
AES aes;

// Our AES key. Note that is the same that is used on the Node-Js side but as hex bytes.
byte key[] = { 0x2B, 0x7E, 0x15, 0x16, 0x28, 0xAE, 0xD2, 0xA6, 0xAB, 0xF7, 0x15, 0x88, 0x09, 0xCF, 0x4F, 0x3C };

// The unitialized Initialization vector
byte my_iv[N_BLOCK] = { 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};

// Our message to encrypt. Static for this example.
String msg = "{\"data\":{\"value\":300}, \"SEQN\":700 , \"msg\":\"IT WORKS!!\" }";


The example function ESP8266 Sming code is:

void testAES128()  {

    char b64data[200];
    byte cipher[1000];
    byte iv [N_BLOCK] ;
    
    Serial.println("Let's encrypt:");
    
    aes.set_key( key , sizeof(key));  // Get the globally defined key
    gen_iv( my_iv );                  // Generate a random IV
    
    // Print the IV
    base64_encode( b64data, (char *)my_iv, N_BLOCK);
    Serial.println(" IV b64: " + String(b64data));
       
    Serial.println(" Mensagem: " + msg );
 
    int b64len = base64_encode(b64data, (char *)msg.c_str(),msg.length());
    Serial.println (" Message in B64: " + String(b64data) );
    Serial.println (" The lenght is:  " + String(b64len) );
    
    // For sanity check purpose
    //base64_decode( decoded , b64data , b64len );
    //Serial.println("Decoded: " + String(decoded));
    
    // Encrypt! With AES128, our key and IV, CBC and pkcs7 padding    
    aes.do_aes_encrypt((byte *)b64data, b64len , cipher, key, 128, my_iv);
    
    Serial.println("Encryption done!");
    
    Serial.println("Cipher size: " + String(aes.get_size()));
    
    base64_encode(b64data, (char *)cipher, aes.get_size() );
    Serial.println ("Encrypted data in base64: " + String(b64data) );
      
    Serial.println("Done...");
}

When the above code/function is executed on the ESP8266 it outputs the following:

Let's encrypt:
 IV b64: cAFviaDMHejlteGn9/4eQQ==
 Mensagem: {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }
 Message in B64: eyJkYXRhIjp7InZhbHVlIjozMDB9LCAiU0VRTiI6NzAwICwgIm1zZyI6IklUIFdPUktTISIgfQ==
 The lenght is:  76
Encryption done!
Cipher size: 80
Encrypted data in base64: /1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=
Done...

And now we can feed the above Base64 IV and encrypted data to our decoding program in Node-Js using Crypto-JS:

var CryptoJS = require("crypto-js");
var request = require('request');

var esp8266_msg = '/1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=';
var esp8266_iv  = 'cAFviaDMHejlteGn9/4eQQ==';

// The AES encryption/decryption key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

var plain_iv =  new Buffer( esp8266_iv , 'base64').toString('hex');
var iv = CryptoJS.enc.Hex.parse( plain_iv );
var key= CryptoJS.enc.Hex.parse( AESKey );

console.log("Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt( esp8266_msg, key , { iv: iv} );
var plaintext = bytes.toString(CryptoJS.enc.Base64);
var decoded_b64msg =  new Buffer(plaintext , 'base64').toString('ascii');
var decoded_msg =     new Buffer( decoded_b64msg , 'base64').toString('ascii');

console.log("Decrypted message: ", decoded_msg);

and the output is:

Decrypted message:  {"data":{"value":300}, "SEQN":700 , "msg":"IT WORKS!" }

So, as the message shows, it WORKS!. AES encryption on the ESP8266 to a Node-JS Crypto-JS based code where decryption occurs.

Final notes:
So all is needed now is to build on the ESP8266 side the message with the encrypted data and the IV, and send through plain HTTP a JSON object to the Node-Red back end.

{
  "msg":"/1aZRwVaw3jv+ct8HS4pCV5lThvTG70M90ARiyAsIDYMkfJE3w8F3bgxaOKVA0rX4m1Mq50VVN0u9gRw9F2gKE4r2OcY8oECv8bKT80F9pY=",
  "iv":"cAFviaDMHejlteGn9/4eQQ=="
}

On the Node-Red back end the decryption can now be done easily as was shown on the above testing code.

CubicSDR and SoapyRemote on Orange PI PC

So I’m using remotely my RTL-SDR dongle connected to an Orange PI PC using GQrx on my desktop computer while on the Orange Pi PC I’m running the rtl_tcp server.

This combination works fine, but still some times I have some lag, not common, but it happens.

Anyway Gqrx is great, but I wanted to try another SDR programs, and one of those programs is Cubic SDR. Cubic SDR uses an abstration layer for accessing the SDR hardware either locally connected or over the network. So I wanted to see how the behaviour of CubicSdr, comparing to Gqrx when accessing the RTLSDR over the network.

Installing CubicSDR on the Desktop
I’ve not used the available binaries, but I’ve used the code from the Git repository: https://github.com/cjcliffe/CubicSDR. So far the git repository works fine.

I’m not posting here the instructions for building the CubicSDR because the full instructions are at the Cubic SDR wiki.

Just make sure, when compiling, to give the correct path to wxWidgets when building Cubic SDR.

So at the desktop, we need to obtain, build and install the following components:

  • SoapySDR – Abstraction layer
  • Liquid-dsp – The digital signal processing libs
  • wxWidgets – The display widgets
  • CubicSDR – The Sdr program itself
  • SoapyRTLSDR – The Soapy abstraction layer driver for the RTLSDR USB type dongles. To use the dongles locally

Installing the above software we can use the attached locally RTLSDR dongle.

Installing SoapySDR Remote on the remote server
The remote server where my RTLSDR dongle is connected is an Orange Pi PC running Armbian. To let my desktop running CubicSDR program to access remotely the dongle, we need to install SoapySDR Remote that allows remote access to the SDR.

So at the remote server we need to obtain, build and install the following components:

  • SoapySDR – The abstraction layer
  • SoapyRTLSDR – The driver for our RTL USB dongle
  • SoapySDR Remote – The server for remotely access the RTL dongle

So basically the instructions are something like this:

  mkdir ~/SDR 
  cd ~/SDR
  git clone https://github.com/pothosware/SoapySDR.git
  cd SoapySDR/
  mkdir build
  cd build
  cmake ../ -DCMAKE_BUILD_TYPE=Release
  make -j4
  sudo make install
  sudo ldconfig
  SoapySDRUtil --info

Building the SoapySDR remote:

 cd ~/SDR
 git clone https://github.com/pothosware/SoapyRemote.git
 cd SoapyRemote/
 cd build
 cmake ..
 make
 sudo make install

and build the RTLSDR driver:

  cd ~/SDR
  sudo apt-get install librtlsdr-dev
  git clone https://github.com/pothosware/SoapyRTLSDR.git
  cd SoapyRTLSDR/
  mkdir build
  cd build
  cmake .. -DCMAKE_BUILD_TYPE=Release
  make
  sudo make install
  sudo ldconfig

So at the end we should have the following outputs:

opi@opi:~# SoapySDRUtil --probe
######################################################
## Soapy SDR -- the SDR abstraction library
######################################################

Probe device 
Found Rafael Micro R820T tuner
Found Rafael Micro R820T tuner

----------------------------------------------------
-- Device identification
----------------------------------------------------
  driver=RTLSDR
  hardware=RTLSDR
  origin=https://github.com/pothosware/SoapyRTLSDR
  rtl=0

----------------------------------------------------
-- Peripheral summary
----------------------------------------------------
  Channels: 1 Rx, 0 Tx
  Timestamps: NO
  Other Settings:
     * Direct Sampling - RTL-SDR Direct Sampling Mode
       [key=direct_samp, default=0, type=string, options=(0, 1, 2)]
     * Offset Tune - RTL-SDR Offset Tuning Mode
       [key=offset_tune, default=false, type=bool]
     * I/Q Swap - RTL-SDR I/Q Swap Mode
       [key=iq_swap, default=false, type=bool]

----------------------------------------------------
-- RX Channel 0
----------------------------------------------------
  Full-duplex: YES
  Supports AGC: YES
  Stream formats: CS8, CS16, CF32
  Native format: CS8 [full-scale=128]
  Stream args:
     * Buffer Size - Number of bytes per buffer, multiples of 512 only.
       [key=bufflen, units=bytes, default=16384, type=int]
     * Buffer Count - Number of buffers per read.
       [key=buffers, units=buffers, default=15, type=int]
  Antennas: RX
  Full gain range: [0, 49.6] dB
    TUNER gain range: [0, 49.6] dB
  Full freq range: [23.999, 1764] MHz
    RF freq range: [24, 1764] MHz
    CORR freq range: [-0.001, 0.001] MHz
  Sample rates: [0.25, 3.2] MHz

And SoapySDR should have the following configuration:

opi@opi:~# SoapySDRUtil --info
######################################################
## Soapy SDR -- the SDR abstraction library
######################################################

API Version: v0.5.0-gfec33c63
ABI Version: v0.5-2
Install root: /usr/local
Module found: /usr/local/lib/SoapySDR/modules/libremoteSupport.so
Module found: /usr/local/lib/SoapySDR/modules/librtlsdrSupport.so
Loading modules... done
Available factories...null, remote, rtlsdr, 

So all we need is to start our server:

opi@opi:~# SoapySDRServer --bind
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

ba73bc08-3d0f-1458-8337-93d5a4502801
Launching the server... tcp://[::]:55132
Server bound to [::]:55132
Launching discovery server... 
Press Ctrl+C to stop the server

Using CubicSDR and SoapySDRRemote
So all we need is now on the startup SDR device screen selection add (by pressing the Add button) the remote IP of the Orange Pi PC server to access remotely the RTLSDR dongle.

My Orange PI PC IP address is 192.168.1.19:

CubicSDR device selection

CubicSDR device selection

And here it is the CubicSDR in action.

CubicSDR in action

CubicSDR in action

Conclusion and final notes
The CPU usage and temperature on the Orange PI PC is not a problem when using the server. CPU usage floats around 40%, and no meaningful or worrying changes on the CPU temperature. So the Orange PI PC is up to the task without any issues when serving data with SoapySDRRemote.

Also with CubicSDR and SoapySDRRemote, I’ve experienced no lag when changing frequencies, namely when dragging the frequency selector.. It seems that all changes are instantaneous and note that my desktop connects to the remote server through a 200Mbps PLC and only then it is cable network to the Orange Pi. According to my desktop PC network widget, when receiving data, I have around 6.5Mbps of data comming in when using the maximum sample rate of 3.2MHz.

Also it took me a while to get used to the CubicSDR user interface, but overall for things like fine tuning, since it has a dedicaded codec screen, is much better than Gqrx.

CubicSDR  fine tuning

CubicSDR fine tuning

Also one great feature is if we keep dragging the spectrogram window, the central frequency changes so it keeps up with the SDR bandwidth, shile in Gqrx we need to dial in.

Still I’m using Gqrx and rtl_tcp since CubicSDR has no data output, other then piping audio. Gqrx can pipe to UDP, that allows the decoding of digital modes locally or on other servers without messing around with PulseAudio and Jackd.
Also bookmarking isn’t as direct/easy as with Gqrx. Not sure if I can give labels/names to bookmarks and search them, like I can in Gqrx, but then the problem might be between the chair and computer…

Anyway CubicSDR is a great SDR application and the future looks bright.

I do recommend to give it a test drive.

ESP8266 and the Micropython firmware

One of the alternative firmwares available for the ESP8266 is MicroPython Python interpreter. I’ve found by chance a great tutorial at Adafruit for building the Micropython firmware and I thought to give it a try.

Building the firmware:
The Adafruit tutorial uses a Vagrant based virtual machine to build the firmware, but since I’m already running Linux (Arch Linux to be more specific) and already have the Falcon open ESP8266 sdk installed (see here) and the esptool.py also available since I’m using the Sming firmware, I’ve just downloaded only the latest Micropython source code from the Github repository to a local directory.

cd ~
git clone https://github.com/micropython/micropython
cd ~/micropython
git submodule update --init
cd ~/micropython/esp8266
export PATH=/opt/esp-open-sdk/xtensa-lx106-elf/bin:$PATH
make axtls
make

So far nothing different from the Adafruit tutorial except that I’m not using the vagrant VM. Also make sure that you first execute the command make axtls otherwise the main make command will compiling about not finding a version.h file. Also make sure that the export command that adds the path to the Xtensa compiler points to the right location.

After compiling, which was fast, I’ve just flashed the firmware on my Wemos mini D1 board. Again I had trouble flashing this board with other speeds than the default 115200 bps.

So:

cd ~/micropython/esp8266/build
esptool.py -p /dev/ttyUSB0 --baud 115200 write_flash 0 firmware-combined.bin

And after flashing, we can connect through the serial terminal, and pressing CTRL-D we should be greeted with the following message:

PYB: soft reboot
could not open file 'main.py' for reading
MicroPython v1.8-157-g480159c on 2016-05-29; ESP module with ESP8266
Type "help()" for more information.

Some basic tests:
When doing some tests I’ve found out that most information is outdated regarding the version of Micropython that I’ve flashed. For example:

>>> import pyb
Traceback (most recent call last):
  File "", line 1, in 
ImportError: no module named 'pyb'
>>> 

The common refered module pyb doesn’t exist, because it’s now machine:

>>> import machine
>>> dir(machine)
['__name__', 'mem8', 'mem16', 'mem32', 'freq', 'reset', 'reset_cause', 'unique_id', 'deepsleep', 'disable_irq', 'enable_irq', 'RTC', 'Timer', 'Pin', 'PWM', 'ADC', 'UART', 'I2C', 'SPI', 'DEEPSLEEP', 'PWR_ON_RESET', 'HARD_RESET', 'DEEPSLEEP_RESET']
>>>

So the following code:

>>> import pyb
>>> pin = pyb.Pin(14, pyb.Pin.OUT)  # Set pin 14 as an output.
>>> for i in range(10):
...    pin.value(0)     # Set pin low (or use pin.low())
...    pyb.delay(1000)  # Delay for 1000ms (1 second)
...    pin.value(1)     # Set pin high (or use pin.high())
...    pyb.delay(1000)

should be now:

>>> import machine
>>> pin = machine.Pin(14, machine.Pin.OUT)  # Set pin 14 as an output.
>>> for i in range(10):
...    pin.value(0)     # Set pin low (or use pin.low())
...    pyb.delay(1000)  # Delay for 1000ms (1 second)
...    pin.value(1)     # Set pin high (or use pin.high())
...    pyb.delay(1000)

Other interesting stuff is for example, the esp module:

>>> import esp
>>> dir(esp)
['__name__', 'osdebug', 'sleep_type', 'deepsleep', 'flash_id', 'flash_read', 'flash_write', 'flash_erase', 'flash_size', 'neopixel_write', 'apa102_write', 'dht_readinto', 'freemem', 'meminfo', 'info', 'malloc', 'free', 'esf_free_bufs', 'SLEEP_NONE', 'SLEEP_LIGHT', 'SLEEP_MODEM', 'STA_MODE', 'AP_MODE', 'STA_AP_MODE']
>>> esp.freemem()
17672
>>> esp.meminfo()
data  : 0x3ffe8000 ~ 0x3ffe8410, len: 1040
rodata: 0x3ffe8410 ~ 0x3ffe9038, len: 3112
bss   : 0x3ffe9038 ~ 0x3fff6bd0, len: 56216
heap  : 0x3fff6bd0 ~ 0x3fffc000, len: 21552

There are examples on GitHub to use the deepsleep functions:

Regarding the Wifi connectivity, by default when starting up the ESP8266 sets up a Wifi access point with the name Micropython-XXXXX where XXXX are some digits from the MAC address. Following the documentation the AP is protected with the password micropythoN, and sure enough the connection works. Still I haven’t tested it enough, for example, accessing the Python interpreter over Wifi, instead of through the serial port.

Anyway, one final test is to use Python to connect to make the ESP8266 to connect to my network. The instructions are simple, just write help(), and the micropython shows how to do it:

import network
>>> help()
Welcome to MicroPython!

For online docs please visit http://docs.micropython.org/en/latest/esp8266/ .
For diagnostic information to include in bug reports execute 'import port_diag'.

Basic WiFi configuration:

import network
sta_if = network.WLAN(network.STA_IF)
sta_if.active(True)
sta_if.scan()                             # Scan for available access points
sta_if.connect("", "") # Connect to an AP

and we can see if it connected successfully:

>>> sta_if.isconnected() 
True

and what IP configuration was set:

>>> sta_if.ifconfig()
('10.42.0.173', '255.255.255.0', '10.42.0.1', '10.42.0.1')

Also I was unable to access the Python interpreter through the access point connection. Supposedly there should be a listener running on port 8266 that allows access over WIFI, but I my tests found that the port 8266 was closed. Probably I need to initialize something first…
Anyway, there is a tool webrepl that allows to use the browser through websockets to connect to the ESP8266 and access the Python prompt and also to copy files to the ESP8266, namely the main.py startup file.

To finish. during my tests I had no crashes or surprise reboots. Using Python also has the advantage, in my opinion, that is more mainstream than Lua, since we leverage desktop programming with device programming. Also the useful tool ESPLorer already supports Micropython, it means that probably it is a better alternative for quick hacks using the ESP8266 instead of Nodemcu running LUA.

ESP8266 – Logging data in a backend with end to end encryption – Storing Data

As we can see in my previous post, for securely store data from a device, we should make the data secure during transport and tamper proof. This can be achieved by using encryption, but the use of encryption only does not solve some attacks, like replay attacks.

So a scheme where data is encrypted and associated with a sequence number is used, to have transport security and replay attack protection, and on the previous post we’ve implemented the base framework that allows the support of our requirements.

So the device generates, for example, the following message:

 { "data":{"value": 300} , "SEQN": 120 }

meaning that it wants to send to the backend the JSON Object {“value”: 300}, and the current Sequence number is 120. On the next message the sequence number should be 121, but that is supposing that algorithm for the sequence number is to increase it one by one per message.

Giving the message, we encrypt it now with our device key, and build another JSON object, the one that will be sent to the Node-red based backend:

 { "msg":"U2FsdGVkX1..."}

Since we still don’t have our device ready to send messages, we will build a simple NodeJs program to generate the messages:

mkdir ~/gen
cd ~/gen
npm init   (Just accept the defaults)
npm install crypto-js --save
npm install request --save

And the code to generate our message is as follow:

var CryptoJS = require("crypto-js");
var request = require('request');

// API endpoint.
var apiEP = 'http://localhost:1880/data/';
var deviceID = '12FA';

// The AES encryption/decription key to be used.
var AESKey = '2B7E151628AED2A6ABF7158809CF4F3C';

// The JSON object that we want to encrypt and transmit
var msgObjs = {"data":{"value":300}, "SEQN":121 };

// Convert the JSON object to string
var message = JSON.stringify(msgObjs);
console.log("Message: " , message );

// Encode the string to base64. Not really needed.
message = new Buffer(message).toString("base64");
console.log("Message B64: " , message );

// Encrypt
var ciphertext = CryptoJS.AES.encrypt(message, AESKey );

console.log("Cypher text: " ,  ciphertext.toString(CryptoJS.enc.base64) );
console.log(" ");
console.log("Let's call the Node-Red API end point: ");

var URL = apiEP + deviceID;
var rawdata = ciphertext.toString(CryptoJS.enc.base64);

console.log(" Calling end point: " , URL);
console.log(" RawData: " , rawdata );

// Let's call the REST API end point.
request( {
    url: URL,
    method: "POST",
    json: true,
    body: { "msg": rawdata}
    } ,
    function (error, response, body) {
        if (!error && response.statusCode == 200) {
            console.log(" ");
            console.log("REST API Output: ");
            console.log(body)
        }
    }
);
console.log("=============================================================================");
console.log(" ");
console.log("Let's do a sanity check: Let's decrypt: ");

// Decrypt
var bytes  = CryptoJS.AES.decrypt(ciphertext.toString(), AESKey );
var plaintext = bytes.toString(CryptoJS.enc.Utf8);

console.log("Decrypted message UTF8 decoded: ", plaintext);
console.log(" ");

console.log("Decrypted message: " , new Buffer(plaintext , 'base64').toString('ascii'));

We can modify the message by modifying the msgObjs variable. We run this program by executing node index.js and at the end the Node-Red answer should be displayed under REST API Output.

If everything is setup correctly the REST API should return:

REST API Output: 
{ status: 'OK' }

But calling a second time without modifying the sequence number should fail:

REST API Output: 
{ status: 'NOT OK' }

The Node-Red flow:

The Node-Red flow, receives the REST POST request for the device ID, obtains from the database the private key associated to the device, decrypts the payload, and checks the message sequence number vs the database sequence number. By design it allows the device to send messages above the current sequence number to allow message gaps (messages that where lost). We can, if needed, process this gaps so we can have an idea of how many messages that we are loosing.

If the decryption is successful and the sequence number is ok, we increment the sequence number to the next value, and store the data:

Node-Red Data Storage flow

Node-Red Data Storage flow

The code is as follow:

[{"id":"501864bd.eea2ec","type":"sqlitedb","z":"ee002ffe.ffd9e8","db":"/home/odroid/Databases/wsn.db"},{"id":"9e2ef.10b2b512","type":"http in","z":"ee002ffe.ffd9e8","name":"setSensorData","url":"/data/:id","method":"post","swaggerDoc":"","x":142.1666717529297,"y":812.0833587646484,"wires":[["fa205b8f.8de96"]]},{"id":"fa205b8f.8de96","type":"function","z":"ee002ffe.ffd9e8","name":"Build Query","func":"// Get the device id\nvar deviceId = msg.req.params.id;\n\n// Build the query. The SQLITE node requires the query in msg.topic\nmsg.topic=\"Select * from Devices where deviceID='\" + deviceId +\"'\";\n\n// Let's pass these parameters forward on its on variables:\nmsg.deviceid = deviceId;\nmsg.rawdata  = msg.payload;\n\nreturn msg;","outputs":1,"noerr":0,"x":199.1666717529297,"y":882.7499694824219,"wires":[["94004285.aa1418"]]},{"id":"94004285.aa1418","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Get Device AES Key","x":404.16668701171875,"y":817.0833435058594,"wires":[["7dd8f03c.ab2778"]]},{"id":"d0ed1bda.b15428","type":"http response","z":"ee002ffe.ffd9e8","name":"","x":1023.1666870117188,"y":956.8333435058594,"wires":[]},{"id":"7dd8f03c.ab2778","type":"function","z":"ee002ffe.ffd9e8","name":"Decrypt Request","func":"var cryptojs = context.global.cryptojs;\ntry {\n    // Get the key, Sequence number and the raw encrypted data\n    var AESKey = msg.payload[0].deviceKey;\n    msg.dbSQN = msg.payload[0].deviceSQN;   // Obtain also the currrent SQN on the database\n    \n    var rawdata= msg.rawdata.msg;\n    \n    node.log(  msg.devSQN );\n    // Decrypt the payload data with the device key. \n    // It returns a string sequence of bytes.\n    var bytes = cryptojs.AES.decrypt( rawdata, AESKey );\n    \n    // Convert bytes to an UTF8 plain string\n    var plaintext = bytes.toString(cryptojs.enc.Utf8);\n    node.log( plaintext );\n    // Convert from base64 to string\n    msg.payload  = new Buffer(plaintext , 'base64').toString('ascii');\n\n    return [ null , msg ];  // Exit the function at output 2.\n    \n} catch (err) {\n    msg.payload = { \"status\":\"NOT OK\"};\n    msg.statusCode = 500;     // Set internal server error\n    node.log(\"Invalid deviceID request for GET SQN Operation\");\n    return [ msg , null ];  // Exit the function at output 1\n}\nreturn msg;","outputs":"2","noerr":0,"x":668.1666870117188,"y":817.7499694824219,"wires":[["d0ed1bda.b15428"],["c7be8b09.893b9"]]},{"id":"c7be8b09.893b9","type":"function","z":"ee002ffe.ffd9e8","name":"Verify MSG SQN","func":"// At this point we should have:\n// msg.payload with the sequence number in JSON: { SEQN: 200} and the rest of the message\n// msg.deviceid with the device id\ntry {\n    var msgObj = JSON.parse(msg.payload);\n    var msgSeqn = msgObj.SEQN;   // The sequence number that the device as sent\n    var msgData = msgObj.data;   // The data sent\n  \n    if ( msg.dbSQN <= msgSeqn )  { // Valid Sequence number\n      node.log( \"Sequence is VALID!!!!!\");\n      msg.payload = { \"status\":\"OK\"};\n      msg.data = msgData;\n      msg.devSQN= msgSeqn;\n      return [ msg, msg];\n    } else {\n        node.log( \"Sequence is invalid!!!!\");\n        msg.payload = { \"status\":\"NOT OK\"};\n        return [ null , msg ];\n    }\n} catch( e ) {\n    node.log(\"Error verifying SEQN...\") \n    msg.payload = { \"status\":\"Internal Error\"};\n    return [ null , msg ];\n}","outputs":"2","noerr":0,"x":164.1666717529297,"y":996.7499694824219,"wires":[["c8ce0d1c.589058","533d8275.e04474"],["d0ed1bda.b15428"]]},{"id":"c8ce0d1c.589058","type":"function","z":"ee002ffe.ffd9e8","name":"Update Sequence ","func":"\n    var dbSQN = msg.devSQN + 1;   // It allows message gaps.\n\n    msg.topic=\"Update Devices Set deviceSQN= \" + dbSQN + \" Where deviceID='\" + msg.deviceid +\"'\";\n \n    //node.log(\"SQL: \" + msg.topic );\nreturn msg;","outputs":1,"noerr":0,"x":584.1666870117188,"y":1034.5833740234375,"wires":[["bef21dc4.4642a"]]},{"id":"bef21dc4.4642a","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Set SQN","x":780.1666870117188,"y":1034.4166259765625,"wires":[[]]},{"id":"533d8275.e04474","type":"function","z":"ee002ffe.ffd9e8","name":"Update Data","func":"  // Let's extract the data.\n  // This step should be modified as needed.\n  var data = msg.data.value;\n  msg.topic=\"Insert into Data Values ( '\" + msg.deviceid + \"', CURRENT_TIMESTAMP , \" + data + \")\";\n \n  //node.log (\"Data: \" + data );\nreturn msg;","outputs":1,"noerr":0,"x":579.1666870117188,"y":922.5833435058594,"wires":[["770abb44.6f54d4"]]},{"id":"770abb44.6f54d4","type":"sqlite","z":"ee002ffe.ffd9e8","mydb":"501864bd.eea2ec","name":"Save Data","x":782.1666870117188,"y":922.4165954589844,"wires":[[]]}]

And so the only thing missing is the ESP8266 code to call and setting data using the encrypted transport. For implementing that we will be using the Sming framework that will use and call the above defined API on this post and previous.

OrangePi, rtl_tcp and power line communications.

The initial reason for buying an Orange Pi PC was to remotely used it as a server for the SDR dongle and run the rtl_tcp program. In my desktop PC, running Arch Linux and using Gqrx, I can connect remotely and avoid the massive interferences produced by my desktop.

Direct connection through ethernet cabling and 1GB switch showed that the Orange PI PC + RTL SDR + rtl_sdr, after some tweaking, would work just fine, with no annoying lag between commands and response to/from the rtl_tcp program when using the Gqrx.

The issue is/was that the location for my remote SDR would mean that the connection would be made through PLC: Power Line communications. PLC uses the power lines distributing electricity around the house as the medium to connect devices without using ethernet cables or wifi.

I had doubts that with my Devolo 200MBps PLC that would provide stable connectivity for the I/Q stream from rtl_tcp, but I was wrong. Using the rtl_tcp through the 200MBps PLC works like a charm, and I only have a slight lag (around 1s) between commands and response.

The problem with PLC is that them by themselves are also huge interference generators, and so I still have some noise issues, but nothing compared to the rtl sdr being near my computer. The 868Mhz ISM band is now pretty much clear and we can see all the devices communicating.

Orange Pi PC, Armbian and SDR

A few weeks ago I’ve bought an SDR RTL2832U+R820T2 dongle to do some tests with Software Defined radio. Despite of being able to catch some signals with the provided antenna, I have a huge interference problem originated from my desktop PC. Using the SDR dongle connected to my laptop and with the desktop PC off, most part of those interferences disappear. Still reception was poor due to the antenna quality and my office location.

So one solution for the above issues is to remotely put the SDR dongle in a better location and with a better antenna and connect the SDR software running on my desktop PC to this remote SDR dongle by using the rtl_tcp program.

Since I didn’t want to shell out a lot of money again for a RPi or Odroid C1/C2, I’ve decided to by an Orange Pi PC from Aliexpress for about 16.5€, postage included.

The Orange PI PC
The Orange PI PC is a small form computer like the Raspberry Pi and Odroid C1/C2 for example. It has a quad-core Allwinner processor that runs at 1.2GHz (under Armbian), with 1GB of memory, 3 USB2 ports and HDMI.
The Orange Pi provided operating system seems to overclock the CPU to 1.6GHz which brings a lot of stability and heat issues to the device.
The Armbian version seems to not suffer from such issues and works out of the box, including HDMI video output and apparently video acceleration (Haven’t tested it yet).
I’ve also bought, separately the acrylic box and a 2A 5V charger with the correct plug to connect to the Orange PI. The Orange PI, box and power supply took less than 3 weeks to arrive.

Starting up
The Orange PI needs an small form micro SD card to have the operating system installed and space for the file system. The recommend cards are Sandisk UHS-1 or Samsung UHS-1, but I’ve bought a Toshiba micro SD Exceria UHS-1 card, that works fine.
The card comes with a standard SD card adapter to be used when connected to a card reader.

After copying the Armbian operating system to the card on my desktop computer, and putting it the micro SD card slot on the Orange PI, I’ve just connected the network, HDMI and power.

The initial power up sequence can take several minutes, since it will expand the file-system on the SD card and probably sets up other things.

At the end there was a RED led steadily lit and a blinking GREEN led, with the Armbian desktop on my monitor.

Setting up
The following steps can be done through the desktop environment or through ssh. I’ve done all these steps through ssh:

– Change the root password from the default 1234 to a secure password.
– Change the Time Zone to my time zone: dpkg-reconfigure tzdata
– Change the hostname: vi /etc/hostname
– Add a working user: adduser opi
– Add the opi user to the sudo group: usermod -aG sudo opi
– Change the password of the opi user: passwd opi
– Move the network from DHCP to a fixed IP: vi /etc/network/interfaces

# Wired adapter #1
auto eth0
#iface eth0 inet dhcp
iface eth0 inet static
address 192.168.1.5
netmask 255.255.255.0
gateway 192.168.1.254

– Update the software: apt-get update and apt-get upgrade
– Install the rtl sdr software: apt-get install rtl-sdr gqrx-sdr librtlsdr-dev libusb-1.0-0-dev
– Blacklist the dvb modules so that the RTL software can load: vi /etc/modprobe.d/rtl-sdr-blacklist.conf

# This system has librtlsdr0 installed in order to
# use digital video broadcast receivers as generic
# software defined radios.
blacklist dvb_usb
blacklist dvb_core
blacklist dvb_usb_rtl2832u
blacklist dvb_usb_rtl28xxu
blacklist e4000
blacklist rtl2832

And finally we can reboot and connect our RTL SDR dongle.

Some final notes:
The SD card speed with the Toshiba Exceria card is:

hdparm -Tt /dev/mmcblk0p1

/dev/mmcblk0p1:
 Timing cached reads:   856 MB in  2.00 seconds = 427.67 MB/sec
 Timing buffered disk reads:  58 MB in  3.03 seconds =  19.12 MB/sec

About 40% slower than my Odroid emmc disk.

To allow the remote Gqrx SDR software to connect we need to run:

 rtl_tcp -a 192.168.1.5

And configure Gqrx as following:

Gqrx Device Config

and start using GQRX.

The Orange Pi CPU and temperatures with rtl_tcp running and connected never rouse above 7/8% CPU and 43ºC, so it looks good!

Gqrx tunning
Just one final note regarding GQRX:

On the Orange Pi, the rtl_tcp program was outputting a lot of ll+:### where ### is an increasing number, and I had several seconds of lag between the change of frequency on the Gqrx aplication.

After checking the rtl_tcp code source, these messages are related to buffering issues, so I’ve changed the Gqrx connection string to:

rtl_tcp=192.168.1.5:1234,buffers=384,psize=65536

and all ll+ messages where eliminated never rising above 5.