A Zephyr RTOS based TTN Lorawan node

My previous post about Zephyr RTOS sample project with the STM32 blue pill board and the LMIC related posts such as Setting the SPI interface for Lorawan TTN LMIC and Some quick notes about Lorawan LMIC Library where the necessary stepping stones to enable the prototype creation for a TTN (The Things Network) Lorawan node but this time using the Zephyr RTOS and the LoraMac Lorawan library for the network connection.

Specifically for the LoraMac Lorawan node library, a Zephyr RTOS LoraMac code port exists, so it provides a device driver/API to the Zephyr applications which enables them to have Lora/Lorawan connectivity.

The Arduino framework and LMIC library was used as a comparative solution while debugging and testing the settings and connections for a successful connection from the node to the TTN network, to make sure that all configuration was correct and the radio (a RFM95W module) was functional. So during testing I went back and forth between the two libraries: LMIC using the Arduino framework and LoraMac-Node with Zephyr RTOS during testing.

The boards used for testing where the STM32F103CB BluePill board, the STM32F407VE board and the Blackpill STM32F411CE board. All boards can be used with either the Arduino framework and the Zephyr RTOS but I settled on doing most of all the tests with the Blackpill F411 board.

This choice was made because the main issue with the BluePill is that it doesn’t have enough flash space to have the Zephyr RTOS, the USB driver for console logging through USB and the LoraMac-node code and sample program to be stored. Using LMIC and Arduino framework on Bluepill is possible and leaves some space, but with Zephyr RTOS it is only possible to do anything useful but without USB support for serial logging. Still just note that modifying the Zephyr project configuration to have the USB driver from removed, and therefore no console logging through USB, allows to flash the board, but at the end not too much space is left do anything useful with this board. So at the end I stopping doing any tests with the Bluepull and not used it further.

Both the two other boards, the F407 and F411, have more than enough space to do any testing, but the smaller form factor of the F411Blackpill  is ideal size to do the tests (it is breadboard friendly), so I ended up using the F411 for all tests, which means that all the following steps are for this board, but can be easily replicated to other boards as long there is enough flash space.

Zephyr RTOS and Lorawan support:

The Lorawan Zephyr RTOS support was out more or less in October 2020, and for a single board 96b-wistrios board with no other information regarding any other possible boards. The support is possible by enabling the SX1276 radio driver and the Lorawan stack on the Zephyr project configuration file.

Zephyr RTOS feature support are enabled on the prj.conf file by defining the feature that we want to enable. So for this testing the following features where enabled:

# Enable GPIO pins and the SPI interface for 
# communicating with the RF95W module
CONFIG_GPIO=y
CONFIG_SPI=y

# Config USB support so that we can use a USB 
# to view log and debug information
CONFIG_USB=y
CONFIG_USB_DEVICE_STACK=y
CONFIG_USB_DEVICE_PRODUCT="Zephyr Console"

# Point the console to the USB
CONFIG_USB_UART_CONSOLE=y
CONFIG_UART_INTERRUPT_DRIVEN=y
CONFIG_UART_LINE_CTRL=y
CONFIG_UART_CONSOLE_ON_DEV_NAME="CDC_ACM_0"

# Enable the console, logging and the printk function
CONFIG_CONSOLE=y
CONFIG_LOG=y
CONFIG_PRINTK=y

# Enable the SX12XX radio driver for Lora
CONFIG_LORA=y
CONFIG_LORA_LOG_LEVEL_DBG=y
CONFIG_LORA_SX12XX=y
CONFIG_LORA_SX1276=y

# Enable the Lorawan stack
CONFIG_LORAWAN=y
CONFIG_LORAWAN_LOG_LEVEL_DBG=y
# Define the Lorawan region to be used: CONFIG_LORAMAC_REGION_EU868=y #CONFIG_LORAWAN_SYSTEM_MAX_RX_ERROR=90

All the above configuration enables the necessary components for building the Lorawan node: the SPI bus, GPIO, the SX1276 radio driver and LoraWan stack and finally logging to the USB console, where we can just use a simple terminal program to see what is going on, by can also use the printk function for the old style printf debugging…

Note that at this point we’ve haven’t defined neither the board that we will be using and the hardware interface to the Lora radio module, since the target hardware is defined at build time, not at configuration time.

Connecting the Lora Radio Module:
I’m using a RFM95W radio module that exposes some pins but not all from the SX1276 radio. To correctly work the LoraMac node library requires that each SX1276 radio DIO pins has its own associated GPIO pin (we can not merge DIO pins using diodes for example). For the several SX1276 DIO pins for Lora support, the DIO0 and DIO1 must be connected to the processor GPIO pins. The pins can be connected directly without any pull-downs or pull-ups.

The SX1276 radio module, and also by definition, the RF95W radio module that uses the SX1276 radio, use the SPI interface for communication, and so we must select what SPI bus we will use (if the board supports several SPI buses), and connect the associated SPI pins from the selected SPI bus to the RFM95W SPI pins.

To do this we need to define which SPI bus is used and how the RFM95W module is connected.

The Zephyr RTOS uses device trees to specify the connected hardware in a portable method, and comes “out of the box” with a series of predefined configurations for the hardware such SPI, I2C, UART, PWM, LED’s and so on.

If we want to change or add something to the hardware configuration, we need to create what is called an overlay file which for a specific board instance it defines or reconfigures the hardware, and overlays the new configuration over the default provided one.

In our Lorawan node we will be using the Blackpill STM32F411CE board, and for this board the Zephyr RTOS board name is blackpill_f411ce which means that we need to create a new overlay file for selecting which SPI bus we will be using and what pins the RFM95W radio module pins will be using. This configuration must be set on a file named blackpill_f411ce.overlay, and as we can see the filename must math the board name. This file must reside on the root of the project side by side with the prj.conf file or under a sub-directory named boards.

In my testing the RFM95W module will be connected to the SPI1 bus and for this board the pins are taken from the following map:

Blackpill F411 pinout

For the SPI1 bus we have:

  1. MISO – PA6
  2. MOSI – PA7
  3. SCLK – PA5

And these are the default SPI pins for the SPI1 bus that, if we need can we change to other alternate SPI1 bus pins through the overlay file. For now we just use the default pins.
To connect the RFM95W module, we need to define at least the chip select pin NSS, the DIO0 and DIO1 pins being other pins optional. Unlike the LMIC library where we could use LMIC_UNUSED_PIN, it seems that there isn’t such alternative on Zephyr.

So the pin mapping is now:

  1. NSS – PB12
  2. DIO0 – PA0
  3. DIO1 – PA1
  4. RESET – PA2

With the above settings our overlay file has now the following configuration:

&spi1 {
       status = “okay”;
       cs-gpios = <&gpiob 12 GPIO_ACTIVE_LOW>;

       lora: sx1276@0 {
                compatible = “semtech,sx1276”;
                reg = <0>;
                label = “sx1276”;
               reset-gpios = <&gpioa 3 GPIO_ACTIVE_LOW>;
               dio-gpios = <&gpioa 0 (GPIO_PULL_DOWN | GPIO_ACTIVE_HIGH)>,
                                    <&gpioa 1 (GPIO_PULL_DOWN | GPIO_ACTIVE_HIGH)>,
                                    <&gpioa 4 (GPIO_PULL_DOWN | GPIO_ACTIVE_HIGH)>,
                                   <&gpioa 4 (GPIO_PULL_DOWN | GPIO_ACTIVE_HIGH)>,
                                   <&gpioa 4 (GPIO_PULL_DOWN | GPIO_ACTIVE_HIGH)>,
                                   <&gpioa 4 (GPIO_PULL_DOWN | GPIO_ACTIVE_HIGH)>;
                rfi-enable-gpios = <&gpioa 4 GPIO_ACTIVE_HIGH>;
                rfo-enable-gpios = <&gpioa 4 GPIO_ACTIVE_HIGH>;
                pa-boost-enable-gpios = <&gpioa 4 GPIO_ACTIVE_HIGH>;
               tcxo-power-gpios = <&gpioa 4 GPIO_ACTIVE_HIGH>;
               tcxo-power-startup-delay-ms = <5>;
              spi-max-frequency = <1000000>;
    };
};

/ {
     aliases {
         lora0 = &lora;
     };
};

But the above file has a lot of unused pins mapped to GPIOA pin 4, that is really not correct or ideal.  If checking the following file: ZEPHYR_BASE/zephyr/dts/bindings/lora/semtech,sx1276.yaml we can see that some definitions are not required, and hence we can simplify our hardware configuration to only the RFM95 module pins that we really use:

&spi1 {
   status = “okay”;
   cs-gpios = <&gpiob 12  GPIO_ACTIVE_LOW>;

   lora: sx1276@0 {
       compatible = “semtech,sx1276”;
       reg = <0>;
       label = “sx1276”;
       reset-gpios = <&gpioa 3 GPIO_ACTIVE_LOW>;
       dio-gpios = <&gpioa 0 (GPIO_PULL_DOWN | GPIO_ACTIVE_HIGH)>,
                            <&gpioa 1 (GPIO_PULL_DOWN | GPIO_ACTIVE_HIGH)>;
       spi-max-frequency = <1000000>;
       power-amplifier-output = “pa-boost”;
   };
};

/ {
   aliases {
      lora0 = &lora;
 };
};

And this ends the hardware configuration for Zephyr RTOS with the RFM95W module on SPI1 bus.
We are now able to build our Lorawan TTN node for the Blackpill F411 board.

Building the Lorawan node code:
For building the Lorawan node code we must first installed and configured Zephyr RTOS so that we can use the west tool.

The code for the example is available here, and is derived from the class_a Lorawan Zephyr example, with added led blinking and USB logging so we can see what is going on. Beware that we need to first configure first the TTN keys (see below).

git clone https://github.com/fcgdam/zLorawan_Node
workon zephyr 
cd zLorawan_Node
west build -b blackpill_f411ce -p
west flash --runner openocd && sleep 2 && screen /dev/ttyACM0

For flashing I’m using the STLink connected to the board SWD pins, and since the Zephyr default for this board is DFU, we need to specify that we want to use StLink to flash it through the option –runner openocd.

We might need to press the reset button on the board so that the west flash tool works. A simple workaround to this is to edit the openocd.cfg file at (…)/zephyrproject/zephyr/boards/arm/blackpill_f411ce/support/ and add the reset_config none to the file:

source [find board/stm32f4discovery.cfg]
reset_config none

$_TARGETNAME configure -event gdb-attach {
    echo "Debugger attaching: halting execution"
    reset halt
    gdb_breakpoint_override hard
}

$_TARGETNAME configure -event gdb-detach {
    echo "Debugger detaching: resuming execution"
    resume
}

With this modification, flashing should work now without the need to press the reset button.

TTN Configuration:
Unlike the LMIC library, the key values for the necessary keys are taken directly in LSB format from the TTN console. So no need to convert anything to MSB format.
The Application EUI is called now JOIN_EUI and that is the value that we should put on that #define LORAWAN_JOIN_EUI.

Sample node output:
Reseting and connecting to the usb port provided by the board we now have the following output:

[00:00:00.130,000]  sx1276: SX1276 Version:12 found
[00:00:00.281,000]  lorawan.lorawan_init: LoRaMAC Initialized
[00:00:00.338,000]  usb_cdc_acm: Device suspended
[00:00:00.775,000]  usb_cdc_acm: Device configured
Starting up Lora node...
Starting Lorawan stack...
Joining TTN  network over  OTTA
[00:00:02.838,000]  lorawan.lorawan_join: Network join request sent!
Sending data...
[00:00:12.162,000]  lorawan.MlmeConfirm: Received MlmeConfirm (for MlmeRequest 0)
[00:00:12.162,000]  lorawan: Joined network! DevAddr: 260XXXXX
Data sent!
[00:00:16.131,000]  lorawan.McpsIndication: Received McpsIndication 0
[00:00:16.131,000]  lorawan.McpsConfirm: Received McpsConfirm (for McpsRequest 1)
[00:00:16.131,000]  lorawan.McpsConfirm: McpsRequest success!
[00:00:26.132,000]  lorawan: LoRaWAN Send failed: Duty-cycle restricted

And that’s it. Data should be shown now at the Device Traffic tab at the TTN Applications/Device console.

Some quick notes about Lorawan LMIC Library

I’m doing some testing with https://github.com/mcci-catena/arduino-lmic with several STM32 processors (STM32F103 Bluepill, Black STM32F407VE board and the newly arrived STM32F411CE blackpill board) and the RFM95 SX1276 Lora radio for connecting to the TTN Lorawan network. For the Arduino framework the MCCI LMIC library is now the library to use, since all the other alternatives have reached end of life.

For the RFM95W radio module I’m using either the Hallard RFM95W Wemos Lora shield or the DIYcon_nl Lora shield. Please note that on this board, the thickness of the PCB is a bit greater than used by SMA edge mount pin gap intervals, so we need to bend the pins a bit to be able to solder the SMA connector.

Anyway, for the Hallard shield to work, since it “merges” all Lora DIO# pins through diodes to a single pin, a 10K pull down resistor is required to be able to work with the STM32 boards, since even with GPIO pulldown enabled I had issues with the STM32 detecting the state transition. With the resistor everything works fine. This was done since the ESP8266 Wemos board hasn’t enough pins to connect SPI + CS + DIO# pins. This doesn’t happen on the STM32 Bluepill and Blackpill and of course much less on the F407VE black board where we can use all the RFM95W/SX1276 pins.

As I said initially, this post is just some quick notes of what I’ve found when using the library with the above hardware.

EV_JOIN_TXCOMPLETE: no JoinAccept
This error is good, since it means that at hardware level everything might be 100% functional.
This issue happens when using the TTN OTAA method the APPEUI, now with the latest Lorawan releases called JOINEUI and DEVEUI are not in LSB format.
Double check the EUI format, and change the order if necessary.

Assert error at radio.c:1065
This error comes from the ASSERT( (readReg(RegOpMode) & OPMODE_MASK) == OPMODE_SLEEP ); radio.c:1065 line.
This is an hardware error. It could be a problematic RF95W module, but so far every instance of this error was caused by the following issues:

1 – Bad wiring
The most obvious issue is just bad wiring/connection, either to the SPI bus or the NSS (chip select) line. Check and double check that the SPI pins used on the STM32 are the correct ones.
For example for the BluePill and the BlackPill the SPI1 bus pins are MISO1 -> PA6, MOSI1 -> A7, SCLK1 -> A5. The NSS pin can be defined on LMIC pin definitions, which takes us to the other possible error:

2 – Bad LMIC pin definition
For the LMIC stack to work, at least the NSS, DIO0 and DIO1 pins must be defined correctly. Without NSS, the RFM95W is never select and hence will never receive the necessary commands to transmit or receive. The correct DIO# pins definition is critical since these indicate when TX and RX have ended and allow the library to retrieve or send more data.
An example that I’m using on the Blackpill F411ce board is as following:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = PB12,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = PA3,
    .dio = {PA0, PA1, PA2},
    //.spi_freq = 4000000
};

Your settings might not be equal, but NSS, DIO0 and DIO1 are mandatory (DIO0 = PA0, DIO1 = PA1 in my case). I have PA2 defined for DIO2, but it can be LMIC_UNUSED_PIN without any consequences.

3 – Again bad wiring
As we can see bad wiring has a lot of influence in how successful we are using the RFM95W module, but in this case even we checked and double checked that everything looks correctly defined, it still doesn’t work and gives uses the radio.c:1065 line assert error.
In all cases that I had with this error it was again bad wiring including cold solder joints.

  • Bad dupont cable
  • Colder solder joint on the NSS pin
  • Bad solder joint on the GND(!) pin

So check all the wires and solder joints for continuity, and if they look suspicious, remake them again. In one of the cases, it was the GND RFM95 board pin, that despite having a blob of solder it didn’t make a proper contact. Also look for shorts and after soldering clean any flux residues to clear out any shorts possibility.

And that’s it. I’ll probably update this post as I find issues along the way.

Upgrading the Arduino MKRWAN Murata Lora module firmware

The Arduino MKRWan 1300 (there is also an improved version MKRWan 1310 that solves some low power issues), is an Arduino compatible board with a SAMD21 ARM processor and a Murata (CMWX1ZZABZ version 078) Lora module that internally has an STM32L0 processor and the Lora transceiver. The STM32L0 Murata module has it’s own firmware that presents an AT modem command type interface to the SAMD21 processor.

While doing some tests I’ve found out that my modules had different Murata firmware versions: 1.1.2, 1.1.5, and so some of the AT commands failed, such as the command to set FPORT AT+PORT that only existed on the 1.1.5 firmware version (or above).

Upgrading the firmware:
My first approach was to download the latest firmware release from the MKRWAN-fw releases and using the Firmware serial bridge combined with the specific STM32 flasher. With this combination it seemed that it was able to flash the STM32L0 through the serial port but I ended up with a bootable Murata module (Could see the +EVENT messages) but no response from the AT commands, so in fact it seemed that I’ve bricked the Murata modules. Reverting to an older firmware version using the same method also exhibited the same behavior.
An example of such upload is as follows:

./stm32flash -b 115200 -e 0 -w mlm32l07x01.bin /dev/ttyACM0 
stm32flash 0.5

http://stm32flash.sourceforge.net/

Using Parser : Raw BINARY
Interface serial_posix: 115200 8E1
Version      : 0x31
Option 1     : 0x00
Option 2     : 0x00
Device ID    : 0x0447 (STM32L07xxx/08xxx)
- RAM        : Up to 20KiB  (8192b reserved by bootloader)
- Flash      : Up to 192KiB (size first sector: 32x128)
- Option RAM : 32b
- System RAM : 8KiB
Write to memory
Wrote address 0x08012ce4 (100.00%) Done.

I’ve also needed to add the -e 0 to not erase the pages, or otherwise the stm32flash failed with an memory erase error so that the command was able to run (it seemed) successfully.
This is probably the issue why the Firmware flashing while sucessufull still ended up with a non responsive module.

Anyway, after some fiddling, there is no need to do anything above. On the MKRWAN library on the examples folder there is a standalone flashing utility with the firmware embedded on the file fw.h as all in one solution. More, the firmware provided seems to be more recent that the MKRWan FW releases folder, version 1.2.0 where on the releases folder it was 1.1.9 with only 1.1.6 providing the binary file.

So all we need is to compile and upload the standalone firmware upload, and it worked straight away:

 miniterm2.py /dev/ttyACM0 
--- Miniterm on /dev/ttyACM0  9600,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Press a key to start FW update
Version      : 0x31
Option 1     : 0x00
Option 2     : 0x00
Device ID    : 0x0447 (STM32L07xxx/08xxx)
- RAM        : Up to 20KiB  (8192b reserved by bootloader)
- Flash      : Up to 192KiB (size first sector: 32x128)
- Option RAM : 32b
- System RAM : 8KiB
Write to memory
Erasing memory
Wrote and verified address 0x08000100 (0%)
 Wrote and verified address 0x08000200 (0%)
 Wrote and verified address 0x08000300 (1%)
 Wrote and verified address 0x08000400 (1%)
 Wrote and verified address 0x08000500 (1%
...
...
 Wrote and verified address 0x08012c00 (100%)
 Done.

Starting execution at address 0x08000000... done.
Flashing ok :)
ARD-078 1.2.0

The odd thing is that the firmware updating is on the Github project for the client Lorawan project, the MKRWan lib and not on the MKRWAN Firmware project.

Using a small terminal/Murata bridge https://github.com/fcgdam/MKRWAN_LoraConsole:

#include <Arduino.h>

void setup() {   
  // Wait for console
  Serial.begin(115200);
  while (!Serial);

  Serial2.begin(19200);                  // Connect to the Murata module through the Serial2 port at 19200

  pinMode(LORA_BOOT0, OUTPUT);
  digitalWrite(LORA_BOOT0, LOW);
  pinMode(LORA_RESET, OUTPUT);
  digitalWrite(LORA_RESET, HIGH);
  delay(200);
  digitalWrite(LORA_RESET, LOW);
  delay(200);
  digitalWrite(LORA_RESET, HIGH);

  Serial.println("Enter AT commands to talk to the Murata module...");
}

void loop() {
	if ( Serial.available() != 0 ) {
		while ( Serial.available() > 0 ) {
			char c = Serial.read();
                if ( c == '\n' ) c = '\r';
		Serial2.print( c );						
		Serial.print( c );						
		if ( c == '\r' )
			Serial.println("");
		}
	}

	if ( Serial2.available() != 0 ) {
		while ( Serial2.available() > 0 ) {
			char c = Serial2.read();
			Serial.print( c );						
			if ( c == '\r' )
				Serial.println("");
		}
	}
}

With this simple sketch flashed onto the MKRWAN board, we can now talk directly to the Murata Module using AT commands, without any dependency from the MKRWAN lib, and hence do any tests that we might want. In my case was just to test:

AT+PORT=5
+OK

Success!.

Zephyr RTOS – Initial setup and some tests with Platformio and the NRF52840 PCA10059 dongle

This posts shows a quick how to for installing and configuring the Zephyr RTOS project on Arch Linux. In reality this post is a mashup of already a set of instructions and tutorials from the Zephyr project home page and also Adafruits Zephyr instructions:

  1. Zephyr RTOS Generic install instructions: https://docs.zephyrproject.org/latest/getting_started/index.html
  2. Adafruits install instructions with setting up Pythons virtual environments: https://learn.adafruit.com/blinking-led-with-zephyr-rtos/installing-zephyr-linux
  3. Specific instructions from the Zephyr RTOS project documentation for Arch Linux: https://docs.zephyrproject.org/latest/getting_started/installation_linux.html

By mashing up all the collected instructions from the above link, here it is my instructions:

Install some needed packages for Arch Linux:

sudo pacman -S git cmake ninja gperf ccache dfu-util dtc wget python-pip python-setuptools python-wheel tk xz file make

Check Python:
Note that Python2 is discontinued, and so all Python programs and packages are for Python 3 version.

One thing that I also had messed up was that the default Python environment on one of my machines was using Platformio penv directory, instead of the Python3 global environment. Make sure that we are using the global environment and not other non global environment.

A (better) approach as described on the Adafruit tutorial is to use Python virtual environments and so we need to install virtual environment support:

sudo pip3 install virtualenv virtualenvwrapper

and we need to change the .bashrc file at our home directory to add virtual environment support:

# For using Python and Venvs
export PATH=~/.local/bin:$PATH
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/bin/virtualenvwrapper.sh

Execute now source ~/.bashrc to load the new configuration

Since we will load our firmware on the NRF52840 dongle through DFU we also install the nrfutil:

pip install nrfutil

Installing Zephyr RTOS:
I’ll be installing the Zephyr RTOS files and SDK on /opt/Develop:

mkvirtualenv zephyr

mkdir /opt/Develop
cd /opt/Develop
mkdir zephyrproject

workon zephyr
pip install west nrfutil
west init ./zephyrproject

cd zephyrproject
west update

we also installed the nrfutil utility on this virtual environment.

To end the Zephyr RTOS setup we install the also the latest requirements:

pip install -r zephyr/scripts/requirements.txt

and that’s it.

Installing the SDK:
We can install the SDK on some of the predefined directories or our own directories, just make sure that in the later case some environmental variables are set to allow the Zephyr RTOS find the SDK:

wget https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v0.11.3/zephyr-sdk-0.11.3-setup.run

(zephyr) [pcortex@pcortex:Develop]$ ./zephyr-sdk-0.11.3-setup.run
Verifying archive integrity... All good.
Uncompressing SDK for Zephyr  100%  
Enter target directory for SDK (default: /home/pcortex/zephyr-sdk/): /opt/Develop/zephyr-sdk-0.11.3

It is recommended to install Zephyr SDK at one of the following locations for automatic discoverability in CMake:
  /opt/zephyr-sdk-0.11.3

Note: The version number '-0.11.3' can be omitted.

Do you want to continue installing to /opt/Develop/zephyr-sdk-0.11.3 (y/n)?
y
md5sum is /usr/bin/md5sum
Do you want to register the Zephyr-sdk at location: /opt/Develop/zephyr-sdk-0.11.3
  in the CMake package registry (y/n)?
y
/opt/Develop/zephyr-sdk-0.11.3 registered in /home/pcortex/.cmake/packages/Zephyr-sdk/847bb3ddf638ff02dce20cf8dc171b02
Installing SDK to /opt/Develop/zephyr-sdk-0.11.3
Creating directory /opt/Develop/zephyr-sdk-0.11.3
Success
 [*] Installing arm tools...
 [*] Installing arm64 tools...
 [*] Installing arc tools...
 [*] Installing nios2 tools...
 [*] Installing riscv64 tools...
 [*] Installing sparc tools...
 [*] Installing x86_64 tools...
 [*] Installing xtensa_sample_controller tools...
 [*] Installing xtensa_intel_apl_adsp tools...
 [*] Installing xtensa_intel_s1000 tools...
 [*] Installing xtensa_intel_bdw_adsp tools...
 [*] Installing xtensa_intel_byt_adsp tools...
 [*] Installing xtensa_nxp_imx_adsp tools...
 [*] Installing xtensa_nxp_imx8m_adsp tools...
 [*] Installing CMake files...
 [*] Installing additional host tools...
Success installing SDK.

You need to setup the following environment variables to use the toolchain:

     export ZEPHYR_TOOLCHAIN_VARIANT=zephyr
     export ZEPHYR_SDK_INSTALL_DIR=/opt/Develop/zephyr-sdk-0.11.3

Update/Create /home/pcortex/.zephyrrc with environment variables setup for you (y/n)?
y
SDK is ready to be used.

and the new .bashrc configuration is now:

# For using Python and Venvs
export PATH=~/.local/bin:$PATH
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/bin/virtualenvwrapper.sh

export ZEPHYR_TOOLCHAIN_VARIANT=zephyr
export ZEPHYR_SDK_INSTALL_DIR=/opt/Develop/zephyr-sdk-0.11.3

If we do not add the lines to the .bashrc file when starting up a project or working on it, we need to execute the zephyr-env.sh script on the Zephyr Rtos project directory.

Flashing the Blink sample program on the NRF52840 dongle:

This is pretty much documented on the NRF52840 Dongle page at NRF52840 Dongle documentation.

In our case is just something like:

cd /opt/Develop/zephyrproject
echo Select the PEnv zephyr
workon zephyr
west build -b nrf52840dongle_nrf52840 zephyr/samples/basic/blinky
nrfutil pkg generate --hw-version 52 --sd-req=0x00 --application build/zephyr/zephyr.hex --application-version 1 blinky.zip

and now we need to plugin and enable the dongle dfu mode to flash the firmware:

nrfutil dfu usb-serial -pkg blinky.zip -p /dev/ttyACM0

and the green led on the board should start to blink.

Using Platformio:
While the NRF52840 development kit from Nordic is supported (PCA10056) in both Zephyr and Platformio, the dongle version (PCA10059) is only supported on Zephyr. Since DFU upload is not supported for these boards, so we need some trickery to be able to do it from the Platformio Upload command.

To use to Platformio to target the dongle board, a project targeting the NRF52840_DK board and the Zephyr framework is created and then modifying the platformio.ini we can also target the dongle. For uploading the firmware a custom upload script is used that uses nrfutil to create a non signed DFU package and upload it.

[env:nrf52840_dongle]
platform = nordicnrf52
board = nrf52840_dk
framework = zephyr
board_build.zephyr.variant = nrf52840dongle_nrf52840
extra_scripts = dfu_upload.py
upload_protocol = custom


[env:nrf52840_dk]
platform = nordicnrf52
board = nrf52840_dk
framework = zephyr

For the NRF52840 dongle we pass to the Platformio build system the board variant used by Zephyr that targets the dongle, which is the nrf52840dongle_nrf52840 (where it was previously nrf52840_pca10059). Since the dongle hasn’t an on board debugger for uploading firmware through JTAG/Stlink, we need to use a custom upload method with an associated python script:

import sys
import os
from os.path import basename
Import("env")

platform = env.PioPlatform()

def dfu_upload(source, target, env):
    firmware_path = str(source[0])
    firmware_name = basename(firmware_path)

    genpkg = "".join(["nrfutil pkg generate --hw-version 52 --sd-req=0x00 --application ", firmware_path, " --application-version 1 firmware.zip"])
    dfupkg = "nrfutil dfu usb-serial -pkg firmware.zip -p /dev/ttyACM0"
    print( genpkg )
    os.system( genpkg )
    os.system( dfupkg )

    print("Uploading done.")


# Custom upload command and program name
env.Replace(PROGNAME="firmware", UPLOADCMD=dfu_upload)

This dfu_upload.py file is put side by side with the platformio.ini file and has some hardcoded values, such as the upload port, but it gets the job done.

The Air Quality Monitor – Data processing

After implementing and deploying the DSM501a based Air quality monitor, data was collected into an InfluxDB database and consumed by a Grafana Dashboard:

DSM501a and BMP180 Grafana Dashboard

While we can see that Temperature (yes it’s hot!) and Pressure looks ok, data collected from the DSM501a sensor is just a mess. It’s just a bunch of samples jumping around across several values, and so doesn’t look very promising that we can take meaningful data out of it.

So we might need to process the data so that it makes sense, and for that based on the fact that:

  1. Data sample is slow: 2 samples per minute
  2. We don’t want high variations, but smooth out the data

I’ve choosen to filter data using an IIR (Infinite Response Filter) LPF (Low pass filter) to remove any high frequency data/noise, and hence obtain a smoother output.
Did it work? Yes it did:

Original data vs IIR LPF filtered data

As we can see for each collected particle size of 1.0 and 2.5 we’ve filtered it with an IIR LPF that smoothed out any wild transitions while keeping the fundamental and underlying data validation.

Implementation:
IIR implementation is quite simple since it is only a set of additions, subtractions and multiplications with some factors that define the behavior of the filter.

IIR Filter

(The picture was taken from here that also explains nicely what is an IIR and FIR filters).

The input x[]n is DSM501a sample time at t=0, t=30s, t=60s, … and so on, and y[]n is the corresponding output. The b0,b1,b2 and a0,a1 and a2 are the filter factors, that define the filter response. For testing purposes I’ve just choose factors for a 1KHz Low Pass filter and tested it during several days, and hence the above output that can be seen on the Grafana dashboard.

The IIR filtering process is done on Node-Red but it can be done easily also on the ESP8266 since there is no complicated math/algorithms involved.

Node-Red IIR LPF filter

The function that implements the IIR LPF filter is (Note that on the code I use the a’s as the input factors and b’s as the output factors which is the contrary of the above IIR picture):

// IIR LPF factors
  f_a0 = 0.0010227586546542474;     // Input factors
  f_a1 = 0.002045517309308495;
  f_a2 = 0.0010227586546542474;
  
  f_b1 = -1.9066459797557103;       // Output factors
  f_b2 = 0.9107370143743273;

// PPM 1.0 input variables
var i0_c10 = msg.payload.cPM10;
var i1_c10 = context.get('i1_c10') || 0;
var i2_c10 = context.get('i2_c10') || 0;

// PPM 1.0 output variables
var o0_c10 = context.get('o0_c10') || 0;
var o1_c10 = context.get('o1_c10') || 0;


// Calculate the IIR
var lpf =   i0_c10 * f_a0 + 
            i1_c10 * f_a1 + 
            i2_c10 * f_a2 -         // We add the negative output factors
            o0_c10 * f_b1 - 
            o1_c10 * f_b2;
            
// Memorize the variables
context.set( 'i2_c10' , i1_c10 );
context.set( 'i1_c10' , i0_c10 );

context.set( 'o1_c10' , o0_c10 );
context.set( 'o0_c10' , lpf );

// PPM 2.5 input variables
var i0_c25 = msg.payload.cPM25;
var i1_c25 = context.get('i1_c25') || 0;
var i2_c25 = context.get('i2_c25') || 0;

// PPM 1.0 output variables
var o0_c25 = context.get('o0_c25') || 0;
var o1_c25 = context.get('o1_c25') || 0;


// Calculate the IIR
var lpf25 =   i0_c25 * f_a0 + 
              i1_c25 * f_a1 + 
              i2_c25 * f_a2 -         // We add the negative output factors
              o0_c25 * f_b1 - 
              o1_c25 * f_b2;
            
// Memorize the variables
context.set( 'i2_c25' , i1_c25 );
context.set( 'i1_c25' , i0_c25 );

context.set( 'o1_c25' , o0_c25 );
context.set( 'o0_c25' , lpf25 );

msg.payload = {}
msg.payload.cfP10 = lpf;
msg.payload.cfP25 = lpf25;

return msg;

We maintain the filter state (the two previous samples from the sensor) on Node-Red global variables (which will be reset if Node-red is restarted), and calculate for each PM1.0 and PM2.5 sample the filtered value, which depends on the previous samples. The final output is then fed to an InfluxDB sink node which saves the filtered data.
The complete code is at this gist.

Conclusion:
While still this being a test by using a probably LPF filter that is not adequate to the sampled data (it was designed for Audio at 96Khz sample rate), it shows that we can do some simple processing to clean up inbound data so that it makes more sense. This mechanisms of using digital filtering signals (DSP) are applied widely in other systems such as electrocardiogram sensors or other biometric sensors the remove or damp signal noise. In this case we can see that after the filtering data looks much more promising to be processed and so be used to calculate the Air Quality Index without the index jumping around as the samples jump.

An ESP8266 Air Quality monitor based on the DSM501a dust sensor

ESP8266 Air Quality Web Page

It’s unfortunate that such bad thing as the current pandemic was the chance that I had to finish this 2018 project… but better late than never.

So, I’m walking through a path that already many people have taken with this DSM501a dust sensor and ESP8266 combination to measure air quality based on dust particle count, and so this is (another) take on this combination. I’ll be using the trusty Wemos D1 ESP8266 based boards, the DSM501a dust sensor, and since that I also have an unused temperature and pressure sensor BMP180 available, I’ll also use this to finish up the project.

Basically I’ve follow up two approaches to building this project, the Arduino site example available at create.arduino.cc and a much more detailed project available at diyprojects.io site.

I’ll also use my “framework” for this kind of projects, already used at the PZEM004 Power Meter project that provides the basic building blocks for a web server and also NTP and logging facilities.

Regarding the project itself, there isn’t really anything new that I can add, except while the Arduino site code sample gave me a Air quality between Clear and Good, the other project gave me an Air Quality Index always of Hazardous, that I suppose is due to a confusion of using PM10 vs PM1.0 that is what this sensor provides. Also I’ve found out that there is several formulas available for calculating the dust concentration in mg/m3 and from that derive the Air Quality. So at the end I just use the ESP8266 to collect data, and use Node-Red to calculate the Air Quality Index with the provided data, which is much easier to debug and test, instead of using a program and flash, test cycle.

So the formulas used to obtain the data from the DSM501a are the original Arduino and diyproject.io formulas, and also this one: 0.001915 * pow(r , 2) + 0.09522 * r – 0.04884 that was discussed in this Github Wiki Post, and provide both data to be published on the MQTT topic.

As usual the code publish data and status information in two MQTT topics that I’ve defined, namely iot/device/device_id/telemetry and iot/device/device_id/attributes. An example of the data that is fed to the MQTT broker:

[AIRQ][INFO] {"AQ":"Clean","cPM10":837.15,"cPM25":0.62,"pPM10":0.11,"pPM25":0.00,"TEMP":27.60,"PRESS":101240}
[AIRQ][INFO] AIRQ Attributes:
[AIRQ][INFO] [{"type":"ESP8266"},{"ipaddr":"192.168.1.228"},{"ssid":"ZHOME3"},{"rssi":"-29"},{"web":"http://192.168.1.228"}]

Also the collected information is provided by a page served by the ESP8266 server, so it is possible to see it directly by using a web browser:

ESP8266 Air Quality Web Page
ESP8266 Air Quality Web Page

Hardware connections:
The DSM501a is trickier to connect since we can’t follow the wire colors to know which pin is which because it varies. I have two of them and both came with cables with wires of different colors for each pin. So guide the connection by pin function and not by wire color. This picture, taken from the Arduino site shows it how:

DSM501a pinout
DSM501a pinout

In my case I’ve connected the pins to the Wemos D1 ESP8266 board this way:

  1. Wemos D1 +5V -> DSM501a +5V
  2. Wemos D1 D6 -> DSM501a PM 1.0 pin
  3. Wemos D1 D5 -> DSM501a PM 2.5 pin<
  4. Wemos D1 GND -> DSM501a GND pin

The BMP180 break out board was connected to 3.3V and directly to the Wemos SCK and SDA pins. The BPM180 is optional, so the firmware code checks if it is connected and if so, it also collects data from the sensor.

Software:
As usual the software is build by using PlatformIO which pulls all the needed libraries to compile the project. All is needed is to just connect the Wemos D1 board to the USB port and do a pio run –target upload at the project root.

We can then monitor the serial port, through the pio device monitor command or run the logServer.sh script on the target monitoring server.

As usual the code is available at Github: ESP8266 Air Quality DSM501a based monitor

ESP32/ESP8266 MQTT Socket error on client – Disconnecting

When using the MQTT library for the ESP8266 or ESP32, namely this one, when publishing data on the Mosquitto I got the bellow error, followed immediately by a client disconnect:

1589388307: New client connected from 192.168.1.155 as ESP32-node (c1, k60, u'ESP32ETHE').
1589388312: Socket error on client ESP32-node, disconnecting.

One of the key issues with this library is first to ensure that the loop() function is periodically called before the MQTT connection timeout is reached.
But this was not the issue.

The issue was that the message payload for a specific topic was too big for the pre-allocated buffer of the MQTT client. So

MQTTClient mqttClient;

must be changed to this

MQTTClient mqttClient(1024); 

where 1024 is the maximum expected payload size. So we can changed to smaller or bigger depending on the situation.

With this change, the issue was gone. So moral of the story: beware of payload size.

Establishing secure ESP8266 and NodeJs communication by using Diffie-Hellman key exchange and Elliptic Curves

Introduction:
One of the issues of my later posts ( ESP8266 and AES128 for end-to-end encryption and ESP8266 – Logging data in a backend – AES and Crypto-JS) is that uses symmetric key AES128 to encrypt and decrypt data, and that the key that is used is pre-shared, meaning that it’s hardcoded on the code and is the same key to be used in all cryptographic operations.

While this might not be an issue for some use cases, in the real world, if the key is not properly protected, anybody who can gain access to it,  gains the capability to inject false data either on the ESP8266 or on the NodeJs Server, rendering in fact, the encryption effort useless.

The solution to not having a pre-shared key but since AES128 (and some other algorithms) require shared symmetric keys, we need to somehow generate a pre-shared key on demand that is not stored anywhere. But how to do that? This is a common actual problem on standard protocols such as SSL and HTTPS, and to solve this problem is where the Diffie-Hellman Key exchange/agreement protocol comes to help.

DH (Diffie-Hellman for short) works by creating at each peer that needs to communicate a set of two keys: one that is private, and one that is public. The peers exchange their public keys, and due to some mathematical properties they can calculate a common shared key using their own private key and the others public key. The key point here is that the shared key is generated without being transmitted between peers, which ensures that it is impossible to intercept it at transit. A possible attacker can see the public keys transmission, but without access to the private keys it can’t calculate the shared key.

We can just generate the shared key at boot up and keep using it until a reboot or restart, or generate a new key for each new transaction, generating in fact what is called ephemeral symmetric keys.
Since the code that will be shown bellow is just a proof of concept to show how it works, there isn’t the concept of session and so the NodeJs Server will just accept one key and peer at a time.

Anyway the DH key agreement protocol can be used at least in two different ways: by using the standard original key pairs based on the multiple groups of integers module N or the more recent and using shorter key lengths based on Elliptic Curve Cryptography, more specifically using the Curve25519 designed for DH Key exchanges for generating the necessary key pairs.

So let’s implement simple prototype that uses a single ESP8266 Wemos D1 based board that will send data to a NodeJs Server using AES128 based encryption, but this time, using as a key an ephemeral AES keys, not pre-shared keys.

NodeJS proof of concept:
From the NodeJS side we need to install a supporting module for Curve25519 DH, which is the Curve25519-N module. I’ve previously had trouble using this module, so just make sure that the used node version v13.13, where at least it compiles and works as expected.

The Curve25519 module provides the necessary functions to generate the private/public key pair and the shared key. The private/public key pair is generated from a random 32 byte secret used as a seed using the module functions. The sharedkey functions based on the given own private key and the peer’s public key can then calculate the shared key common to both peers.

A simple proof of concept is as following:

npm init
npm i curve25519-n --save

and the code testDH.js is:

// NodeJs simple DH key exchange using Ecliptic curves with the Curve25519
//
const curve = require('curve25519-n');

// Generate random 32 bytes secret
function randomSecret() {
   var result           = '';
   var characters       = 'ABCDEFGHIJKLMNOPQRSTUVXZabcdefghijklmnopqrstuvxz0123456789';
   var charactersLength = characters.length;
   for ( var i = 0; i < 32; i++ ) {
      result += characters.charAt(Math.floor(Math.random() * charactersLength));
   }
   //console.log( result );
   return result;
}
// Generate the cryptographic material
var aliceServerSecret = Buffer.from( randomSecret() );

var alicePrivateKey = curve.makeSecretKey( aliceServerSecret );
var alicePublicKey  = curve.derivePublicKey( alicePrivateKey );

var bobServerSecret = Buffer.from( randomSecret() );

var bobPrivateKey = curve.makeSecretKey( bobServerSecret );
var bobPublicKey  = curve.derivePublicKey( bobPrivateKey );

var alice_shkey = curve.deriveSharedSecret(  alicePrivateKey , bobPublicKey );
var bob_shkey   = curve.deriveSharedSecret(  bobPrivateKey , alicePublicKey );

console.log("Alice public key: " ,  Buffer.from( alicePublicKey).toString('hex') );
console.log("Bob public key:   " ,  Buffer.from( bobPublicKey).toString('hex') );
console.log("");
console.log("------ Calculated shared keys: ");
console.log("Alice shared key: ", Buffer.from(alice_shkey).toString('hex') );
console.log("Bob shared key:   ",  Buffer.from(bob_shkey).toString('hex') );

Running this will held:

node testDH.js 

Alice public key:  64ec19b47ae105ca00b7e7e088fd2c809e93118fb961d33a118c95e2ee3a9d19
Bob public key:    98d9c8d93fceed09efb15d7629d449d66892ecb4bb4a16a486b1d656a2a1501d

------ Calculated shared keys: 
Alice shared key:  6692e8240a64b595698ef98440e89affbe5102082595631ebdf472897d432c2a
Bob shared key:    6692e8240a64b595698ef98440e89affbe5102082595631ebdf472897d432c2a

and lo and behold the shared keys are the same.

Now we just need to make Alice key the NodeJS key, and Bob’s public key the ESP8266/ESP32 key.

The ESP8266/ESP32 side:
On the ESP8266/ESP32 side we also have with the Arduino framework the Curve25519 ECDH functions for an Ecliptic Curve based Dilfie-Hellman key exchange.
As usual, using Platformio we need to add the Crypto library that does support ECDH Curve25519 based DH, and also AES128.

So on the ESP side, we generate again a set of public/private key pairs, and send the public key to the NodeJS server. As a response we receive the NodeJS server public key, and then we can calculate the shared key:

void    generateKeys() {
    Curve25519::dh1( m_publicKey, m_privateKey);
}

void    initSession() {
    // We contact the NodeJS server to send our Public key
    // and as a response we receive the Nodejs Public key
    generateKeys();                 // Generate a set of Curve25519 key pair for the DH key Agreement protocol

    // The Server end-point
    String url = "http://" + NODEServer_Address + ":" + NODEServer_Port + "/getSession";

    char    s_pubkey[65];
    Bytes2Str( s_pubkey, m_publicKey, KEY_SIZE );

    // Build the post body
    String postBody = "{\"pubkey\": \"" + String(s_pubkey) +"\"}";

    // Send the request
    http.begin( url );
    http.addHeader("content-type", "application/json");

    int httpCode = http.POST( postBody );
    if (httpCode > 0) {

        String payload = http.getString();
    
        if ( httpCode == 200 ) {            
            deserializeJson( jsonDoc, payload.c_str() );

            // Obtain the foreign public key
            const char *pubkey = jsonDoc["pubkey"];
            if ( pubkey != NULL) {
                Str2Bytes(m_fpublickey, (char *)pubkey, 64 );
                printHex( "Foreign Key: ", m_fpublickey , 32 );

                // Calculate now the shared key
                Curve25519::dh2( m_fpublickey, m_privateKey ); 
                printHex ( "Shared Key", m_fpublickey , 32 );     
                memcpy( m_shkey, m_fpublickey, 32 );
            }               
        }
        else {
            Serial.println("Error on HTTP request a session.");
        }
    }
    http.end();
}

The Crypto library for the Curve25519 offers two functions: Curve25519::dh1 for generating the keys pair where the public key is generated to be sent to the peer and Curve25519::dh2 function that given the private key and foreign public key, generates the shared key.

At the end, hopefully both sides end up with the same shared key, which they do, and from there we can use that key as the AES128 symmetric key to establish communications.

The resulting shared key has more bits than the necessary for the AES128 encryption/decryption, so we derive the AES128 symmetric key from the shared key. This can be done in several ways, but I just took the easier way and only used the necessary first 16 bytes of the pre-shared key to get the AES128 key. Other approaches are to take a SHA256 or SHA512 from the key to generate any missing bits if necessary.

We also can see that on the initSession() function we generate a new set of keys for each transmission so making all used keys ephemeral since they are only used once. The drawback is that for sending data we need two transactions, one for the key exchange and other for the data transmission itself.

Testing:
The testing code that shows this ECDH (Elliptic Curve DH key agreement working) is in this repository: https://github.com/fcgdam/AESCrypto_ECDH_KeyExchange.

As usual we use Platformio to flash the firmware on the ESP8266, and to run the NodeJs server, just run npm install and node server.js. Just make sure that on the ESP8266 the SSID, Password and node server IP address are correctly set.

Running we can see on the ESP side the AES128 key to be used, and compare it with the key that was generated on the NodeJS server side.

Foreign Key: :
C8 C9 74 6E BE E9 F3 63 33 46 39 A7 4C CC 88 AB 17 14 47 3F D8 10 E0 B9 4D 9C 5B BF 3A A3 30 02 
Shared Key:
4B 40 3A A1 E2 6E 56 3C B2 5B 15 3A A6 24 6F 77 D2 C5 D5 0D 96 17 73 90 09 3A B6 38 0F C4 70 40 
AES128 key to be used: :
4B 40 3A A1 E2 6E 56 3C B2 5B 15 3A A6 24 6F 77 

IV B64: wB2astutocBMfv+xsTvAKg==
------- Sending data:
 Data: wirA/v+JcsjnP9dAVml0W/20apkQqFnY4jYMrRnw9tM=

Foreign Key: :
C8 C9 74 6E BE E9 F3 63 33 46 39 A7 4C CC 88 AB 17 14 47 3F D8 10 E0 B9 4D 9C 5B BF 3A A3 30 02 
Shared Key:
E2 4A F6 17 F3 3F B5 79 3F 6F B4 B7 8A D9 5B 5C A9 6D 65 FF 88 F3 2C 9A 18 99 99 6B B0 0F C1 4A 
AES128 key to be used: :
E2 4A F6 17 F3 3F B5 79 3F 6F B4 B7 8A D9 5B 5C 

IV B64: VW1Lnm21M1UFE45E80eNfw==
------- Sending data:
 Data: V+FdoIzYORrKiA3DjyRn9CPdYREqaQWZf8fatKFFWY0=

The associated output on the server side. Note that the calculated shared key is the same, hence we can decrypt the messages without any problems.

POST /setdata 200 0.309 ms - 37
Foreign Public Key:  D9A799D46919A2B257E112678635D7061AB589B61C42714C7B7216315AAC961B
Shared key:  4b403aa1e26e563cb25b153aa6246f77d2c5d50d96177390093ab6380fc47040
AES128 key to be used:  4b403aa1e26e563cb25b153aa6246f77
POST /getSession 200 0.421 ms - 77
Data request:  {
  iv: 'wB2astutocBMfv+xsTvAKg==',
  data: 'wirA/v+JcsjnP9dAVml0W/20apkQqFnY4jYMrRnw9tM='
}
Decrypted message:  {"testdata": "346"}
POST /setdata 200 0.296 ms - 37
Foreign Public Key:  74B73F83D4E1FFD587B0A1E14C5546CEF3EEA50E517B2ED94E64BD585C278B2A
Shared key:  e24af617f33fb5793f6fb4b78ad95b5ca96d65ff88f32c9a1899996bb00fc14a
AES128 key to be used:  e24af617f33fb5793f6fb4b78ad95b5c
POST /getSession 200 0.422 ms - 77
Data request:  {
  iv: 'VW1Lnm21M1UFE45E80eNfw==',
  data: 'V+FdoIzYORrKiA3DjyRn9CPdYREqaQWZf8fatKFFWY0='
}
Decrypted message:  {"testdata": "347"}

Conclusion:
This example shows that there is no need to preset keys on the ESP8266 device to be able to encrypt data as long that both the device and the server agree on the process for the generating the necessary key(s). Of course the server must support different sets of keys for different devices, which is not the case of the provided example, it’s just proof of concept.
Also another key element is to know who is doing the key agreement (ensuring device identity) since the above code accepts anyone to do the key agreement, which is another issue in itself.

ESP8266 and AES128 for end-to-end encryption

One of my older posts that has more hits is this one:ESP8266 – Logging data in a backend – AES and Crypto-JS where it’s explained how we can send data that is encrypted with AES128 from the ESP8266 to a backend server, either a NodeJS server or a Node-Red based service.
On the comments section I had a lot of questions and issues with the implementation, and so I’ve crafted a full implementation of the End-to-End encryption that works both ways.

The code is available at Github: AESCrypto_Test and implements the firmware for an ESP8266 based device, I’m using the Wemos D1, and two node programs: one is the server and other is the client.

The Node Server just starts and waits for incoming data from the ESP8266 and decrypts the incoming data and just shows it on the screen. The server is always running to receive requests at any time.
The Node client is run interactively by the user to send data encrypted data to the ESP8266. The ESP8266 then decrypts the data and can do whatever it needs to do. The decrypted data is output to the serial console as usual.

For this communication to happen both the ESP8266 and the Node client need to know the IP of each other, and so there is the need to change that on the code before things start to work ok.

In this example, the AES key being used is pre-shared, by another words, its known from the start by both the ESP8266 the Node Client and the Node Server. The initialization vector at the ESP8266 is random, and on the Node Client can be fixed (NOT SECURE!!) or random. Both cases are shown to show how it works.

Anyway this is just a sample code show how it works and the example can be used as stepping stone for implementing other things.

The key aspect on this code is nevertheless the use of a pre-shared key, that while it simplifies things up, is not really that secure, but anyway allows to see the concepts involved.

PZEM-004T ESP8266 software

Following up the home energy meter post based on an ESP8266 and PZEM-004T hardware, this post describes succinctly the software for using the energy meter.

There are at least two components to the solution:

  1. ESP8266 software for driving the power meter and make the measurements.
  2. The backend software for receiving and processing data.

The ESP8266 software:
The power meter software for the ESP8266 available on this GitHub repository, uses an available PZEM-004T library for accessing the power meter, and sends the collected data through MQTT to any subscribers of the power meter topic.
I’m using the convention that is also used on Thingsboard, namely an MQTT attributes topic to publish the device status, and a telemetry topic to post the data in JSON format.
Around lines 80 on main.cpp of PowerMeter sources, the topics are defined as:

  1. Attributes: “iot/device/” + String(MQTT_ClientID) + “/attributes”
  2. Telemetry: “iot/device/” + String(MQTT_ClientID) + “/telemetry”

MQTT_ClientID is defined on the secrets.h file, where we also define a list of available WIFI connections for our ESP8266. The attributes topic periodically sends the current device status (RSSI, HEAP, wifi SSID), while the data on the telemetry topic is fed into a timeseries database such as InfluxDB where then a Grafana Dashboard shows and allows to see the captured data across time.

As also my previous post regarding framework and libraries versions, I needed to block the ESP8266 framework version and the SoftwareSerial library because the combination of these with the PZE-004T library was (is ?) broken of more recent versions. As is currently defined on the platformio.ini file, the current set of versions, work fine.

A lot of people had problems working with the use of SoftwareSerial library for the PZEM library to communicate with the hardware. The issue, that I accidentally found out, are related with timing issues to communicate with the PZEM hardware. There are periods of time that the PZEM is not responsive, probably because is making some measurement.

The solution to this issue is at start up to try the connection during some time, at 3 seconds interval until it succeeds. After the connection is successful, we need to keep an interval around one minute between reads to encounter no issues/failures . If this interval is kept, the connection to the PZEM hardware works flawlessly, at least with the hardware that I have.

So the connection phase is checked and tried several times to synchronize the ESP8266 with the PZEM, and them every single minute there is a data read. If the interval is shorter, lets say, 30s, it will fail, until the elapsed time to one minute is completed.

The firmware solves the above issue, and after reading the data, it posts it to a MQTT broker. The firmware also makes available a web page with the current status and measurements:

Power Meter Web Page

Then there are other bits, namely since the meter will be on the electric mains board, an UDP logging facility that allows on the computer to run an UDP server and see what is going on.

The back-end software:
I’ve not done much on this area, since most of it is just standard stuff. An MQTT broker and Node-Red flow. The flow just receives the data, saves it into an InfluxDB database and creates a Node-Red UI dashboard.

Power Meter Node-Red UI

This screenshot shows some of the information that was collected on the last minute and it is updated in real time as soon the PowerMeter information arrives to the MQTT broker.

Future work:
Basically what is missing is two things:

  1. Grafana Dashboard based on the InfluxDB data (Already done, to be described in a future post).
  2. Some kind of exporter to CSV or Spreadsheat to allow further data analysis such as the daily power consumption totals.