The Air Quality Monitor – Data processing

After implementing and deploying the DSM501a based Air quality monitor, data was collected into an InfluxDB database and consumed by a Grafana Dashboard:

DSM501a and BMP180 Grafana Dashboard

While we can see that Temperature (yes it’s hot!) and Pressure looks ok, data collected from the DSM501a sensor is just a mess. It’s just a bunch of samples jumping around across several values, and so doesn’t look very promising that we can take meaningful data out of it.

So we might need to process the data so that it makes sense, and for that based on the fact that:

  1. Data sample is slow: 2 samples per minute
  2. We don’t want high variations, but smooth out the data

I’ve choosen to filter data using an IIR (Infinite Response Filter) LPF (Low pass filter) to remove any high frequency data/noise, and hence obtain a smoother output.
Did it work? Yes it did:

Original data vs IIR LPF filtered data

As we can see for each collected particle size of 1.0 and 2.5 we’ve filtered it with an IIR LPF that smoothed out any wild transitions while keeping the fundamental and underlying data validation.

IIR implementation is quite simple since it is only a set of additions, subtractions and multiplications with some factors that define the behavior of the filter.

IIR Filter

(The picture was taken from here that also explains nicely what is an IIR and FIR filters).

The input x[]n is DSM501a sample time at t=0, t=30s, t=60s, … and so on, and y[]n is the corresponding output. The b0,b1,b2 and a0,a1 and a2 are the filter factors, that define the filter response. For testing purposes I’ve just choose factors for a 1KHz Low Pass filter and tested it during several days, and hence the above output that can be seen on the Grafana dashboard.

The IIR filtering process is done on Node-Red but it can be done easily also on the ESP8266 since there is no complicated math/algorithms involved.

Node-Red IIR LPF filter

The function that implements the IIR LPF filter is (Note that on the code I use the a’s as the input factors and b’s as the output factors which is the contrary of the above IIR picture):

// IIR LPF factors
  f_a0 = 0.0010227586546542474;     // Input factors
  f_a1 = 0.002045517309308495;
  f_a2 = 0.0010227586546542474;
  f_b1 = -1.9066459797557103;       // Output factors
  f_b2 = 0.9107370143743273;

// PPM 1.0 input variables
var i0_c10 = msg.payload.cPM10;
var i1_c10 = context.get('i1_c10') || 0;
var i2_c10 = context.get('i2_c10') || 0;

// PPM 1.0 output variables
var o0_c10 = context.get('o0_c10') || 0;
var o1_c10 = context.get('o1_c10') || 0;

// Calculate the IIR
var lpf =   i0_c10 * f_a0 + 
            i1_c10 * f_a1 + 
            i2_c10 * f_a2 -         // We add the negative output factors
            o0_c10 * f_b1 - 
            o1_c10 * f_b2;
// Memorize the variables
context.set( 'i2_c10' , i1_c10 );
context.set( 'i1_c10' , i0_c10 );

context.set( 'o1_c10' , o0_c10 );
context.set( 'o0_c10' , lpf );

// PPM 2.5 input variables
var i0_c25 = msg.payload.cPM25;
var i1_c25 = context.get('i1_c25') || 0;
var i2_c25 = context.get('i2_c25') || 0;

// PPM 1.0 output variables
var o0_c25 = context.get('o0_c25') || 0;
var o1_c25 = context.get('o1_c25') || 0;

// Calculate the IIR
var lpf25 =   i0_c25 * f_a0 + 
              i1_c25 * f_a1 + 
              i2_c25 * f_a2 -         // We add the negative output factors
              o0_c25 * f_b1 - 
              o1_c25 * f_b2;
// Memorize the variables
context.set( 'i2_c25' , i1_c25 );
context.set( 'i1_c25' , i0_c25 );

context.set( 'o1_c25' , o0_c25 );
context.set( 'o0_c25' , lpf25 );

msg.payload = {}
msg.payload.cfP10 = lpf;
msg.payload.cfP25 = lpf25;

return msg;

We maintain the filter state (the two previous samples from the sensor) on Node-Red global variables (which will be reset if Node-red is restarted), and calculate for each PM1.0 and PM2.5 sample the filtered value, which depends on the previous samples. The final output is then fed to an InfluxDB sink node which saves the filtered data.
The complete code is at this gist.

While still this being a test by using a probably LPF filter that is not adequate to the sampled data (it was designed for Audio at 96Khz sample rate), it shows that we can do some simple processing to clean up inbound data so that it makes more sense. This mechanisms of using digital filtering signals (DSP) are applied widely in other systems such as electrocardiogram sensors or other biometric sensors the remove or damp signal noise. In this case we can see that after the filtering data looks much more promising to be processed and so be used to calculate the Air Quality Index without the index jumping around as the samples jump.

An ESP8266 Air Quality monitor based on the DSM501a dust sensor

ESP8266 Air Quality Web Page

It’s unfortunate that such bad thing as the current pandemic was the chance that I had to finish this 2018 project… but better late than never.

So, I’m walking through a path that already many people have taken with this DSM501a dust sensor and ESP8266 combination to measure air quality based on dust particle count, and so this is (another) take on this combination. I’ll be using the trusty Wemos D1 ESP8266 based boards, the DSM501a dust sensor, and since that I also have an unused temperature and pressure sensor BMP180 available, I’ll also use this to finish up the project.

Basically I’ve follow up two approaches to building this project, the Arduino site example available at and a much more detailed project available at site.

I’ll also use my “framework” for this kind of projects, already used at the PZEM004 Power Meter project that provides the basic building blocks for a web server and also NTP and logging facilities.

Regarding the project itself, there isn’t really anything new that I can add, except while the Arduino site code sample gave me a Air quality between Clear and Good, the other project gave me an Air Quality Index always of Hazardous, that I suppose is due to a confusion of using PM10 vs PM1.0 that is what this sensor provides. Also I’ve found out that there is several formulas available for calculating the dust concentration in mg/m3 and from that derive the Air Quality. So at the end I just use the ESP8266 to collect data, and use Node-Red to calculate the Air Quality Index with the provided data, which is much easier to debug and test, instead of using a program and flash, test cycle.

So the formulas used to obtain the data from the DSM501a are the original Arduino and formulas, and also this one: 0.001915 * pow(r , 2) + 0.09522 * r – 0.04884 that was discussed in this Github Wiki Post, and provide both data to be published on the MQTT topic.

As usual the code publish data and status information in two MQTT topics that I’ve defined, namely iot/device/device_id/telemetry and iot/device/device_id/attributes. An example of the data that is fed to the MQTT broker:

[AIRQ][INFO] {"AQ":"Clean","cPM10":837.15,"cPM25":0.62,"pPM10":0.11,"pPM25":0.00,"TEMP":27.60,"PRESS":101240}
[AIRQ][INFO] AIRQ Attributes:
[AIRQ][INFO] [{"type":"ESP8266"},{"ipaddr":""},{"ssid":"ZHOME3"},{"rssi":"-29"},{"web":""}]

Also the collected information is provided by a page served by the ESP8266 server, so it is possible to see it directly by using a web browser:

ESP8266 Air Quality Web Page
ESP8266 Air Quality Web Page

Hardware connections:
The DSM501a is trickier to connect since we can’t follow the wire colors to know which pin is which because it varies. I have two of them and both came with cables with wires of different colors for each pin. So guide the connection by pin function and not by wire color. This picture, taken from the Arduino site shows it how:

DSM501a pinout
DSM501a pinout

In my case I’ve connected the pins to the Wemos D1 ESP8266 board this way:

  1. Wemos D1 +5V -> DSM501a +5V
  2. Wemos D1 D6 -> DSM501a PM 1.0 pin
  3. Wemos D1 D5 -> DSM501a PM 2.5 pin<
  4. Wemos D1 GND -> DSM501a GND pin

The BMP180 break out board was connected to 3.3V and directly to the Wemos SCK and SDA pins. The BPM180 is optional, so the firmware code checks if it is connected and if so, it also collects data from the sensor.

As usual the software is build by using PlatformIO which pulls all the needed libraries to compile the project. All is needed is to just connect the Wemos D1 board to the USB port and do a pio run –target upload at the project root.

We can then monitor the serial port, through the pio device monitor command or run the script on the target monitoring server.

As usual the code is available at Github: ESP8266 Air Quality DSM501a based monitor

ESP32/ESP8266 MQTT Socket error on client – Disconnecting

When using the MQTT library for the ESP8266 or ESP32, namely this one, when publishing data on the Mosquitto I got the bellow error, followed immediately by a client disconnect:

1589388307: New client connected from as ESP32-node (c1, k60, u'ESP32ETHE').
1589388312: Socket error on client ESP32-node, disconnecting.

One of the key issues with this library is first to ensure that the loop() function is periodically called before the MQTT connection timeout is reached.
But this was not the issue.

The issue was that the message payload for a specific topic was too big for the pre-allocated buffer of the MQTT client. So

MQTTClient mqttClient;

must be changed to this

MQTTClient mqttClient(1024); 

where 1024 is the maximum expected payload size. So we can changed to smaller or bigger depending on the situation.

With this change, the issue was gone. So moral of the story: beware of payload size.

Establishing secure ESP8266 and NodeJs communication by using Diffie-Hellman key exchange and Elliptic Curves

One of the issues of my later posts ( ESP8266 and AES128 for end-to-end encryption and ESP8266 – Logging data in a backend – AES and Crypto-JS) is that uses symmetric key AES128 to encrypt and decrypt data, and that key is pre-shared, meaning that it’s hardcoded on the code and is the same at all times.

While this might not be an issue for some use cases, in the real world, if the key is not properly protected, anybody with access to it, can inject false data either on the ESP8266 or on the NodeJs Server, rendering in fact the encryption efforts useless.

The solution to not having a pre-shared key and since AES128 (and some other algorithms) require shared symmetric keys, is to somehow generate a pre-shared key on demand that is not stored anywhere, but how to do that? This is a common actual problem on standard protocols such as SSL and HTTPS, and to solve this problem is where the Diffie-Hellman Key exchange/agreement protocol comes to help.

DH (Diffie-Hellman for short) works by creating at each peer that needs to communicate a set of two keys: one that is private, and one that is public. The peers exchange their public keys, and due to some mathematical properties they can calculate a common shared key using their own private key and the others public key. The key point here is that the shared key is generated without being transmitted between peers which ensures that it is impossible to intercept it at transit. A possible attacker can see the public keys transmission, but without access to the private keys it can’t calculate the shared key.

We can just generate the shared key at boot up and keep using it until a reboot or restart, or generate a new key for each new transaction, generating in fact ephemeral symmetric keys.
Since the code that will be shown bellow is just a proof of concept to show how it works, there isn’t the concept of session and so the NodeJs Server will just accept on key and peer at a time.

Anyway the DH key agreement protocol can be used at least in two different ways: by using the standard original key pairs based on the multiple groups of integers module N or the more recent and using shorter key lengths based on Elliptic Curve Cryptography, more specifically using the Curve25519 designed for DH Key exchanges for generating the necessary key pairs.

So let’s implement a single ESP8266 Wemos D1 based board that sends data to a NodeJs Server using AES128 based encryption, but this time using ephemeral AES keys, not pre-shared keys.

NodeJS proof of concept:
From the NodeJS side we need to install a supporting module for Curve25519 DH, which is the Curve25519-N module. I’ve previously had trouble using this module, so just make sure that the used node version v13.13, where at least it compiles and works as expected.

The provided module functions are the necessary functions to generate the private/public key pair, where the private key pair is generated from a random 32 byte secret, and the function that given the own private key and the peer’s public key calculates the shared key.

A simple proof of concept is as following:

npm init
npm i curve25519-n --save

and the code testDH.js is:

// NodeJs simple DH key exchange using Ecliptic curves with the Curve25519
const curve = require('curve25519-n');

// Generate random 32 bytes secret
function randomSecret() {
   var result           = '';
   var characters       = 'ABCDEFGHIJKLMNOPQRSTUVXZabcdefghijklmnopqrstuvxz0123456789';
   var charactersLength = characters.length;
   for ( var i = 0; i < 32; i++ ) {
      result += characters.charAt(Math.floor(Math.random() * charactersLength));
   //console.log( result );
   return result;
// Generate the cryptographic material
var aliceServerSecret = Buffer.from( randomSecret() );

var alicePrivateKey = curve.makeSecretKey( aliceServerSecret );
var alicePublicKey  = curve.derivePublicKey( alicePrivateKey );

var bobServerSecret = Buffer.from( randomSecret() );

var bobPrivateKey = curve.makeSecretKey( bobServerSecret );
var bobPublicKey  = curve.derivePublicKey( bobPrivateKey );

var alice_shkey = curve.deriveSharedSecret(  alicePrivateKey , bobPublicKey );
var bob_shkey   = curve.deriveSharedSecret(  bobPrivateKey , alicePublicKey );

console.log("Alice public key: " ,  Buffer.from( alicePublicKey).toString('hex') );
console.log("Bob public key:   " ,  Buffer.from( bobPublicKey).toString('hex') );
console.log("------ Calculated shared keys: ");
console.log("Alice shared key: ", Buffer.from(alice_shkey).toString('hex') );
console.log("Bob shared key:   ",  Buffer.from(bob_shkey).toString('hex') );

Running this will held:

node testDH.js 

Alice public key:  64ec19b47ae105ca00b7e7e088fd2c809e93118fb961d33a118c95e2ee3a9d19
Bob public key:    98d9c8d93fceed09efb15d7629d449d66892ecb4bb4a16a486b1d656a2a1501d

------ Calculated shared keys: 
Alice shared key:  6692e8240a64b595698ef98440e89affbe5102082595631ebdf472897d432c2a
Bob shared key:    6692e8240a64b595698ef98440e89affbe5102082595631ebdf472897d432c2a

and lo and behold the shared keys are the same.

Now we just need to make Alice key the NodeJS key, and Bob’s public key the ESP8266/ESP32 key.

The ESP8266/ESP32 side:
On the ESP8266/ESP32 side we also have with the Arduino framework the Curve25519 ECDH functions for an Ecliptic Curve based Dilfie-Hellman key exchange.
As usual, using Platformio we need to add the Crypto library that does support ECDH Curve25519 based DH, and also AES128.

So on the ESP side, we generate again a set of public/private key pairs, and send the public key to the NodeJS server. As a response we receive the NodeJS server public key, and then we can calculate the shared key:

void    generateKeys() {
    Curve25519::dh1( m_publicKey, m_privateKey);

void    initSession() {
    // We contact the NodeJS server to send our Public key
    // and as a response we receive the Nodejs Public key
    generateKeys();                 // Generate a set of Curve25519 key pair for the DH key Agreement protocol

    // The Server end-point
    String url = "http://" + NODEServer_Address + ":" + NODEServer_Port + "/getSession";

    char    s_pubkey[65];
    Bytes2Str( s_pubkey, m_publicKey, KEY_SIZE );

    // Build the post body
    String postBody = "{\"pubkey\": \"" + String(s_pubkey) +"\"}";

    // Send the request
    http.begin( url );
    http.addHeader("content-type", "application/json");

    int httpCode = http.POST( postBody );
    if (httpCode > 0) {

        String payload = http.getString();
        if ( httpCode == 200 ) {            
            deserializeJson( jsonDoc, payload.c_str() );

            // Obtain the foreign public key
            const char *pubkey = jsonDoc["pubkey"];
            if ( pubkey != NULL) {
                Str2Bytes(m_fpublickey, (char *)pubkey, 64 );
                printHex( "Foreign Key: ", m_fpublickey , 32 );

                // Calculate now the shared key
                Curve25519::dh2( m_fpublickey, m_privateKey ); 
                printHex ( "Shared Key", m_fpublickey , 32 );     
                memcpy( m_shkey, m_fpublickey, 32 );
        else {
            Serial.println("Error on HTTP request a session.");

The Crypto library for the Curve25519 offers two functions: Curve25519::dh1 for generating the keys pair where the public key is generated to be sent to the peer and Curve25519::dh2 function that given the private key and foreign public key, generates the shared key.

At the end, hopefully both sides end up with the same shared key, which they do, and from there we can use that key as the AES128 symmetric key to establish communications.

The resulting shared key has more bits than the necessary for the AES128 encryption/decryption, so we derive the AES128 symmetric key from the shared key. This can be done in several ways, but I just took the easier way and only used the necessary first 16 bytes of the pre-shared key to get the AES128 key. Other approaches are to take a SHA256 or SHA512 from the key to generate any missing bits if necessary.

We also can see that on the initSession() function we generate a new set of keys for each transmission so making all used keys ephemeral since they are only used once. The drawback is that for sending data we need two transactions, one for the key exchange and other for the data transmission itself.

The testing code that shows this ECDH (Elliptic Curve DH key agreement working) is in this repository:

As usual we use Platformio to flash the firmware on the ESP8266, and to run the NodeJs server, just run npm install and node server.js. Just make sure that on the ESP8266 the SSID, Password and node server IP address are correctly set.

Running we can see on the ESP side the AES128 key to be used, and compare it with the key that was generated on the NodeJS server side.

Foreign Key: :
C8 C9 74 6E BE E9 F3 63 33 46 39 A7 4C CC 88 AB 17 14 47 3F D8 10 E0 B9 4D 9C 5B BF 3A A3 30 02 
Shared Key:
4B 40 3A A1 E2 6E 56 3C B2 5B 15 3A A6 24 6F 77 D2 C5 D5 0D 96 17 73 90 09 3A B6 38 0F C4 70 40 
AES128 key to be used: :
4B 40 3A A1 E2 6E 56 3C B2 5B 15 3A A6 24 6F 77 

IV B64: wB2astutocBMfv+xsTvAKg==
------- Sending data:
 Data: wirA/v+JcsjnP9dAVml0W/20apkQqFnY4jYMrRnw9tM=

Foreign Key: :
C8 C9 74 6E BE E9 F3 63 33 46 39 A7 4C CC 88 AB 17 14 47 3F D8 10 E0 B9 4D 9C 5B BF 3A A3 30 02 
Shared Key:
E2 4A F6 17 F3 3F B5 79 3F 6F B4 B7 8A D9 5B 5C A9 6D 65 FF 88 F3 2C 9A 18 99 99 6B B0 0F C1 4A 
AES128 key to be used: :
E2 4A F6 17 F3 3F B5 79 3F 6F B4 B7 8A D9 5B 5C 

IV B64: VW1Lnm21M1UFE45E80eNfw==
------- Sending data:
 Data: V+FdoIzYORrKiA3DjyRn9CPdYREqaQWZf8fatKFFWY0=

The associated output on the server side. Note that the calculated shared key is the same, hence we can decrypt the messages without any problems.

POST /setdata 200 0.309 ms - 37
Foreign Public Key:  D9A799D46919A2B257E112678635D7061AB589B61C42714C7B7216315AAC961B
Shared key:  4b403aa1e26e563cb25b153aa6246f77d2c5d50d96177390093ab6380fc47040
AES128 key to be used:  4b403aa1e26e563cb25b153aa6246f77
POST /getSession 200 0.421 ms - 77
Data request:  {
  iv: 'wB2astutocBMfv+xsTvAKg==',
  data: 'wirA/v+JcsjnP9dAVml0W/20apkQqFnY4jYMrRnw9tM='
Decrypted message:  {"testdata": "346"}
POST /setdata 200 0.296 ms - 37
Foreign Public Key:  74B73F83D4E1FFD587B0A1E14C5546CEF3EEA50E517B2ED94E64BD585C278B2A
Shared key:  e24af617f33fb5793f6fb4b78ad95b5ca96d65ff88f32c9a1899996bb00fc14a
AES128 key to be used:  e24af617f33fb5793f6fb4b78ad95b5c
POST /getSession 200 0.422 ms - 77
Data request:  {
  iv: 'VW1Lnm21M1UFE45E80eNfw==',
  data: 'V+FdoIzYORrKiA3DjyRn9CPdYREqaQWZf8fatKFFWY0='
Decrypted message:  {"testdata": "347"}

This example shows that there is no need to preset keys on the ESP8266 device to be able to encrypt data as long that both the device and the server agree on the process for the generating the necessary key(s). Of course the server must support different sets of keys for different devices, which is not the case of the provided example, it’s just proof of concept.
Also another key element is to know who is doing the key agreement since the above code accepts anyone to do the key agreement, which is another issue in itself.

ESP8266 and AES128 for end-to-end encryption

One of my older posts that has more hits is this one:ESP8266 – Logging data in a backend – AES and Crypto-JS where it’s explained how we can send data that is encrypted with AES128 from the ESP8266 to a backend server, either a NodeJS server or a Node-Red based service.
On the comments section I had a lot of questions and issues with the implementation, and so I’ve crafted a full implementation of the End-to-End encryption that works both ways.

The code is available at Github: AESCrypto_Test and implements the firmware for an ESP8266 based device, I’m using the Wemos D1, and two node programs: one is the server and other is the client.

The Node Server just starts and waits for incoming data from the ESP8266 and decrypts the incoming data and just shows it on the screen. The server is always running to receive requests at any time.
The Node client is run interactively by the user to send data encrypted data to the ESP8266. The ESP8266 then decrypts the data and can do whatever it needs to do. The decrypted data is output to the serial console as usual.

For this communication to happen both the ESP8266 and the Node client need to know the IP of each other, and so there is the need to change that on the code before things start to work ok.

In this example, the AES key being used is pre-shared, by another words, its known from the start by both the ESP8266 the Node Client and the Node Server. The initialization vector at the ESP8266 is random, and on the Node Client can be fixed (NOT SECURE!!) or random. Both cases are shown to show how it works.

Anyway this is just a sample code show how it works and the example can be used as stepping stone for implementing other things.

The key aspect on this code is nevertheless the use of a pre-shared key, that while it simplifies things up, is not really that secure, but anyway allows to see the concepts involved.

PZEM-004T ESP8266 software

Following up the home energy meter post based on an ESP8266 and PZEM-004T hardware, this post describes succinctly the software for using the energy meter.

There are at least two components to the solution:

  1. ESP8266 software for driving the power meter and make the measurements.
  2. The backend software for receiving and processing data.

The ESP8266 software:
The power meter software for the ESP8266 available on this GitHub repository, uses an available PZEM-004T library for accessing the power meter, and sends the collected data through MQTT to any subscribers of the power meter topic.
I’m using the convention that is also used on Thingsboard, namely an MQTT attributes topic to publish the device status, and a telemetry topic to post the data in JSON format.
Around lines 80 on main.cpp of PowerMeter sources, the topics are defined as:

  1. Attributes: “iot/device/” + String(MQTT_ClientID) + “/attributes”
  2. Telemetry: “iot/device/” + String(MQTT_ClientID) + “/telemetry”

MQTT_ClientID is defined on the secrets.h file, where we also define a list of available WIFI connections for our ESP8266. The attributes topic periodically sends the current device status (RSSI, HEAP, wifi SSID), while the data on the telemetry topic is fed into a timeseries database such as InfluxDB where then a Grafana Dashboard shows and allows to see the captured data across time.

As also my previous post regarding framework and libraries versions, I needed to block the ESP8266 framework version and the SoftwareSerial library because the combination of these with the PZE-004T library was (is ?) broken of more recent versions. As is currently defined on the platformio.ini file, the current set of versions, work fine.

A lot of people had problems working with the use of SoftwareSerial library for the PZEM library to communicate with the hardware. The issue, that I accidentally found out, are related with timing issues to communicate with the PZEM hardware. There are periods of time that the PZEM is not responsive, probably because is making some measurement.

The solution to this issue is at start up to try the connection during some time, at 3 seconds interval until it succeeds. After the connection is successful, we need to keep an interval around one minute between reads to encounter no issues/failures . If this interval is kept, the connection to the PZEM hardware works flawlessly, at least with the hardware that I have.

So the connection phase is checked and tried several times to synchronize the ESP8266 with the PZEM, and them every single minute there is a data read. If the interval is shorter, lets say, 30s, it will fail, until the elapsed time to one minute is completed.

The firmware solves the above issue, and after reading the data, it posts it to a MQTT broker. The firmware also makes available a web page with the current status and measurements:

Power Meter Web Page

Then there are other bits, namely since the meter will be on the electric mains board, an UDP logging facility that allows on the computer to run an UDP server and see what is going on.

The back-end software:
I’ve not done much on this area, since most of it is just standard stuff. An MQTT broker and Node-Red flow. The flow just receives the data, saves it into an InfluxDB database and creates a Node-Red UI dashboard.

Power Meter Node-Red UI

This screenshot shows some of the information that was collected on the last minute and it is updated in real time as soon the PowerMeter information arrives to the MQTT broker.

Future work:
Basically what is missing is two things:

  1. Grafana Dashboard based on the InfluxDB data (Already done, to be described in a future post).
  2. Some kind of exporter to CSV or Spreadsheat to allow further data analysis such as the daily power consumption totals.

Measuring home energy consumption with the PZEM004T and ESP8266

First of all a very BIG WARNING: This project works with AC mains current, which, where I live, is 220V AC, meaning that extra precautions must be taken, since risk of serious injuries and/or death is possible.

The PZEM004T
The Peacefair PZEM004T device (available at the usual far east shopping web sites) is a device that can measure energy consumption by monitoring a live AC mains wire using an inductor as the measuring sensor. Take notice there are currently at least (3/2020) two versions: V2.0 which uses the standard serial protocol, and V3.0 that uses Modbus. This project uses V2.0.

One of the wires that carries the current (normally the AC power phase) goes through the inductor so that the current that flows through it can be measured and hence the other measurements, including power consumption, can be also measured.

The PZEM004T can be bought with two types of inductor, one that opens up and can clip on the wire of interest, and the other type that requires to disconnect and connect the wire of interest, so that it passes through the inductor core. I’ve chosen the former, since in this case I do not need to do any disconnection/connections on the electric mains board, and so it is way safer and easier to add and remove the measurement device.


The PZEMM04T outputs the collected data through an opto-coupled isolated serial port that allows to retrieve values for voltage, current/intensity, current power consumption and energy accumulated consumption.

The device that connects to the PZEM serial port must provide power to it (5V), and so the serial port data lines are 5V level, which means that we should use a 5v to 3.3V level converter to connect to the ESP8266. While there are several hacks to make the PZEM004T serial port to use 3.3V on the serial port, and hence have 3.3V data lines, I just used a simple level converter to connect the serial port to the ESP8266, and avoid in this way any modifications to the PZEM-004T. The serial port connector is a 4 Pin JST-XH connector.

So the basic schematics for using the PZEM004T is as simple as the following highly professionally drawn schematic shows:

PZEM004T And Wemos D1 connection schematic

Two things of notice:

  • The Ground connection – The serial port uses the same ground as the Wemos D1.
  • The Power supply – Wemos D1 is powered through the 5V pin, NOT through the 3.3v pin, since we need 5V to power up the PZEM serial port.

The level converter is just a simple, cheap I2C level converter, used in this case to level convert the serial data lines.
Also the above schematic shows that the TX and RX pins connect to the Wemos D6 and D5 pins, since I’ll be using software serial, but the depicted connections are just an example, since the pins to be used can be software defined.
In my code I use the connections the other way around ( D5-TX, D6-RX) so beware to how the pins are connected and how they are defined at the software level.

Powering up the ESP8266 Wemos D1
I’ll be using the Wemos D1 ESP8266 based boards, as we can see on the above schematics (associated to a prototype shield to solder the connections and the level converter), we need to power it up using 5V. The ESP8266 uses 3.3v, but the Wemos board has a 5V input and a 5V to 3.3V converter, so no issues there. The PZEM004T on the other hand uses 220V, and since the ESP8266 will be near the PZEM004T, it makes sense to get the 5V CC from the 220V AC to power up the Wemos D1 board.

The 220V AC to 5V CC can be achieved in several ways, and since I’ll be installing all this in a DIN case on my home electricity mains board, the easiest solution is just to buy a 5V output 220V based DIN power supply for around 10/15€. This is the easiest and safest solution.

There are other solutions, including the one that I’m using that is based on the 5V HLK-PM01 based modules. This requires some assembly and also be aware that there are fake HLK modules around.

Do not connect the HLK-PM01 without the associated protection components, namely fuses, VDR, and the most important component the thermal fuse of 72ºC (Celsius!) that will cut off the power to anything after it (including the VDR) if the temperature of the HLK module or it’s surroundings rises above the 72ºC temperature. I’ve not soldered the thermal fuse, since the heat from the soldering iron can destroy it, just used a two terminal with screws to connect it.

The schematic used is the following one:

5V Power supply

The PZEM-004T, the HLK based power supply and the Wemos D1 ESP8266 module are inside a double length project DIN case so that all components can be safely installed on the mains electricity board.

Since all is self contained on the DIN case, all is needed is to clip the inductor on the main phase wire entering the mains board (and it is easy since the inductor is an open clip on type), and connect the components to the 220V AC power. I’ve derived the power from one of the circuit breakers that already protects a house circuit, which adds an additional layer of protection.

The software
On the next post I’ll discuss the software for driving the ESP8266 to gather data from the PZEM004T and how it works.
The firmware for driving this is already available at:

Lorawan/TTN with the TTGO ESP32 board

So I’m testing my TTGO Lora32 board with the RIOT-OS operating system to connect to the The Things Network, and while at first I had some success, then out of the blue things didn’t worked anymore.

The code that I’m using is based on the standard RIOT-Os Loramac example but with supporting code for ABP instead of the OTA activation.

The code is available at this Github repository.

Since I saw no activity on any channel for the Lora transmission by using the RTLSDR dongle, I suspected that something was up.
The Arduino based code from for connecting to TTN using Lorawan worked fine, and so it excluded an issue with the hardware.

By going to RIOT-OS installation directory, under drivers and on the sx127x directory, I’ve enabled on th sx127x.c file the debug to 1, since enabling debug on the semtech_loramac.c file held no useful information:

#define ENABLE_DEBUG (1)

With debug enabled, lo and behold, at startup:

2019-01-22 14:11:02,993 - INFO # [sx127x] error: failed to initialize SPI_0 device (code -1)
2019-01-22 14:11:02,993 - INFO # [sx127x] error: failed to initialize SPI

This is rather strange since at the initial tests the sx127x was detected and worked.
The only thing that changed is that I periodically update the RIOT local repository, and checking my board definition with other ESP32 boards that have a sx127x transceiver, the way that the pins where declared were quite different from what I had initially.

So I’ve changed the way that the pins for the SPI and the sx127x peripheral where defined, and the result:

2019-01-22 14:28:00,847 - INFO #  -> Node activation by: ABP
2019-01-22 14:28:00,847 - INFO # [sx127x] SPI_0 initialized with success
2019-01-22 14:28:00,848 - INFO # Set ABP information.

And the data arrives at TTN without any issue:

So either I had initially had the pins for SPI and the sx127x defined wrong or the latest RIOT build changed the way pins where defined.

A more complete example, working with multiple threads, is also available at this Github repository.

A simple start with the RiotOS operating system for IoT devices

The RiotOS operating system is an Open Source operating system that targets different embedded platforms while allowing to use the same code base for some aspects of IoT development: Connectivity, protocols and security. It also supports other key aspects such as threads, Inter process communication, synchronization support on the kernel side. It also supports several communications protocols, suche BLE, 6LoWPAN, Lorawan, and so on.

At the target level for the RiotOS operating system, RiotOS supports a wide scope of different platforms, including a special native platform. What does this means? It means that within certain limitations, it is entirely possible to develop an IoT application that runs where the code is developed, this includes our PC and any SBC such as the RaspberryPI. This support opens a new window of opportunities where it is possible to integrate different classes of devices, such as Arduino, STM32 and RPI while leveraging the knowledge on the same supporting code base.

How to start:

This is a very quick start instructions for building a native demo application with network support. As usual all the instructions apply to a Linux Operating system, in my case Arch Linux.

Clone the RiotOS repository:

Just run:

cd /opt
git clone

I’ve chosen the /opt directory for the sake of example.

Compile one the example application:

All example applications are under the example applications.
In each directory, there is a Makefile file with a specific content.

For example on the examples/default there is Makefile that targets the native environment.

Of key interest is the line in this file that defines the target board:

BOARD ?= native

If the BOARD variable is not defined on the running environment it will use the native board.

We just need now to run the make command:

[pcortex@pcortex:default|master]$ make
Building application "default" for "native" with MCU "native".

"make" -C /opt/RIOT/boards/native
"make" -C /opt/RIOT/boards/native/drivers
"make" -C /opt/RIOT/core
"make" -C /opt/RIOT/cpu/native
"make" -C /opt/RIOT/sys/shell
"make" -C /opt/RIOT/sys/shell/commands
   text    data     bss     dec     hex filename
  88189    1104   72088  161381   27665 /opt/RIOT/examples/default/bin/native/default.elf

We can see that the compilation output was placed at /opt/RIOT/examples/default/bin/native/default.elf.

But if we ran the default.elf executable, we get:

[pcortex@pcortex:default|master]$ bin/native/default.elf 
usage: bin/native/default.elf  [-i ] [-d] [-e|-E] [-o] [-c ]
 help: bin/native/default.elf -h

    -h, --help
        print this help message
    -i , --id=
        specify instance id (set by config module)
    -s , --seed=
        specify srandom(3) seed (/dev/random is used instead of random(3) if
        the option is omitted)
    -d, --daemonize
        daemonize native instance
    -e, --stderr-pipe
        redirect stderr to file
    -E, --stderr-noredirect
        do not redirect stderr (i.e. leave sterr unchanged despite
        daemon/socket io)
    -o, --stdout-pipe
        redirect stdout to file (/tmp/riot.stdout.PID) when not attached
        to socket
    -c , --uart-tty=
        specify TTY device for UART. This argument can be used multiple
        times (up to UART_NUMOF)

A little side note, under bin there is a native directory but if we target the compilation to a different platform, a new directory under bin will be created to that platform.

Returning to result to the above command, we notice that the command requires a TAP (terminal interface point) interface. The fact is that the RiotOS code to access the network resources of the hosting operating system doesn’t access the network interface directly, but does it through the TAP interface. We can create only one TAP interface, or several TAP interfaces, which in such case, means that we can have several RiotOS applications using an TAP based network to communicate between themselves and also with the external network. Neat!

RiotOS offers a script to the TAP intefaces, but before running the script, just make sure if any Linux operating system Kernel was done recently, a reboot migh be needed to the interfaces be successfully created.

[pcortex@pcortex:RIOT|master]$ ip a

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp6s0:  mtu 4000 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:24:8c:02:dd:7e brd ff:ff:ff:ff:ff:ff
    inet brd scope global noprefixroute enp6s0
       valid_lft forever preferred_lft forever
    inet6 fe80::224:8cff:fe02:dd7e/64 scope link 
       valid_lft forever preferred_lft forever

Now we run the tapsetup script that creates the TAP interfaces. This script when called without any parameters, it creates two TAP interfaces. If we pass the -c argument with a number, for example -c 4, it will create 4 TAP interfaces. The -d parameter deletes all interface that where created.

[pcortex@pcortex:tapsetup|master]$ pwd
[pcortex@pcortex:tapsetup|master]$ sudo ./tapsetup 
creating tapbr0
creating tap0
creating tap1
[pcortex@pcortex:tapsetup|master]$ ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp6s0:  mtu 4000 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:24:8c:02:dd:7e brd ff:ff:ff:ff:ff:ff
    inet brd scope global noprefixroute enp6s0
       valid_lft forever preferred_lft forever
    inet6 fe80::224:8cff:fe02:dd7e/64 scope link 
       valid_lft forever preferred_lft forever
8: tapbr0:  mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 2e:bd:d8:88:64:55 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2cbd:d8ff:fe88:6455/64 scope link 
       valid_lft forever preferred_lft forever
9: tap0:  mtu 1500 qdisc fq_codel master tapbr0 state DOWN group default qlen 1000
    link/ether 9a:a0:bc:fa:c7:61 brd ff:ff:ff:ff:ff:ff
10: tap1:  mtu 1500 qdisc fq_codel master tapbr0 state DOWN group default qlen 1000
    link/ether 2e:bd:d8:88:64:55 brd ff:ff:ff:ff:ff:ff

We can see that both tap0 and tap1 where created and are down.

Within two different windows we can run the default.elf with each interface now:

Window one: > sudo ./default.elf tap0
Window two: > sudo ./default.elf tap1

The two instances will start communicate and the tap interface have addresses and are up now.

11: tapbr0:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:db:a6:77:3b:e5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::cdb:a6ff:fe77:3be5/64 scope link 
       valid_lft forever preferred_lft forever
12: tap0:  mtu 1500 qdisc fq_codel master tapbr0 state UP group default qlen 1000
    link/ether 0e:db:a6:77:3b:e5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::cdb:a6ff:fe77:3be5/64 scope link 
       valid_lft forever preferred_lft forever
13: tap1:  mtu 1500 qdisc fq_codel master tapbr0 state UP group default qlen 1000
    link/ether be:65:4f:4d:a0:ed brd ff:ff:ff:ff:ff:ff
    inet6 fe80::bc65:4fff:fe4d:a0ed/64 scope link 
       valid_lft forever preferred_lft forever

The above is a simple example, but RiotOS offers several examples including COAP, MQTT-SN and OpenThread examples.

The next step is to test RiotOS on ESP8266 and ESP32.

Simple BLE bridge to TTN Lora using the TTGO ESP32 LoRa32 board

The TTGO LoRa32 is an ESP32 based board that features Wifi and BlueTooth low energy but also includes an external Lora chip, in my case the SX1276 868Mhz version.

The following code/hack is just to test the feasibility of bridging BLE devices over the ESP32 and then to Lorawan, more specifically sending BLE data to the LoraWan TTN network.

I’m using Neil Koban ESP32 BLE library, that under platformIO is library number 1841 and the base ABP code for connecting to TTN.

In simple terms this code just makes the ESP32 to emulate a BLE UART device for sending and receiving data. It does that by using the Nordic UART known UUID for specifying the BLE UART service and using also the Nordic mobile applications, that supports such device, for sending/receiving data.

Using the Nordic mobile Android phone applications, data can be sent to the Lora32 board either by using the excellent Nordic Connect application or by also using the simpler and direct Nordic UART application.

The tests program just receives data through BLE and buffers it onto an internal message buffer that, periodically, is sent through Lora to the TTN network. I’ve decided arbitrary that the buffer is 32 bytes maximum. We should keep our message size to the necessary minimum, and also just send few messages to keep the lorawan duty factor usage within the required limits.

So, using the following code we can use our phone to scan from the ESP32 BLE device named TTGOLORAESP32 connect to it and send data to the device.

After a while, when the transmission event fires up, data is transmitted, and the BLE device just receives a simple notification with the EV_TXCOMPLETE message.

That’s it.