The Air Quality Monitor – Data processing

After implementing and deploying the DSM501a based Air quality monitor, data was collected into an InfluxDB database and consumed by a Grafana Dashboard:

DSM501a and BMP180 Grafana Dashboard

While we can see that Temperature (yes it’s hot!) and Pressure looks ok, data collected from the DSM501a sensor is just a mess. It’s just a bunch of samples jumping around across several values, and so doesn’t look very promising that we can take meaningful data out of it.

So we might need to process the data so that it makes sense, and for that based on the fact that:

  1. Data sample is slow: 2 samples per minute
  2. We don’t want high variations, but smooth out the data

I’ve choosen to filter data using an IIR (Infinite Response Filter) LPF (Low pass filter) to remove any high frequency data/noise, and hence obtain a smoother output.
Did it work? Yes it did:

Original data vs IIR LPF filtered data

As we can see for each collected particle size of 1.0 and 2.5 we’ve filtered it with an IIR LPF that smoothed out any wild transitions while keeping the fundamental and underlying data validation.

Implementation:
IIR implementation is quite simple since it is only a set of additions, subtractions and multiplications with some factors that define the behavior of the filter.

IIR Filter

(The picture was taken from here that also explains nicely what is an IIR and FIR filters).

The input x[]n is DSM501a sample time at t=0, t=30s, t=60s, … and so on, and y[]n is the corresponding output. The b0,b1,b2 and a0,a1 and a2 are the filter factors, that define the filter response. For testing purposes I’ve just choose factors for a 1KHz Low Pass filter and tested it during several days, and hence the above output that can be seen on the Grafana dashboard.

The IIR filtering process is done on Node-Red but it can be done easily also on the ESP8266 since there is no complicated math/algorithms involved.

Node-Red IIR LPF filter

The function that implements the IIR LPF filter is (Note that on the code I use the a’s as the input factors and b’s as the output factors which is the contrary of the above IIR picture):

// IIR LPF factors
  f_a0 = 0.0010227586546542474;     // Input factors
  f_a1 = 0.002045517309308495;
  f_a2 = 0.0010227586546542474;
  
  f_b1 = -1.9066459797557103;       // Output factors
  f_b2 = 0.9107370143743273;

// PPM 1.0 input variables
var i0_c10 = msg.payload.cPM10;
var i1_c10 = context.get('i1_c10') || 0;
var i2_c10 = context.get('i2_c10') || 0;

// PPM 1.0 output variables
var o0_c10 = context.get('o0_c10') || 0;
var o1_c10 = context.get('o1_c10') || 0;


// Calculate the IIR
var lpf =   i0_c10 * f_a0 + 
            i1_c10 * f_a1 + 
            i2_c10 * f_a2 -         // We add the negative output factors
            o0_c10 * f_b1 - 
            o1_c10 * f_b2;
            
// Memorize the variables
context.set( 'i2_c10' , i1_c10 );
context.set( 'i1_c10' , i0_c10 );

context.set( 'o1_c10' , o0_c10 );
context.set( 'o0_c10' , lpf );

// PPM 2.5 input variables
var i0_c25 = msg.payload.cPM25;
var i1_c25 = context.get('i1_c25') || 0;
var i2_c25 = context.get('i2_c25') || 0;

// PPM 1.0 output variables
var o0_c25 = context.get('o0_c25') || 0;
var o1_c25 = context.get('o1_c25') || 0;


// Calculate the IIR
var lpf25 =   i0_c25 * f_a0 + 
              i1_c25 * f_a1 + 
              i2_c25 * f_a2 -         // We add the negative output factors
              o0_c25 * f_b1 - 
              o1_c25 * f_b2;
            
// Memorize the variables
context.set( 'i2_c25' , i1_c25 );
context.set( 'i1_c25' , i0_c25 );

context.set( 'o1_c25' , o0_c25 );
context.set( 'o0_c25' , lpf25 );

msg.payload = {}
msg.payload.cfP10 = lpf;
msg.payload.cfP25 = lpf25;

return msg;

We maintain the filter state (the two previous samples from the sensor) on Node-Red global variables (which will be reset if Node-red is restarted), and calculate for each PM1.0 and PM2.5 sample the filtered value, which depends on the previous samples. The final output is then fed to an InfluxDB sink node which saves the filtered data.
The complete code is at this gist.

Conclusion:
While still this being a test by using a probably LPF filter that is not adequate to the sampled data (it was designed for Audio at 96Khz sample rate), it shows that we can do some simple processing to clean up inbound data so that it makes more sense. This mechanisms of using digital filtering signals (DSP) are applied widely in other systems such as electrocardiogram sensors or other biometric sensors the remove or damp signal noise. In this case we can see that after the filtering data looks much more promising to be processed and so be used to calculate the Air Quality Index without the index jumping around as the samples jump.

ESP32/ESP8266 MQTT Socket error on client – Disconnecting

When using the MQTT library for the ESP8266 or ESP32, namely this one, when publishing data on the Mosquitto I got the bellow error, followed immediately by a client disconnect:

1589388307: New client connected from 192.168.1.155 as ESP32-node (c1, k60, u'ESP32ETHE').
1589388312: Socket error on client ESP32-node, disconnecting.

One of the key issues with this library is first to ensure that the loop() function is periodically called before the MQTT connection timeout is reached.
But this was not the issue.

The issue was that the message payload for a specific topic was too big for the pre-allocated buffer of the MQTT client. So

MQTTClient mqttClient;

must be changed to this

MQTTClient mqttClient(1024); 

where 1024 is the maximum expected payload size. So we can changed to smaller or bigger depending on the situation.

With this change, the issue was gone. So moral of the story: beware of payload size.

Establishing secure ESP8266 and NodeJs communication by using Diffie-Hellman key exchange and Elliptic Curves

Introduction:
One of the issues of my later posts ( ESP8266 and AES128 for end-to-end encryption and ESP8266 – Logging data in a backend – AES and Crypto-JS) is that uses symmetric key AES128 to encrypt and decrypt data, and that key is pre-shared, meaning that it’s hardcoded on the code and is the same at all times.

While this might not be an issue for some use cases, in the real world, if the key is not properly protected, anybody with access to it, can inject false data either on the ESP8266 or on the NodeJs Server, rendering in fact the encryption efforts useless.

The solution to not having a pre-shared key and since AES128 (and some other algorithms) require shared symmetric keys, is to somehow generate a pre-shared key on demand that is not stored anywhere, but how to do that? This is a common actual problem on standard protocols such as SSL and HTTPS, and to solve this problem is where the Diffie-Hellman Key exchange/agreement protocol comes to help.

DH (Diffie-Hellman for short) works by creating at each peer that needs to communicate a set of two keys: one that is private, and one that is public. The peers exchange their public keys, and due to some mathematical properties they can calculate a common shared key using their own private key and the others public key. The key point here is that the shared key is generated without being transmitted between peers which ensures that it is impossible to intercept it at transit. A possible attacker can see the public keys transmission, but without access to the private keys it can’t calculate the shared key.

We can just generate the shared key at boot up and keep using it until a reboot or restart, or generate a new key for each new transaction, generating in fact ephemeral symmetric keys.
Since the code that will be shown bellow is just a proof of concept to show how it works, there isn’t the concept of session and so the NodeJs Server will just accept on key and peer at a time.

Anyway the DH key agreement protocol can be used at least in two different ways: by using the standard original key pairs based on the multiple groups of integers module N or the more recent and using shorter key lengths based on Elliptic Curve Cryptography, more specifically using the Curve25519 designed for DH Key exchanges for generating the necessary key pairs.

So let’s implement a single ESP8266 Wemos D1 based board that sends data to a NodeJs Server using AES128 based encryption, but this time using ephemeral AES keys, not pre-shared keys.

NodeJS proof of concept:
From the NodeJS side we need to install a supporting module for Curve25519 DH, which is the Curve25519-N module. I’ve previously had trouble using this module, so just make sure that the used node version v13.13, where at least it compiles and works as expected.

The provided module functions are the necessary functions to generate the private/public key pair, where the private key pair is generated from a random 32 byte secret, and the function that given the own private key and the peer’s public key calculates the shared key.

A simple proof of concept is as following:

npm init
npm i curve25519-n --save

and the code testDH.js is:

// NodeJs simple DH key exchange using Ecliptic curves with the Curve25519
//
const curve = require('curve25519-n');

// Generate random 32 bytes secret
function randomSecret() {
   var result           = '';
   var characters       = 'ABCDEFGHIJKLMNOPQRSTUVXZabcdefghijklmnopqrstuvxz0123456789';
   var charactersLength = characters.length;
   for ( var i = 0; i < 32; i++ ) {
      result += characters.charAt(Math.floor(Math.random() * charactersLength));
   }
   //console.log( result );
   return result;
}
// Generate the cryptographic material
var aliceServerSecret = Buffer.from( randomSecret() );

var alicePrivateKey = curve.makeSecretKey( aliceServerSecret );
var alicePublicKey  = curve.derivePublicKey( alicePrivateKey );

var bobServerSecret = Buffer.from( randomSecret() );

var bobPrivateKey = curve.makeSecretKey( bobServerSecret );
var bobPublicKey  = curve.derivePublicKey( bobPrivateKey );

var alice_shkey = curve.deriveSharedSecret(  alicePrivateKey , bobPublicKey );
var bob_shkey   = curve.deriveSharedSecret(  bobPrivateKey , alicePublicKey );

console.log("Alice public key: " ,  Buffer.from( alicePublicKey).toString('hex') );
console.log("Bob public key:   " ,  Buffer.from( bobPublicKey).toString('hex') );
console.log("");
console.log("------ Calculated shared keys: ");
console.log("Alice shared key: ", Buffer.from(alice_shkey).toString('hex') );
console.log("Bob shared key:   ",  Buffer.from(bob_shkey).toString('hex') );

Running this will held:

node testDH.js 

Alice public key:  64ec19b47ae105ca00b7e7e088fd2c809e93118fb961d33a118c95e2ee3a9d19
Bob public key:    98d9c8d93fceed09efb15d7629d449d66892ecb4bb4a16a486b1d656a2a1501d

------ Calculated shared keys: 
Alice shared key:  6692e8240a64b595698ef98440e89affbe5102082595631ebdf472897d432c2a
Bob shared key:    6692e8240a64b595698ef98440e89affbe5102082595631ebdf472897d432c2a

and lo and behold the shared keys are the same.

Now we just need to make Alice key the NodeJS key, and Bob’s public key the ESP8266/ESP32 key.

The ESP8266/ESP32 side:
On the ESP8266/ESP32 side we also have with the Arduino framework the Curve25519 ECDH functions for an Ecliptic Curve based Dilfie-Hellman key exchange.
As usual, using Platformio we need to add the Crypto library that does support ECDH Curve25519 based DH, and also AES128.

So on the ESP side, we generate again a set of public/private key pairs, and send the public key to the NodeJS server. As a response we receive the NodeJS server public key, and then we can calculate the shared key:

void    generateKeys() {
    Curve25519::dh1( m_publicKey, m_privateKey);
}

void    initSession() {
    // We contact the NodeJS server to send our Public key
    // and as a response we receive the Nodejs Public key
    generateKeys();                 // Generate a set of Curve25519 key pair for the DH key Agreement protocol

    // The Server end-point
    String url = "http://" + NODEServer_Address + ":" + NODEServer_Port + "/getSession";

    char    s_pubkey[65];
    Bytes2Str( s_pubkey, m_publicKey, KEY_SIZE );

    // Build the post body
    String postBody = "{\"pubkey\": \"" + String(s_pubkey) +"\"}";

    // Send the request
    http.begin( url );
    http.addHeader("content-type", "application/json");

    int httpCode = http.POST( postBody );
    if (httpCode > 0) {

        String payload = http.getString();
    
        if ( httpCode == 200 ) {            
            deserializeJson( jsonDoc, payload.c_str() );

            // Obtain the foreign public key
            const char *pubkey = jsonDoc["pubkey"];
            if ( pubkey != NULL) {
                Str2Bytes(m_fpublickey, (char *)pubkey, 64 );
                printHex( "Foreign Key: ", m_fpublickey , 32 );

                // Calculate now the shared key
                Curve25519::dh2( m_fpublickey, m_privateKey ); 
                printHex ( "Shared Key", m_fpublickey , 32 );     
                memcpy( m_shkey, m_fpublickey, 32 );
            }               
        }
        else {
            Serial.println("Error on HTTP request a session.");
        }
    }
    http.end();
}

The Crypto library for the Curve25519 offers two functions: Curve25519::dh1 for generating the keys pair where the public key is generated to be sent to the peer and Curve25519::dh2 function that given the private key and foreign public key, generates the shared key.

At the end, hopefully both sides end up with the same shared key, which they do, and from there we can use that key as the AES128 symmetric key to establish communications.

The resulting shared key has more bits than the necessary for the AES128 encryption/decryption, so we derive the AES128 symmetric key from the shared key. This can be done in several ways, but I just took the easier way and only used the necessary first 16 bytes of the pre-shared key to get the AES128 key. Other approaches are to take a SHA256 or SHA512 from the key to generate any missing bits if necessary.

We also can see that on the initSession() function we generate a new set of keys for each transmission so making all used keys ephemeral since they are only used once. The drawback is that for sending data we need two transactions, one for the key exchange and other for the data transmission itself.

Testing:
The testing code that shows this ECDH (Elliptic Curve DH key agreement working) is in this repository: https://github.com/fcgdam/AESCrypto_ECDH_KeyExchange.

As usual we use Platformio to flash the firmware on the ESP8266, and to run the NodeJs server, just run npm install and node server.js. Just make sure that on the ESP8266 the SSID, Password and node server IP address are correctly set.

Running we can see on the ESP side the AES128 key to be used, and compare it with the key that was generated on the NodeJS server side.

Foreign Key: :
C8 C9 74 6E BE E9 F3 63 33 46 39 A7 4C CC 88 AB 17 14 47 3F D8 10 E0 B9 4D 9C 5B BF 3A A3 30 02 
Shared Key:
4B 40 3A A1 E2 6E 56 3C B2 5B 15 3A A6 24 6F 77 D2 C5 D5 0D 96 17 73 90 09 3A B6 38 0F C4 70 40 
AES128 key to be used: :
4B 40 3A A1 E2 6E 56 3C B2 5B 15 3A A6 24 6F 77 

IV B64: wB2astutocBMfv+xsTvAKg==
------- Sending data:
 Data: wirA/v+JcsjnP9dAVml0W/20apkQqFnY4jYMrRnw9tM=

Foreign Key: :
C8 C9 74 6E BE E9 F3 63 33 46 39 A7 4C CC 88 AB 17 14 47 3F D8 10 E0 B9 4D 9C 5B BF 3A A3 30 02 
Shared Key:
E2 4A F6 17 F3 3F B5 79 3F 6F B4 B7 8A D9 5B 5C A9 6D 65 FF 88 F3 2C 9A 18 99 99 6B B0 0F C1 4A 
AES128 key to be used: :
E2 4A F6 17 F3 3F B5 79 3F 6F B4 B7 8A D9 5B 5C 

IV B64: VW1Lnm21M1UFE45E80eNfw==
------- Sending data:
 Data: V+FdoIzYORrKiA3DjyRn9CPdYREqaQWZf8fatKFFWY0=

The associated output on the server side. Note that the calculated shared key is the same, hence we can decrypt the messages without any problems.

POST /setdata 200 0.309 ms - 37
Foreign Public Key:  D9A799D46919A2B257E112678635D7061AB589B61C42714C7B7216315AAC961B
Shared key:  4b403aa1e26e563cb25b153aa6246f77d2c5d50d96177390093ab6380fc47040
AES128 key to be used:  4b403aa1e26e563cb25b153aa6246f77
POST /getSession 200 0.421 ms - 77
Data request:  {
  iv: 'wB2astutocBMfv+xsTvAKg==',
  data: 'wirA/v+JcsjnP9dAVml0W/20apkQqFnY4jYMrRnw9tM='
}
Decrypted message:  {"testdata": "346"}
POST /setdata 200 0.296 ms - 37
Foreign Public Key:  74B73F83D4E1FFD587B0A1E14C5546CEF3EEA50E517B2ED94E64BD585C278B2A
Shared key:  e24af617f33fb5793f6fb4b78ad95b5ca96d65ff88f32c9a1899996bb00fc14a
AES128 key to be used:  e24af617f33fb5793f6fb4b78ad95b5c
POST /getSession 200 0.422 ms - 77
Data request:  {
  iv: 'VW1Lnm21M1UFE45E80eNfw==',
  data: 'V+FdoIzYORrKiA3DjyRn9CPdYREqaQWZf8fatKFFWY0='
}
Decrypted message:  {"testdata": "347"}

Conclusion:
This example shows that there is no need to preset keys on the ESP8266 device to be able to encrypt data as long that both the device and the server agree on the process for the generating the necessary key(s). Of course the server must support different sets of keys for different devices, which is not the case of the provided example, it’s just proof of concept.
Also another key element is to know who is doing the key agreement since the above code accepts anyone to do the key agreement, which is another issue in itself.

Lorawan/TTN with the TTGO ESP32 board

So I’m testing my TTGO Lora32 board with the RIOT-OS operating system to connect to the The Things Network, and while at first I had some success, then out of the blue things didn’t worked anymore.

The code that I’m using is based on the standard RIOT-Os Loramac example but with supporting code for ABP instead of the OTA activation.

The code is available at this Github repository.

Since I saw no activity on any channel for the Lora transmission by using the RTLSDR dongle, I suspected that something was up.
The Arduino based code from for connecting to TTN using Lorawan worked fine, and so it excluded an issue with the hardware.

By going to RIOT-OS installation directory, under drivers and on the sx127x directory, I’ve enabled on th sx127x.c file the debug to 1, since enabling debug on the semtech_loramac.c file held no useful information:

#define ENABLE_DEBUG (1)

With debug enabled, lo and behold, at startup:

2019-01-22 14:11:02,993 - INFO # [sx127x] error: failed to initialize SPI_0 device (code -1)
2019-01-22 14:11:02,993 - INFO # [sx127x] error: failed to initialize SPI

This is rather strange since at the initial tests the sx127x was detected and worked.
The only thing that changed is that I periodically update the RIOT local repository, and checking my board definition with other ESP32 boards that have a sx127x transceiver, the way that the pins where declared were quite different from what I had initially.

So I’ve changed the way that the pins for the SPI and the sx127x peripheral where defined, and the result:

2019-01-22 14:28:00,847 - INFO #  -> Node activation by: ABP
2019-01-22 14:28:00,847 - INFO # [sx127x] SPI_0 initialized with success
2019-01-22 14:28:00,848 - INFO # Set ABP information.

And the data arrives at TTN without any issue:

So either I had initially had the pins for SPI and the sx127x defined wrong or the latest RIOT build changed the way pins where defined.

A more complete example, working with multiple threads, is also available at this Github repository.

The ESP32 TTGO board using RIOT-OS

In the last few months, RIOT-OS operating system gained support for the ESP32 and ESP8266 based boards. RIOT-OS also supports several communication protocols, namely Lorawan through the usage of the Semtech implementation.

My post connecting the ESP32 TTGO board to TTN (The Things Network) describes (more or less) how to connect this board to the The Things Network using the Arduino and LMIC stack for the ESP32.

So why don’t give it a go with RIOT-OS?

Board definition:
RIOT-OS works with board definitions that describes several key features of the targeting board. In the case of the TTGO ESP32 Version 1 that I have several things needed to be changed from the base ESP32 board definitions that are available now on the RIOT-OS code base.

The most important issue was the clock speed definition for this ESP32 board, since it seems it uses a 26MHz clock instead of the standard(?) 40Mhz clock. Without the change for the clock speed all the serial output on the USB port was garbled and unreadable.

With the clock speed correctly set, important information regarding the pins for the I2C, SPI buses and the sx1276 chip needed to be defined. That was also correctly defined, and preliminary tests show that the Oled and sx1276 is detected using the Semtech Loramac stack.

I haven’t yet change the other pin definitions, so for the basic I2C, SPI, Button, and sx1276 are correctly defined.

The board definition can be found in my repository located at RIOT TTGO ESP32 repository under the directory boards.

To use this board definition, we just need to copy the board definition directory to the boards directory of the cloned RIOT-OS repository.

Testing:
For testing we can just compile the examples under the example directory for the RIOT TTGO ESP32 repository.

There are two examples: One for testing the OLED screen and other to test the sx1276 and the Semtech Loramac stack.
This last example is based on the provided RIOT-OS example for Lorawan, but I’ve added support for ABP and debug prints to see what is going on.
On the Make file, we should select the type of activation and fill out the required data from the TTN device console. All data just needs to be copied and pasted directly and it should work.

To build the examples I’m using Docker according to this instructions.

Additionally, for the ESP32 flashing to work, the ESP-IDF repository must be cloned and its location set for RIOT build by setting the ESP32_SDK_DIR variable.

Results:
The following things work fine:

– The serial console works fine.
– The blue led, on and off
– The button, detecting it release and pressed.

The OLED works fine through I2C, but the I2C speed must be set exactly as defined on the board definition, otherwise it won’t work. It took me several hours to debug this issue.

The sx1276 works, it is detected and the Semtech stack seems to be able to use it. Adding some debugging to Loramac stack shows that the DIO# interrupts are detected but there are issues. For example I have two boards and with the same code I have different behaviours which seems rather strange… I was able to send data to the TTN gateway, but when waiting for the Semtech stack it hangs waiting for a stack status message. The other board is always complaining regarding MAC timeout.

So, all in all, as a first experience with RIOT-OS on the TTGO board, excepting the Lorawan stack, everything seems to work as expected. Regarding the Lorawan stack, probably going to try the LMIC Arduino implementation to see if it is possible to port it to RIOT-OS.

ESP32 TTGO board and Nordic Thingy:52 sensor device

The Nordic Thingy:52 is a 40€/50€ device based on the nRF52832 Nordic microcontroller that has, in a ready to use package, several environmental sensors, that can be accessed by low power Bluetooth (BLE). Nordic provides a complete solution that comprises the Thingy:52 firmware already flashed on the device (Source at GitHub) and an very nice Android Nordic Thingy:52 application, with also sources available at GitHub.

Anyway I have some of these devices for some months now, for other uses, but I decided to test the ESP32 based boards, since the ESP32 has Bluetooth and theoretically can connect and gather gather data from the Thingy. So this post is about the use of the TTGO ESP32 Lora based boards with an OLED to gather data, show it on the OLED, and send it to The Things Network. Seems simple, right?

So when a application connects to the Thingy:52 it can be notified when a sensor value changes throught the standard BLE notification mechanisms. The way the Thingy firmware works, this notification happens at a fixed intervals instead of a value change, and that interval, 5 seconds, 10 seconds, be defined by the Android App or programmatically by our application.

The application is developed by using the PlatformIO and for using the ESP32 Bluetooth interface, I’ve used the NKolban ESP32 BLE Library that happens to be library 1841 at the Platformio repository.

To cut a long story short, as still of today, the ESP32 BLE library doesn’t work correctly with the Thingy:52 notifications. This means that the application subscribes to have notifications, but those never happen. Fortunately someone already hit this problem and solved the issue, but the correction still hasn’t hit the library.

So basically to have my code example to work the following steps are needed:

  1. 1. Clone the TTGO ESP32 repository from . The repository uses the PlatformIO to build the application.
  2. 2. At the root of the repository run the command pio run so that the libraries are downloaded and installed.

At this point we need to correct the Arduino ESP32 library to add the patch to the notification issue.
Just execute the command:

[pcortex@pcortex:ESP32_NordicThingy|master *]$ cd .piolibsdeps/ESP32\ BLE\ Arduino_ID1841/src

At this directory (.piolibsdeps/ESP32\ BLE\ Arduino_ID1841/src edit the file BLERemoteDescriptor.cpp and at around line 151 (the exact line number will probably change in the future) we must change the ::esp_ble_gattc_write_char_descr function parameters:

/**
 * @brief Write data to the BLE Remote Descriptor.
 * @param [in] data The data to send to the remote descriptor.
 * @param [in] length The length of the data to send.
 * @param [in] response True if we expect a response.
 */
void BLERemoteDescriptor::writeValue(
        uint8_t* data,
        size_t   length,
        bool     response) {
    ESP_LOGD(LOG_TAG, ">> writeValue: %s", toString().c_str());
    // Check to see that we are connected.
    if (!getRemoteCharacteristic()->getRemoteService()->getClient()->isConnected()) {
        ESP_LOGE(LOG_TAG, "Disconnected");
        throw BLEDisconnectedException();
    }

    esp_err_t errRc = ::esp_ble_gattc_write_char_descr(
        m_pRemoteCharacteristic->getRemoteService()->getClient()->getGattcIf(),
        m_pRemoteCharacteristic->getRemoteService()->getClient()->getConnId(),
        getHandle(),
        length,                           // Data length
        data,                             // Data
        ESP_GATT_WRITE_TYPE_NO_RSP,
        ESP_GATT_AUTH_REQ_NONE
    );
    if (errRc != ESP_OK) {
        ESP_LOGE(LOG_TAG, "esp_ble_gattc_write_char_descr: %d", errRc);
    }
    ESP_LOGD(LOG_TAG, "<< writeValue");
} // writeValue

We need to change the highlighted line to:

    esp_err_t errRc = ::esp_ble_gattc_write_char_descr(
        m_pRemoteCharacteristic->getRemoteService()->getClient()->getGattcIf(),
        m_pRemoteCharacteristic->getRemoteService()->getClient()->getConnId(),
        getHandle(),
        length,                           // Data length
        data,                             // Data
        response ? ESP_GATT_WRITE_TYPE_RSP : ESP_GATT_WRITE_TYPE_NO_RSP,
        ESP_GATT_AUTH_REQ_NONE
    );

With this change, the code at my github repositories has a working example:

– The ESP32 connects to the Nordic Thingy:52 device.
– It programs the Nordic Device to notify sensor values each 5 seconds (in real use cases it should be much larger period)
– Current sensor data is shown on the serial terminal.

What needs to be done:
– When notified by the Thingy:52, the ESP32 shows the new data on the OLED screen (WIP – Work in progress).
– To keep the application obeying the ISM bands duty cycle, it collects the data, calculates the medium, and sends the data to the Things network each 10 minutes (Also work in progress).

Docker container web interface – Portainer and Riot-OS Development

This post is a follow up of starting up with RIOT-OS. To be able to develop with RIOT-OS an easy (and easier) way to do so is just to install docker and web UI docker interface Portainer to control docker.

So we will install Docker, Portainer, and finally the RIOT-OS building environment.

Installing Docker and Portainer, is an initial stepping stone for using the dockerized development environment for RIOT-OS, since I don’t want to install all the development environments in my machine.

Installing Docker:
On Arch-Linux is as simple as installing the Docker package using pacman, enabling the services and rebooting.
Basically we need to run, as root the following commands:

pacman -S docker
systemctl enable docker.service
reboot

After rebooting the following command should return some information

docker info

A sample output is:

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 9
Server Version: 18.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
...
...
...

Installing Portainer
Installing the Docker Portainer Web UI is as simple as:

docker pull portainer/portainer

To run Portainer a set of complete instructions on this page, but basically on the simplest way is:

$ docker volume create portainer_data
$ docker run -d -p 9000:9000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer

We can now check if the docker image is up:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS                    NAMES
7a38ae7fc922        portainer/portainer   "/portainer"        4 seconds ago       Up 3 seconds        0.0.0.0:9000->9000/tcp   portainer

Since I have already ran the Portainer container, the initial setting up steps when accessing the URL HTTP://localhost:9000 do not appear, but we need to choose:

  1. A set of credentials to use as de administrator for portainer
  2. The local machine registry to connect to the local docker containers.

1- At initial access we define an user and password:

Portainer Credentials

2- Then we connect to our local docker instance:
Portainer Local Docker

Press Connect and then we can now access our Docker instance from Portainer:
Portainer Main Screen

Pressing the Local Docker Connection we can now manage our docker resources.

Installing the build environment for RIOT-OS
We can do it by two ways:

From the command line:

docker pull riot/riotbuild

or use Portainer:

This container is very big, so we need to wait some time for the container image download. The command line shows in greater detail the download process.

After the image is downloaded, we can follow these instructions for building our apps using the docker container as the build environment.

After the image is installed:

To use is is as simple as going the the examples directory and do:

make BUILD_IN_DOCKER=1

From this we are now able to build based RIOT-OS applications for several targets, including the ESP8266/ESP32.

As we can see we even don’t need to have a running container, just the image.

Simple BLE bridge to TTN Lora using the TTGO ESP32 LoRa32 board

The TTGO LoRa32 is an ESP32 based board that features Wifi and BlueTooth low energy but also includes an external Lora chip, in my case the SX1276 868Mhz version.

The following code/hack is just to test the feasibility of bridging BLE devices over the ESP32 and then to Lorawan, more specifically sending BLE data to the LoraWan TTN network.

I’m using Neil Koban ESP32 BLE library, that under platformIO is library number 1841 and the base ABP code for connecting to TTN.

In simple terms this code just makes the ESP32 to emulate a BLE UART device for sending and receiving data. It does that by using the Nordic UART known UUID for specifying the BLE UART service and using also the Nordic mobile applications, that supports such device, for sending/receiving data.

Using the Nordic mobile Android phone applications, data can be sent to the Lora32 board either by using the excellent Nordic Connect application or by also using the simpler and direct Nordic UART application.

The tests program just receives data through BLE and buffers it onto an internal message buffer that, periodically, is sent through Lora to the TTN network. I’ve decided arbitrary that the buffer is 32 bytes maximum. We should keep our message size to the necessary minimum, and also just send few messages to keep the lorawan duty factor usage within the required limits.

So, using the following code we can use our phone to scan from the ESP32 BLE device named TTGOLORAESP32 connect to it and send data to the device.

After a while, when the transmission event fires up, data is transmitted, and the BLE device just receives a simple notification with the EV_TXCOMPLETE message.

That’s it.

The ESP32 Oled Lora TTGO LoRa32 board and connecting it to TTN

The TTGO LoRa32 board is an ESP32 based board that has both an Oled and a Lora transceiver, in my case, the SX1276 transceiver for the 868Mhz band. So it is very similar to some of the ESP32 Oled boards available on the Internet. The board looks like this:

And the interesting part of this board is the new Wifi antenna located in the back that is made of bend metal:

The board also has a LiPo connector, and probably a charger circuit, since I haven’t tried it yet, a user controlled blue led, and a very dim red power led. The led is so dim that at first I thought the board was broken/short circuited, but it is normal.
The Lora Antenna is connected by U.FL/IPEX connector. Both a U.FL to SMA adapter cable is provided and also a cable to connect to the LiPo connector.

An important information to use this board for the LMIC LoraWan based communication is the location of the Lora transceiver DI01 and DIO2 pins. Fortunately they are exposed and connected internally to the ESP32 processor GPIO33 and GPIO32 pins respectively. I’ve updated the pin out for this board:

TTGOESP32Lora_wrongPins

EDIT: Thanks to Andreas on the comment section to point out that this image, while is correct for my board version (with the “3D” metal antenna under the board), the pin labels ARE WRONG. So much for copy it from the seller page.

The (so far yet…) pins mapping are on the bellow image. I’ve checked with my physical board and it seems right now.

I hope this corrects definitely the issue.

So back to basics, the LMIC definition pins for using this board are:

const lmic_pinmap lmic_pins = {
    .nss = 18,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = 14,
    .dio = {26, 33, 32}  // Pins for the Heltec ESP32 Lora board/ TTGO Lora32 with 3D metal antenna
};

The Blue Led Pin is at Pin 2, and according to the sample code the Oled Display is at I2C address 0x3C. The I2C bus where the OLed is at SDA pin 4 and SCLK pin 15.

Also it seems there are at least two revisions for the ESP32 silicon, Revision 0 (Zero) for the initial one, and the latest, at the current date, Revision one.

By executing the Andreas Spiess revision check code it seems that my board is using the latest revision:

REG_READ(EFUSE_BLK0_RDATA3_REG) 1000000000000000
EFUSE_RD_CHIP_VER_RESERVE_S 1100
EFUSE_RD_CHIP_VER_RESERVE_V 111

Chip Revision (official version): 1
Chip Revision from shift Operation 1

Programming the board:
The board can be programmed easily with Platformio IDE by selecting as the target board the Heltec Wifi Lora board. Probably both boards are identical.

The platformio.ini file is as follows:

[env:heltec_wifi_lora_32]
platform = espressif32
board = heltec_wifi_lora_32
framework = arduino

For supporting the OLed and the Lora transceiver we also need to install the ESP8266_SSD1306 lib (ID: 562) and the IBM LMIC library (ID: 852) by either manually installing them on the project root or by adding the following line to the platformio.ini file:

[env:heltec_wifi_lora_32]
platform = espressif32
board = heltec_wifi_lora_32
framework = arduino
lib_deps= 852, 562

With this, the sample TTN INO sketchs for connecting either through ABP or OTAA work flawlessly without any issue by using the above LMIC pins configuration.

The sample sketch for the board: Connecting to TTN and display the packet RSSI:
Since we have the OLed, we can use the RX window to display the received RSSI of our messages on the gateway. This only works if the downlink messages from the gateway can reach back our node, so it might not work always. In my case, I’m about 3Km from the gateway in dense urban area, and not always I can display the packet RSSI.

How this works? Simple, just send our packet, and on the backend we send back the received RSSI as downlink message by using Node-Red, the TTN nodes, and some code:

Since our packet can be received by several gateways, we iterate over the TTN message and calculate the better RSSI and SNR:

// Build an object that allows us to track
// node data better than just having the payload

//For the debug inject node. Comment out when in real use
//var inmsg = msg.payload;
var inmsg = msg;  // from the TTN node

var newmsg = {};
var devicedata = {};
var betterRSSI = -1000;  // Start with a low impossible value
var betterSNR = -1000;

// WARNING only works with String data
// Use TTN decode functions is a better idea
var nodercvdata = inmsg.payload.toString("utf-8");

devicedata.device = inmsg.dev_id;
devicedata.deviceserial = inmsg.hardware_serial;
devicedata.rcvtime = inmsg.metadata.time;
devicedata.nodedata = nodercvdata;

// Iterate over the gateway data to get the best RSSI and SNR data
var gws = inmsg.metadata.gateways;

for ( var i = 0 ; i  betterRSSI )
        betterRSSI = gw.rssi;
        
    if ( gw.snr > betterSNR )
        betterSNR = gw.snr;
}

devicedata.rssi = betterRSSI;
devicedata.snr = betterSNR;

newmsg.payload = devicedata;

return newmsg;

We build then the response object and send it back to the TTN servers that send it to our node. The received data is then displayed on the Oled.

The Node-Red code is as follows:

[{"id":"d4536a72.6e6d7","type":"ttn message","z":"66b897a.7ab5c68","name":"TTN APP Uplink","app":"b59d5696.cde318","dev_id":"","field":"","x":140,"y":220,"wires":[["facbde95.14894"]]},{"id":"facbde95.14894","type":"function","z":"66b897a.7ab5c68","name":"Calculate better RSSI","func":"// Build an object that allows us to track\n// node data better than just having the payload\n\n//For the debug inject node. Comment out when in real use\n//var inmsg = msg.payload;\nvar inmsg = msg;  // from the TTN node\n\nvar newmsg = {};\nvar devicedata = {};\nvar betterRSSI = -1000;  // Start with a low impossible value\nvar betterSNR = -1000;\n\n// WARNING only works with String data\n// Use TTN decode functions is a better idea\nvar nodercvdata = inmsg.payload.toString(\"utf-8\");\n\ndevicedata.device = inmsg.dev_id;\ndevicedata.deviceserial = inmsg.hardware_serial;\ndevicedata.rcvtime = inmsg.metadata.time;\ndevicedata.nodedata = nodercvdata;\n\n// Iterate over the gateway data to get the best RSSI and SNR data\nvar gws = inmsg.metadata.gateways;\n\nfor ( var i = 0 ; i  betterRSSI )\n        betterRSSI = gw.rssi;\n        \n    if ( gw.snr > betterSNR )\n        betterSNR = gw.snr;\n}\n\ndevicedata.rssi = betterRSSI;\ndevicedata.snr = betterSNR;\n\nnewmsg.payload = devicedata;\n\nreturn newmsg;","outputs":1,"noerr":0,"x":400,"y":220,"wires":[["1ac970ec.4cfabf","94515e56.904228"]]},{"id":"1ac970ec.4cfabf","type":"debug","z":"66b897a.7ab5c68","name":"","active":false,"console":"false","complete":"payload","x":670,"y":260,"wires":[]},{"id":"2bea15d8.18f88a","type":"ttn send","z":"66b897a.7ab5c68","name":"TTN APP Downlink","app":"b59d5696.cde318","dev_id":"","port":"","x":970,"y":100,"wires":[]},{"id":"94515e56.904228","type":"function","z":"66b897a.7ab5c68","name":"set Payload","func":"msg.dev_id  = msg.payload.device;\nmsg.payload = Buffer.from(\"RSSI: \" + msg.payload.rssi);\n\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":100,"wires":[["2bea15d8.18f88a","cd04abb9.ccd278"]]},{"id":"cd04abb9.ccd278","type":"debug","z":"66b897a.7ab5c68","name":"","active":true,"console":"false","complete":"true","x":930,"y":200,"wires":[]},{"id":"b59d5696.cde318","type":"ttn app","z":"","appId":"TTNAPPLICATIONID","region":"eu","accessKey":"ttn-account-v2.CHANGEMECHANGEME"}]

Just make sure that we have the TTN nodes installed, and change the credentials for your TTN Application.

On the TTGO ESP32 Lora32 board we just modify the event handling code to display the downlink message:

void onEvent (ev_t ev) {
    if (ev == EV_TXCOMPLETE) {
        display.clear();
        display.drawString (0, 0, "EV_TXCOMPLETE event!");


        Serial.println(F("EV_TXCOMPLETE (includes waiting for RX windows)"));
        if (LMIC.txrxFlags & TXRX_ACK) {
          Serial.println(F("Received ack"));
          display.drawString (0, 20, "Received ACK.");
        }

        if (LMIC.dataLen) {
          int i = 0;
          // data received in rx slot after tx
          Serial.print(F("Data Received: "));
          Serial.write(LMIC.frame+LMIC.dataBeg, LMIC.dataLen);
          Serial.println();

          display.drawString (0, 20, "Received DATA.");
          for ( i = 0 ; i < LMIC.dataLen ; i++ )
            TTN_response[i] = LMIC.frame[LMIC.dataBeg+i];
          TTN_response[i] = 0;
          display.drawString (0, 32, String(TTN_response));
        }

        // Schedule next transmission
        os_setTimedCallback(&sendjob, os_getTime()+sec2osticks(TX_INTERVAL), do_send);
        digitalWrite(LEDPIN, LOW);
        display.drawString (0, 50, String (counter));
        display.display ();
    }
}

For example we can now see on the serial port monitor:

EV_TXCOMPLETE (includes waiting for RX windows)
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Data Received: RSSI: -118
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Data Received: RSSI: -114
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Data Received: RSSI: -105

Thats it!

Some final notes:
Probably not related to the board, but when connecting it to an USB3 port, the Linux Operating system was unable to configure a device for the board. Connecting it to an USB2 port worked flawlessly:

usb 2-1: new full-speed USB device number 2 using xhci_hcd
usb 2-1: string descriptor 0 read error: -71
usb 2-1: can't set config #1, error -71      

As additional information the serial chip on this board is an umarked CP210x chip:

usb 4-1.3: new full-speed USB device number 6 using ehci-pci
cp210x 4-1.3:1.0: cp210x converter detected
usb 4-1.3: cp210x converter now attached to ttyUSB0

lsusb:

Bus 004 Device 006: ID 10c4:ea60 Cygnal Integrated Products, Inc. CP2102/CP2109 UART Bridge Controller [CP210x family]

I haven’t yet tried the WiFi and checked if the metal antenna is any good, but with my preliminary tests, it seems it’s not very good.

Sample code:

Sample code for the board is on this github link: https://github.com/fcgdam/TTGO_LoRa32