Simple BLE bridge to TTN Lora using the TTGO ESP32 LoRa32 board

The TTGO LoRa32 is an ESP32 based board that includes an external Lora chip, and also Wifi and BlueTooth low energy incorporated on the ESP32 chip.

The following code/hack is just to test the feasibility of bridging BLE devices over the ESP32 and then to Lora, more specifically to the TTN network.

I’m using Neil Koban ESP32 BLE library that under platformIO is library number 1841 and the base ABP code for connecting to TTN.

In simple terms the code just emulates an UART device for sending and receiving data by using the Nordic UART know UUID for specifying the BLE service and using also the Nordic mobile applications that supports such device. Using the mobile Android phone applications, data can be sent to the Lora32 board either by using the excellent Nordic Connect application or by also using the simpler and direct Nordic UART application.

The tests program just receives data through BLE and buffers it onto an internal message buffer that, periodically, is sent through Lora to the TTN network. I’ve decided arbitrary that the buffer is 32 bytes maximum. We should keep our message size to the necessary minimum.

So, using the following code we can use our phone to scan fro the ESP32 BLE device named TTGOLORAESP32 connect to it and send data to the device.

After a while, when the transmission event fires up, data is transmitted, and the BLE device just receives a simple notification with the EV_TXCOMPLETE message.

That’s it.

Advertisements

The ESP32 Oled Lora TTGO LoRa32 board and connecting it to TTN

The TTGO LoRa32 board is an ESP32 based board that has both an Oled and a Lora transceiver, in my case, the SX1276 transceiver for the 868Mhz band. So it is very similar to some of the ESP32 Oled boards available on the Internet. The board looks like this:

And the interesting part of this board is the new Wifi antenna located in the back that is made of bend metal:

The board also has a LiPo connector, and probably a charger circuit, since I haven’t tried it yet, a user controlled blue led, and a very dim red power led. The led is so dim that at first I thought the board was broken/short circuited, but it is normal.
The Lora Antenna is connected by U.FL/IPEX connector. Both a U.FL to SMA adapter cable is provided and also a cable to connect to the LiPo connector.

An important information to use this board for the LMIC LoraWan based communication is the location of the Lora transceiver DI01 and DIO2 pins. Fortunately they are exposed and connected internally to the ESP32 processor GPIO33 and GPIO32 pins respectively. I’ve updated the pin out for this board:

TTGO LoRa32 Pinout

So the LMIC definition pins for using this board are:

const lmic_pinmap lmic_pins = {
    .nss = 18,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = 14,
    .dio = {26, 33, 32}  // Pins for the Heltec ESP32 Lora board/ TTGO Lora32 with 3D metal antenna
};

The Blue Led Pin is at Pin 2, and according to the sample code the Oled Display is at I2C address 0x3C. The I2C bus where the OLed is at SDA pin 4 and SCLK pin 15.

Also it seems there are at least two revisions for the ESP32 silicon, Revision 0 (Zero) for the initial one, and the latest, at the current date, Revision one.

By executing the Andreas Spiess revision check code it seems that my board is using the latest revision:

REG_READ(EFUSE_BLK0_RDATA3_REG) 1000000000000000
EFUSE_RD_CHIP_VER_RESERVE_S 1100
EFUSE_RD_CHIP_VER_RESERVE_V 111

Chip Revision (official version): 1
Chip Revision from shift Operation 1

Programming the board:
The board can be programmed easily with Platformio IDE by selecting as the target board the Heltec Wifi Lora board. Probably both boards are identical.

The platformio.ini file is as follows:

[env:heltec_wifi_lora_32]
platform = espressif32
board = heltec_wifi_lora_32
framework = arduino

For supporting the OLed and the Lora transceiver we also need to install the ESP8266_SSD1306 lib (ID: 562) and the IBM LMIC library (ID: 852) by either manually installing them on the project root or by adding the following line to the platformio.ini file:

[env:heltec_wifi_lora_32]
platform = espressif32
board = heltec_wifi_lora_32
framework = arduino
lib_deps= 852, 562

With this, the sample TTN INO sketchs for connecting either through ABP or OTAA work flawlessly without any issue by using the above LMIC pins configuration.

The sample sketch for the board: Connecting to TTN and display the packet RSSI:
Since we have the OLed, we can use the RX window to display the received RSSI of our messages on the gateway. This only works if the downlink messages from the gateway can reach back our node, so it might not work always. In my case, I’m about 3Km from the gateway in dense urban area, and not always I can display the packet RSSI.

How this works? Simple, just send our packet, and on the backend we send back the received RSSI as downlink message by using Node-Red, the TTN nodes, and some code:

Since our packet can be received by several gateways, we iterate over the TTN message and calculate the better RSSI and SNR:

// Build an object that allows us to track
// node data better than just having the payload

//For the debug inject node. Comment out when in real use
//var inmsg = msg.payload;
var inmsg = msg;  // from the TTN node

var newmsg = {};
var devicedata = {};
var betterRSSI = -1000;  // Start with a low impossible value
var betterSNR = -1000;

// WARNING only works with String data
// Use TTN decode functions is a better idea
var nodercvdata = inmsg.payload.toString("utf-8");

devicedata.device = inmsg.dev_id;
devicedata.deviceserial = inmsg.hardware_serial;
devicedata.rcvtime = inmsg.metadata.time;
devicedata.nodedata = nodercvdata;

// Iterate over the gateway data to get the best RSSI and SNR data
var gws = inmsg.metadata.gateways;

for ( var i = 0 ; i  betterRSSI )
        betterRSSI = gw.rssi;
        
    if ( gw.snr > betterSNR )
        betterSNR = gw.snr;
}

devicedata.rssi = betterRSSI;
devicedata.snr = betterSNR;

newmsg.payload = devicedata;

return newmsg;

We build then the response object and send it back to the TTN servers that send it to our node. The received data is then displayed on the Oled.

The Node-Red code is as follows:

[{"id":"d4536a72.6e6d7","type":"ttn message","z":"66b897a.7ab5c68","name":"TTN APP Uplink","app":"b59d5696.cde318","dev_id":"","field":"","x":140,"y":220,"wires":[["facbde95.14894"]]},{"id":"facbde95.14894","type":"function","z":"66b897a.7ab5c68","name":"Calculate better RSSI","func":"// Build an object that allows us to track\n// node data better than just having the payload\n\n//For the debug inject node. Comment out when in real use\n//var inmsg = msg.payload;\nvar inmsg = msg;  // from the TTN node\n\nvar newmsg = {};\nvar devicedata = {};\nvar betterRSSI = -1000;  // Start with a low impossible value\nvar betterSNR = -1000;\n\n// WARNING only works with String data\n// Use TTN decode functions is a better idea\nvar nodercvdata = inmsg.payload.toString(\"utf-8\");\n\ndevicedata.device = inmsg.dev_id;\ndevicedata.deviceserial = inmsg.hardware_serial;\ndevicedata.rcvtime = inmsg.metadata.time;\ndevicedata.nodedata = nodercvdata;\n\n// Iterate over the gateway data to get the best RSSI and SNR data\nvar gws = inmsg.metadata.gateways;\n\nfor ( var i = 0 ; i  betterRSSI )\n        betterRSSI = gw.rssi;\n        \n    if ( gw.snr > betterSNR )\n        betterSNR = gw.snr;\n}\n\ndevicedata.rssi = betterRSSI;\ndevicedata.snr = betterSNR;\n\nnewmsg.payload = devicedata;\n\nreturn newmsg;","outputs":1,"noerr":0,"x":400,"y":220,"wires":[["1ac970ec.4cfabf","94515e56.904228"]]},{"id":"1ac970ec.4cfabf","type":"debug","z":"66b897a.7ab5c68","name":"","active":false,"console":"false","complete":"payload","x":670,"y":260,"wires":[]},{"id":"2bea15d8.18f88a","type":"ttn send","z":"66b897a.7ab5c68","name":"TTN APP Downlink","app":"b59d5696.cde318","dev_id":"","port":"","x":970,"y":100,"wires":[]},{"id":"94515e56.904228","type":"function","z":"66b897a.7ab5c68","name":"set Payload","func":"msg.dev_id  = msg.payload.device;\nmsg.payload = Buffer.from(\"RSSI: \" + msg.payload.rssi);\n\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":100,"wires":[["2bea15d8.18f88a","cd04abb9.ccd278"]]},{"id":"cd04abb9.ccd278","type":"debug","z":"66b897a.7ab5c68","name":"","active":true,"console":"false","complete":"true","x":930,"y":200,"wires":[]},{"id":"b59d5696.cde318","type":"ttn app","z":"","appId":"TTNAPPLICATIONID","region":"eu","accessKey":"ttn-account-v2.CHANGEMECHANGEME"}]

Just make sure that we have the TTN nodes installed, and change the credentials for your TTN Application.

On the TTGO ESP32 Lora32 board we just modify the event handling code to display the downlink message:

void onEvent (ev_t ev) {
    if (ev == EV_TXCOMPLETE) {
        display.clear();
        display.drawString (0, 0, "EV_TXCOMPLETE event!");


        Serial.println(F("EV_TXCOMPLETE (includes waiting for RX windows)"));
        if (LMIC.txrxFlags & TXRX_ACK) {
          Serial.println(F("Received ack"));
          display.drawString (0, 20, "Received ACK.");
        }

        if (LMIC.dataLen) {
          int i = 0;
          // data received in rx slot after tx
          Serial.print(F("Data Received: "));
          Serial.write(LMIC.frame+LMIC.dataBeg, LMIC.dataLen);
          Serial.println();

          display.drawString (0, 20, "Received DATA.");
          for ( i = 0 ; i < LMIC.dataLen ; i++ )
            TTN_response[i] = LMIC.frame[LMIC.dataBeg+i];
          TTN_response[i] = 0;
          display.drawString (0, 32, String(TTN_response));
        }

        // Schedule next transmission
        os_setTimedCallback(&sendjob, os_getTime()+sec2osticks(TX_INTERVAL), do_send);
        digitalWrite(LEDPIN, LOW);
        display.drawString (0, 50, String (counter));
        display.display ();
    }
}

For example we can now see on the serial port monitor:

EV_TXCOMPLETE (includes waiting for RX windows)
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Data Received: RSSI: -118
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Data Received: RSSI: -114
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Data Received: RSSI: -105

Thats it!

Some final notes:
Probably not related to the board, but when connecting it to an USB3 port, the Linux Operating system was unable to configure a device for the board. Connecting it to an USB2 port worked flawlessly:

usb 2-1: new full-speed USB device number 2 using xhci_hcd
usb 2-1: string descriptor 0 read error: -71
usb 2-1: can't set config #1, error -71      

As additional information the serial chip on this board is an umarked CP210x chip:

usb 4-1.3: new full-speed USB device number 6 using ehci-pci
cp210x 4-1.3:1.0: cp210x converter detected
usb 4-1.3: cp210x converter now attached to ttyUSB0

lsusb:

Bus 004 Device 006: ID 10c4:ea60 Cygnal Integrated Products, Inc. CP2102/CP2109 UART Bridge Controller [CP210x family]

I haven’t yet tried the WiFi and checked if the metal antenna is any good, but with my preliminary tests, it seems it’s not very good.

Sample code:

Sample code for the board is on this github link: https://github.com/fcgdam/TTGO_LoRa32

Using the BSFrance Lora32U4 board to connect to the Things Network Lorawan

The BSFrance Lora32u4 II (Lora32U4II for helping Google out) board is an Atmega32U4 processor with a HDP13 Lora transceiver on the same board. As far as I’m aware, the HDP13 is similar to the RFM95W (including pinout), and in my case it seems it has an original Semtech SX1276 (868Mhz radio transceiver) chip installed on the HDP13 module. This board is similar to the Adafruit 32U4 Lora feather, if not equal… (possible schematics for the Lora32u4 board)

The board hardware includes beside the Lora HDP13 module a LiPo connector with an 2 pin JST PH 2.0mm pin spacing connector and the power supporting electronics.
There are two leds, on orange for LiPo and charger status, that blinks very fast when no LiPo is connected, and a very bright white led that fades in and out when the bootloader is in the programming mode or programming is ongoing. After the bootloader exists and starts the main program, the led shuts off.
This led, as usual in Arduino boards is connected to I/O pin 13, so it is software controllable.

Also the only way to power up the board is either trough the USB port, LiPo battery or 5V to an input pin. No other voltages, like RAW voltages above 5V are supported.

As a final note, the board that I’ve bought also came with an uFL adapter cable for SMA, an antenna and a link for accessing documentation, so, excluding the LiPo battery, the complete kit.

Starting up using the board:

I’m testing the board to send data to the Things Network and doing so by using PlatformioIO as the developing IDE. Platformio IDE is much better than the Arduino IDE, since each project has it’s own depending libraries directory .piolibdeps which we can modify and edit the library code without breaking other projects.

The platformio.ini board definition for the Lora32u4II board is just a clone of Adafruit feather 32u4:

[env:feather32u4]
platform = atmelavr
board = feather32u4
framework = arduino

As the code to send data to the TTN network, I’ve just used ABP lorawan device connection that I’ve used on my previous hand build node.

I’m testing the node with both the IBM LMIC Library (ID: 852) and the Arduino LMIC Library (ID: 1729).

After setting the correct keys and device ID, all we need is to change the LMIC pins configuration for this board: LoRa32u4II pinout diagram

According to documentation the pins are:

  1. nss (SS – Chip Select): Pin 8
  2. rst (Reset): Pin 4
  3. DIO (Lora TX/RX indicator): Pin 7

So the Lmic Pins configuration is:

const lmic_pinmap lmic_pins = {
    .nss = 8,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = 4,
    .dio = {7, 6 , LMIC_UNUSED_PIN}
};

Regarding Pin 6, is the chosen pin to connect to the DIO1 pin. This pin signals receive timeouts generated by the radio module. The connection of this pin is required for LMIC and for the onEvent() function signaling of EV_TXCOMPLETE to be triggered/fired, otherwise the onEvent() funciton is never called.
Since this is a LoraWan Class A node, after the transmission, two receive windows are opened for any downlink data that might be sent to the node. The DIO1 pin signals the receive timeout, and at the end of the receive windows, triggers the EV_TXCOMPLETE event. I’ve tried to use other pins, for example, pin 3, but then the EV_TXCOMPLETE event was never fired… Strange. Anyway, with the above configuration and with DIO1 connected through a wire bridge to pin 6 works fine.

If we do not connect DIO1 by removing the DIO1 pin configuration:

 .dio = {7, LMIC_UNUSED_PIN , LMIC_UNUSED_PIN}

with the platformio IBM Lmic library (Id: 852), or with the Arduino LMIC Library the LMIC fails. An example:

pio device monitor --port /dev/ttyACM0 --baud 115200
[cortex@brightlight:TTN32u4ABP]$ pio device monitor --port /dev/ttyACM0 --baud 115200
--- Miniterm on /dev/ttyACM0  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Starting...
FAILURE
.piolibdeps/IBM LMIC framework_ID852/src/hal/hal.cpp:24

The line hal.cpp:24 point to an ASSERT that doesn’t allow a LMIC_UNUSED_PIN for DIO1.

Putting pin 6 and making sure that it is connected to DI1 is required. Otherwise if the pin is defined but not connected we have the following behaviour:

--- Miniterm on /dev/ttyACM0  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Starting...
Sending uplink packet...

As we can see the EV_TXCOMPLETE event is never fired, and the associated reschedule of another transmission never happens, since that code is inside the code for the EV_TXCOMPLETE event. The only way, in this case, is to reset the board so another transmission happens.

So if using the above LMIC pins configuration and connecting DIO1 to pin 6, sending data to the The Things Network works just fine:

Data received at the TTN side

Some final notes, tips and tricks:

The ATMega 32U4 USB Serial port:
The ATMega 32U4 USB serial port is a bit fiddly when using it from the Arduino framework. At reset or connection first the USB port is used by the bootloader (white led fading in and out). After a while the board starts to execute the flash program (white led off), but it resets the USB port. The host computer might have an issue with this and fails to assign an USB address.

The solution is just to add at the start of the setup function a delay:

void setup() {
  delay(2500);   // Give time to the ATMega32u4 port to wake up and be recognized by the OS.
  
  Serial.begin(115200);
...
...

Using minicom instead of PlatformIO serial monitor:
This one is quite simple to explain, since minicom survives to the USB port resets since they appear and disappear through the board reset.
Against it, is that we need to explicitly exit minicom to be able to program the board.

# minicom -D /dev/ttyACM0 -b 115200

The PlatformIO Arduino LMIC library is outdated:
This is solved now. Lib 852 is now updated.
The Arduino LMIC version (1729) on the PlatformIO is outdated, since, for example doesn’t have neither the LMIC_UNUSED_PIN definition and the LMIC_setClockError function needed for a successful OTAA TTN network join.

The solution is just clone the Arduino LMIC library and copy the src folder to .piolibdeps/IBM LMIC framework_ID852/ removing the original src folder version.

Comparing Library sizes:

Using the IBM LMIC Library (ID:852) with PINGS and BEACONS disabled on the config.h file, otherwise it doesn’t fit on the 32u4 32K flash space, our sketch uses the following space:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   26040 bytes (79.5% Full)
(.text + .data + .bootloader)

Data:       1014 bytes (39.6% Full)
(.data + .bss + .noinit)

Using the Arduino LMIC library (ID: 1729) with PINGS and BEACONS enabled, but a more efficient AES implementation, we get:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   22776 bytes (69.5% Full)
(.text + .data + .bootloader)

Data:        954 bytes (37.3% Full)
(.data + .bss + .noinit)

With PINGS and BEACONS disabled we get:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   19032 bytes (58.1% Full)
(.text + .data + .bootloader)

Data:        903 bytes (35.3% Full)
(.data + .bss + .noinit)

So we get, with this last change, and while keeping support for OTTA, at least 8K/9K for program space not related to the Lorawan/TTN code support.

TTN LoraWan Atmega32U4 based node – ABP version

TTN is the The Things Network that provides the required backend services and infra-structure for supporting IoT (Internet of Things) connectivity that uses the LORAWAN protocol.

Anybody can participate on the Things Network by either providing the radio gateways that feed the received data to the TTN backend that, then, delivers it to the user applications, and so increasing the coverage of the TTN network, or just use the network by building TTN Lorawan nodes.

This post is regarding the later case, the build of a simple node based on an Arduino board: the Arduino Micro Pro. So why the Micro PRO, these are quite more expensive than the normal Arduinos, but come in two versions: 5V and 3.3V.
Since I’m using the SX1276 Lora radio that works with 3.3V, I’ve chosen the 3.3V Arduino Pro version so that I do not need to use level shifters if using a 5V based board. Also the Arduino Micro PRO chip, the Atmega32u4 has embedded USB connectivity/port, so no need for serial adapters and/or supporting chips which, at the end, might lead to lower power consumption.

Right now, on sites like eBay and Aliexpress, boards like the Lora32u4 come at least in two versions: with the Atmega328p and with the Atmega32u4. Both suffer the same problem, the Atmel micro processor used only has 32K of RAM available which might be too short to be used for some applications.
This is because the LMIC, the Lorawan stack, takes a huge amount of space if using the original IBM version. A much more memory efficient version for Arduino, originally ported from IBM code, but using a different AES encryption algorithm also exists and saves a lot of memory space. We will see about that. The great advantage of these boards is they also have connection and charger for a LiPo battery, so in reality all we need is to add sensors, battery and our code. An example of such board is the BSFrance Lora32u4 board.

The node build:
While I’m waiting for my Atmega32U4 based Lora32u4 board, I’m using an Hoperf RFM95 radio soldered on board/shield designed for the Wemos ESP8266: Wemos RFM95 Lora shield. this way I can use the RFM radio either on the ESP8266 Wemos based set of boards, or, as in this case, with the Arduino 32u4.

The Hallard shield as one interesting feature that is that merges all the Lora transceiver status pins by using diodes and hence only use one Arduino pin for inquiring Lora SX1276 radio status. This is needed due to the lack of I/O pins on the Wemos ESP8266 board. For this to work on Arduino we need to add a pull-down resistor to the Arduino pin that connects to the merged output. In my case I used a 10K resistor.
The RFM95 radio is controlled using SPI, so we need to use also the SPI Arduino Pins, and also need to connect the Chip Select pin.
The schematics is as follows:

Arduino Pro Micro and RFM95 Wemos Shield

The node software:
After the node hardware build is done, from the software perspective the node needs now at least another two things: the LMIC stack for implementing the Lorawan protocol support over the Lora radio and, at the TTN site, the device configuration.

Since I’m using Platformio to develop, the LMIC library is the library 852: pio lib show 852. We need to install it and add the reference to it on the file platformio.ini. Also since there is no ATMega 32U4 board on the Platformio IDE available boards, we can use the Adafruit Feather 32u4 board, which is the same thing:

[env:feather32u4]
platform = atmelavr
board = feather32u4
framework = arduino
lib_install= 852

The device registration can be done so that the node device access the TTN network in two different ways:

  1. ABP – Activation by personalisation – This means that all set of keys required to join the Lorawan network are embedded into the software.
  2. OTAA – Over the Air Activation – The network session keys needed to join the Lorawan network are generated when the device tries to join the network.

On this post we will ABP first, since I have no nearby TTN gateway capable o OTTA (I’m using a single channel gateway without downlink support.).

Anyway, the node code is really nothing special, except the necessary configuration for the LMIC to communicate with our RFM95 board.

On the ABP device registration TTN page we need to register our device, so that, on main.cpp code file we can fill the required keys and device ID.

As a quick introduction, after registering onto the TTN site, we go to the console and choose Applications. We can there create or reuse an existing application and register the device, making sure we choose ABP as the method to join the network.

On the Device EUI field, either we fill it or press the crossing arrows to generate an ID. We let the system generate an ID, and then we can finally press the Register button.

The newly register device is configured as an OTAA device:

So we go to Settings and change the OTAA to ABP. After this step we have the required data to put on our code.

Since our node doesn’t have any memory to track frame counting that survives reboots or power cycles, we disable the frame counter checks.

Don’t forget to press save. Again on the main device screen we can now copy the keys to the code:

We can now copy the keys:

static u1_t NWKSKEY[16] = { 0xEE, ... ... ... ... }; // <- Put here the NETWORK KEY
static u1_t APPSKEY[16] = { 0x4E, 0x12, ... ... ... ... };  // <- Put here the APPLICATION KEY
static u4_t DEVADDR = 0x26304050;   // Put here the device id in hexadecimal form.

Testing:

Compiling the code with the pio run command, we have the following output when using the original IBM LMIC library:

Calculating size .pioenvs/feather32u4/firmware.elf
AVR Memory Usage
----------------
Device: atmega32u4

Program:   28542 bytes (87.1% Full)
(.text + .data + .bootloader)

Data:        957 bytes (37.4% Full)
(.data + .bss + .noinit)

And we can flash the firmware with the command: pio run -t upload.

The result is data on the TTN console referring to our device:

The problem… :
So, everything runs OK, and we can send data to the TTN Network, everything looks good, right?

As soon we start to add functionality to our code, for example reading some I2C sensors, our some serial debug messages, we hit this problem:

Linking .pioenvs/feather32u4/firmware.elf
Checking program size
text       data     bss     dec     hex filename
Error: The program size (28756 bytes) is greater than maximum allowed (28672 bytes)
28548       208     749   29505    7341 .pioenvs/feather32u4/firmware.elf
*** [.pioenvs/feather32u4/firmware.elf] Explicit exit, status 1

So in reality we can’t add much functionality to our code if using a full LMIC stack, since it occupies a lot of the available flash memory.

Trimming down the IBM LMIC stack:
Since our node is ABP only we can strip out some LMIC functionality for OTAA an other Lorawan features. For this we need to edit the config.h file from the LMIC library. Since we are using platformio, this file is located at project_root/.piolibdeps/IBM LMIC framework_ID852/src/lmic

We only leave support for ABP by enabling the disable lines for other LMIC functionality:

...
...
// Any runtime assertion failures are printed to this serial port (or
// any other Print object). If this is unset, any failures just silently
// halt execution.
#define LMIC_FAILURE_TO Serial

// Uncomment this to disable all code related to joining
#define DISABLE_JOIN
// Uncomment this to disable all code related to ping
#define DISABLE_PING
// Uncomment this to disable all code related to beacon tracking.
// Requires ping to be disabled too
#define DISABLE_BEACONS

// Uncomment these to disable the corresponding MAC commands.
// Class A
//#define DISABLE_MCMD_DCAP_REQ // duty cycle cap
//#define DISABLE_MCMD_DN2P_SET // 2nd DN window param
//#define DISABLE_MCMD_SNCH_REQ // set new channel
// Class B
#define DISABLE_MCMD_PING_SET // set ping freq, automatically disabled by DISABLE_PING
#define DISABLE_MCMD_BCNI_ANS // next beacon start, automatical disabled by DISABLE_BEACON

By uncommenting the above lines, our code now takes (we can and should ignore the LMIC compile warnings):

AVR Memory Usage
----------------
Device: atmega32u4

Program:   23324 bytes (71.2% Full)
(.text + .data + .bootloader)

Data:        796 bytes (31.1% Full)
(.data + .bss + .noinit)

So around 5KB less without the OTAA and Class B support.

So we have a bit more memory to do something useful.

Enabling OTAA by commenting the line //#define DISABLE_JOIN:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   25048 bytes (76.4% Full)
(.text + .data + .bootloader)

Data:        912 bytes (35.6% Full)
(.data + .bss + .noinit)

We still have around 3K free. Tight but might be enough.

If using the Arduino ported LMIC library (852) we have:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   18944 bytes (57.8% Full)
(.text + .data + .bootloader)

Data:        813 bytes (31.8% Full)
(.data + .bss + .noinit)

Much better!

Conclusion:
The availability of boards with the AtMega32u4 processor, Lora Radio and LiPo charge and battery connectivity, is a great step to start using the TTN (or other) Lorawan networks. But with only with 32K or flash memory, for some applications, these boards might not be the best solution.

Also the price for such boards are still a bit on the expensive side, since a discrete 32u4 + RFM95 + Lipo charger is a bit cheaper than the single board solution.

Anyway, the STM32F103 blue pill boards cost half of the 32U4 price and have double the flash size and 9x the clock, are also 3.3v compatible and so it would be great that such single Lora boards used the STM32F103 instead of the 328p or 32u4…

So my conclusion is, without power considerations taken into account, a STM32F103 + RFM95 and LiPo charger, is a better alternative than the one that I’ve used here.

Using Node-Red and Grafana WorldMap for geolocalized data visualization

Based on my previous posts we are now able to build a system that can receive, store and visualize data by using Node-Red, InfluxDB and Graphana. Grafana allows us build dashboards, query and visualize the stored data across time efficiently by using, in our case, the InfluxDB database engine. So far we’ve used simple line/bar charts to visualize data but we can use both Node-Red and Grafana to plot data onto a map:

  1. NodeRed Contrib World Map: Openstreet UI based map for plotting data with several options, including icon types, vectors, circles and heatmaps totally controlled through nodered flows.
  2. Grafana WorldMap plugin: Grafana panel with also an OpenStreet map for visualizing data.

Both have pros and cons, but the main differences between the two is that Node-Red Worldmap is suited more to real time display, and the Grafana plugin is better adapted to display data based on some time based query. Other major difference is that Node-Red Worldmap would require some coding, but, at least I consifer it, at an easy level, and the Grafana plugin is much harder to make it work.

Mapping data using Node-Red Worldmap:
One of the easiest ways for mapping data in realtime is using Node Red Worldmap node. The map is plotted and updated in real time.

cd ~
cd .node-red
npm install node-red-contrib-web-worldmap

After restarting and deploying a worldmap node, the map should be available at: http://server:1880/worldmap or other URL depending on the Node-Red base configuration.

One thing to keep in mind is that Node-Red is single user, so all instances of world maps (several different clients/browsers) will always have the same view.
The simplest way to start using the worldmap is just to copy and deploy the demo workflow provided by the node, but the key concept is that each point has a name and a set of coordinates.

msg.payload = {};
msg.payload.name = "CentralLX";
msg.payload.lat = 38.7223;
msg.payload.lon = -9.1393;
msg.payload.layer = "SensorData";
msg.payload.UVLevel = getUVLevel() ;
msg.payload.Temperature = getTemp();

The cool thing is that if we inject repeatedly the above message (keeping the same name) but with different coordinates, the data point will move across the map in real time, and as I said earlier, the move will be reflect onto every client.

So all we need is an Inject node to a function node with the above code and feed it to the world map:

At the end we get this at the URL http://server:1880/worldmap:

Mapping data using Grafana Worldmap plugin:

The Grafana Worldmap plugin can get the location data in several ways. One of them is to use geohash data that is associated to the values/measurements.
There is a Node Red Geohash node that generates the geohash value from the latitude and longitude of data location. As usual we install the node:

cd ~
cd .node-red
npm install node-red-node-geohash

and then the Grafana plugin. We just follow the plugin instructions:

cd ~
grafana-cli plugins install grafana-worldmap-panel
/etc/init.d/grafana-server restart

With this per-requisites installed we can now feed data onto the database, in our case InfluxDB, that will be used by Grafana. We just need make sure that we add the geohash field. The geohash node will calculate from the node-red message properties lat – latitude and lon – longitude the required info:

A simple example:

Using the Influx tool, we can query our database and see that the geohash localization is now set:

> select * from SensorData limit 2
name: DemoValue
time                Temp UVLevel geohash    lat       lon        
----                ---- ------- -------   ---------- ----------  
1490706639630929463 22   8       eyckpjywh 38.7045055 -9.1754669  
1490706651606553077 21   7       eyckpjzjr 38.7044008 -9.1746488 

Anyway for setting up the World map plugin to display the above data was not straight forward, so the following instructions are more for a startup point rather than a solution.

The first thing to know is that the plugin is waiting for two fields: geohash and metric. With this in mind, before wasting too much time with the map plugin, a table panel that is filled with the required query is a precious tool to debug the query:

After we infer from the table that the data is more or less the data we want, we just transfer the query to WorldMap plugin:

Notice two important things: The aliasing for the query field to metric with the alias(metric) instruction, and the Format as: Table.

We can now setup the specific Worldmap settings:

On the Map Visual Options , I’ve centered the map in my location and set the zoom level. Fiddling around here can be seen in real time.

On the Map Data Options for this specific example, the Location Data comes from a table filled with the previous query (hence the format as table on the query output), and we want to see the current values with no aggregation.

When hoovering around a spot plotted on the map we can see a label: value, and the label used is obtained from a table field. In my case I just used geohash (not really useful…). Anyway these changes only work after saving and reloading the panel with F5 in my experience.

At the end we have now graphed data and localized data:

If we drag the selector on the left graphic panel, or select another time interval on top right menu of the grafana dashboard, the visualized information on the map changes.

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 2: Database configuration and data collection

On the previous post we’ve installed and the base software for our Grafana based dash board.

We need now to configure our InfluxDB database and Node Red to start collecting data.

Configuring InfluxDB:
Detailed instructions for configuring an InfluxDB database are on this InfluxDB documentation link..

The main concepts that we need to be aware when using the InfluxDB is that record of data has a time stamp, a set of tags and a measured value. This allows, for example to create a value named Temperature and tag it depending on the source sensor:

Temperature: Value=22.1 , Sensor=Kitchen
Temperature: Value=21.9 , Sensor=Room1

This allows to process all the data or only process data based on a certain tag or tags. Values and tags can be created on the fly without previously define them, which is a bit different from standard RDBMS engines.

Creating an InfluxDB database:
To create the database, we need to access the machine hosting the InfluxDB server and execute the command influx:

odroid@odroid:~$ influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> create database SensorData
> show databases
name: databases
name
----
_internal
SensorData

> 

Now we have our database created and I’ve named SensorData. To make an example with the above temperature data we can do the following:

> insert Temperature,Sensor=kitchen value=22.1
ERR: {"error":"database is required"}

Note: error may be due to not setting a database or retention policy.
Please set a database with the command "use " or
INSERT INTO . 
> use SensorData
Using database SensorData
> 

As we can see we need first to select the database where we are going to insert data with the command use SensorData:

> use SensorData
Using database SensorData
> insert Temperature, Sensor=kitchen value=22.1
ERR: {"error":"unable to parse 'Temperature, Sensor=kitchen value=22.1': missing tag key"}

> insert Temperature,Sensor=kitchen value=22.1
> insert Temperature,Sensor=Room1 value=21.9
> select * from Temperature
name: Temperature
time                Sensor  value
----                ------  -----
1487939008959909164 kitchen 22.1
1487939056354678353 Room1   21.9

Note that we can’t use spaces between the Measure name and the tags. The correct syntax is as follows:

 insert MeasureName,tag1=t1,tag2=t2,...   value1=val1,value2=val2,value3=val3,....

Also note that no DDL (data definition language) was used to create the tags or the measured value, we’ve just inserted data for our measurement with the our tags and value(s) without the need of previously define the schema.

Configuring Node-Red
Since we now have a database we can configure the InfluxDB Node Red nodes to store data onto the database:

There are two types of InfluxDB nodes, one that has an Input and Output and other that only has Input. The former is for making queries to the database where we provide on the input node the query, and on the output the results are returned. The later is for storing data only onto the database.
For both nodes we need to configure an InfluxDB server:

InfluxDB Server Configuration

We need to press the Pen icon right next to the server to add or reconfigure a new InfluxDB server:

InfluxDB server

A set of credentials are required, but since I’ve yet configured security, we can just put admin/admin as username and password. In a real deployment we must activate security.

From now on it is rather simple. Referring to InfluxDB node configuration screenshot (Not the InfluxDB server configuration) we have a configuration field named Measurement. This is our measure name that we associate a value. Picking up on the above example with the Insert command it will be Temperature, for example.

Now if the msg.payload provided has input to the node is a single value, let’s say 21, this is equivalent to do:

Insert Temperature value=12

We other formats for msg.payload that allows to associate tags and measures. Just check the Info tab for the node.

Simple example:

The following flow shows a simple example of a value received through MQTT, in this case the free heap from one of my ESP8266 and its storage in InfluxDB:

Sample Flow

[{"id":"20bec5de.8881c2","type":"mqtt in","z":"ced40abb.3c92e","name":"Heap","topic":"/outbox/ESP12DASH/Heap","qos":"2","broker":"2a552b3c.de8d2c","x":83.16668701171875,"y":206.41668701171875,"wires":[["e0d9c912.8c57f8","876fb151.6f2fa"]]},{"id":"876fb151.6f2fa","type":"debug","z":"ced40abb.3c92e","name":"","active":true,"console":"false","complete":"false","x":408.5,"y":177,"wires":[]},{"id":"e0d9c912.8c57f8","type":"influxdb out","z":"ced40abb.3c92e","influxdb":"bbd62a93.1a7108","name":"","measurement":"heap","x":446.1666717529297,"y":224.58335876464844,"wires":[]},{"id":"2a552b3c.de8d2c","type":"mqtt-broker","broker":"192.168.1.17","port":"1883","clientid":"node-red","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":15,"cleansession":true,"willQos":"0","birthQos":"0"},{"id":"bbd62a93.1a7108","type":"influxdb","z":"","hostname":"127.0.0.1","port":"8086","protocol":"http","database":"SensorData","name":"ODroid InfluxDB"}]

We can see with this flow the data stored in InfluxDB:

> select * from heap;
name: heap
time                value
----                -----
1487946319638000000 41600
1487946440913000000 41600
1487946562206000000 41600
1487946683474000000 41600
1487946804751000000 41600
1487946926061000000 41600
1487947047309000000 41616
1487947168594000000 41600

Now we have data that we can graph with Grafana, subject of my next posts.

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 1: Installing

A more or less standard software stack used for control, processing and displaying data, has emerged that is almost used by everyone when hacking around on Arduinos, ESP8266, Raspeberry Pi’s and other plethora of devices. This “standard” software stack basically always includes the MQTT protocol, some sort of Web based services, Node-Red and several different cloud based services like Thingspeak, PubNub and so on. For displaying data locally, solutions like Freeboard and Node-Red UI are a great resources, but they only shows current data/status, and has no easy way to see historical data.

So on this post I’ll document a software stack based on Node-Red, InfluxDB and Graphana that I use to store and display data from sensors that I have around while keeping and be able to display historical memory of data. The key asset here is the specialized time-series database InfluxDB that keeps data stored and allows fast retrieval based on time-stamps: 5 minutes ago, the last 7 days, and so on. InfluxDB is not the only Time-Series database that is available, but it integrates directly with Grafana the software that allows the building of dashboards based on stored data.

I’m running an older version of InfluxDB on my ARM based Odroid server, since a long time ago, ARM based builds of InfluxDB and Grafana where not available. This is now not the case, but InfluxDB and Grafana have ARM based builds so we can use them on Raspberry PI and Odroid ARM based boards.

So let’s start:

Setting up Node-Red with InfluxDB
I’ll not detail the Node-Red installation itself since it is already documented thoroughly everywhere. To install the supporting nodes for InfluxDB we need to install the package node-red-contrib-influxdb

cd ~/.node-red
npm install  node-red-contrib-influxdb

We should now restart Node-red to assume/detect the new nodes.

Node Red InfluxDB nodes

Installing InfluxDB
We can go to the InfluxDB downloads page and follow the installation instructions for our platform. In my case I need the ARM build to be used on Odroid.

cd ~
wget https://dl.influxdata.com/influxdb/releases/influxdb-1.2.0_linux_armhf.tar.gz
tar xvzf influxdb-1.2.0_linux_armhf.tar.gz

The InfluxDB engine is now decompressed in the newly created directory influxdb-1.2.0-1. Inside this directory there are the directories that should be copied to the system directories /etc, /usr and /var:

sudo -s
cd /home/odroid/influxdb-1.2.0-1

Copy the files to the right location. I’ve added the -i switch just to make sure that I don’t overwrite nothing.

root@odroid:~/influxdb-1.2.0-1# cp -ir etc/ /etc
root@odroid:~/influxdb-1.2.0-1# cp -ir usr/* /usr
root@odroid:~/influxdb-1.2.0-1# cp -ir var/* /var

We need now to create the influxdb user and group:

root@odroid:~/influxdb-1.2.0-1# groupadd influxdb
root@odroid:~/influxdb-1.2.0-1# useradd -M -s /bin/false -d /var/lib/influxdb -G influxdb influxdb

We need now to change permissions on /var/lib/influxdb:

cd /var/lib
chown influxdb:influxdb influxdb

We can now set up the automatic start up script. On the directory /usr/lib/influxdb/scripts there are scripts for the systemctl based Linux versions and init.d based versions that is my case. So all I have to do is to copy the init.sh script from that directory to the /etc/init.d and link it to my run level:

root@odroid:~# cd /etc/init.d
root@odroid:/etc/init.d# cp /usr/lib/influxdb/scripts/init.sh influxdb
root@odroid:/etc/init.d# runlevel
 N 2
root@odroid:/etc/init.d# cd /etc/rc2.d
root@odroid:/etc/init.d# ln -s /etc/init.d/influxdb S90influxdb

And that’s it. We can now start the database with the command /etc/init.d/influxdb start

root@odroid:~# /etc/init.d/influxdb start
Starting influxdb...
influxdb process was started [ OK ]

We can see the influxdb logs at /var/log/influxdb and start using it through the command line client influx:

root@odroid:~# influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> show databases
name: databases
name
----
_internal

> 

Installing Grafana
We need now to download Grafana. In my case for Odroid since it is an ARMv7 based processor, no release/binary is available.
But a ARM builds are available on this GitHub Repository: https://github.com/fg2it/grafana-on-raspberry for both the Raspberry Pi and other ARM based computer boards, but only for Debian/Ubuntu based OS’s. Just click on download button on the description for the ARMv7 based build and at the end of the next page a download link should be available:

odroid@odroid:~$ wget https://bintray.com/fg2it/deb/download_file?file_path=main%2Fg%2Fgrafana_4.1.2-1487023783_armhf.deb -O grafana.deb

And install:

root@odroid:~# dpkg -i grafana.deb
Selecting previously unselected package grafana.
(Reading database ... 164576 files and directories currently installed.)
Preparing to unpack grafana.deb ...
Unpacking grafana (4.1.2-1487023783) ...
Setting up grafana (4.1.2-1487023783) ...
Installing new version of config file /etc/default/grafana-server ...
Installing new version of config file /etc/grafana/grafana.ini ...
Installing new version of config file /etc/grafana/ldap.toml ...
Installing new version of config file /etc/init.d/grafana-server ...
Installing new version of config file /usr/lib/systemd/system/grafana-server.service ...

Set the automatic startup at boot:

root@odroid:~# ln -s /etc/init.d/grafana-server /etc/rc2.d/S91grafana-server

And we can now start the server:

root@odroid:~# /etc/init.d/grafana-server start
 * Starting Grafana Server    [ OK ] 
root@odroid:~# 

We can now access the server at the address: http://server:3000/ where server is the IP or DNS name of our ODroid or RPi.

Conclusion:
This ends the installation part for the base software.

The following steps are:

  • Create the Influx databases –
  • Receive data from sensors/devices and store it on the previously created database
  • Configure and create Grafana data sources and dashboards
  • Add some plugins to Grafana