Simple BLE bridge to TTN Lora using the TTGO ESP32 LoRa32 board

The TTGO LoRa32 is an ESP32 based board that includes an external Lora chip, and also Wifi and BlueTooth low energy incorporated on the ESP32 chip.

The following code/hack is just to test the feasibility of bridging BLE devices over the ESP32 and then to Lora, more specifically to the TTN network.

I’m using Neil Koban ESP32 BLE library that under platformIO is library number 1841 and the base ABP code for connecting to TTN.

In simple terms the code just emulates an UART device for sending and receiving data by using the Nordic UART know UUID for specifying the BLE service and using also the Nordic mobile applications that supports such device. Using the mobile Android phone applications, data can be sent to the Lora32 board either by using the excellent Nordic Connect application or by also using the simpler and direct Nordic UART application.

The tests program just receives data through BLE and buffers it onto an internal message buffer that, periodically, is sent through Lora to the TTN network. I’ve decided arbitrary that the buffer is 32 bytes maximum. We should keep our message size to the necessary minimum.

So, using the following code we can use our phone to scan fro the ESP32 BLE device named TTGOLORAESP32 connect to it and send data to the device.

After a while, when the transmission event fires up, data is transmitted, and the BLE device just receives a simple notification with the EV_TXCOMPLETE message.

That’s it.

Advertisements

Using the BSFrance Lora32U4 board to connect to the Things Network Lorawan

The BSFrance Lora32u4 II (Lora32U4II for helping Google out) board is an Atmega32U4 processor with a HDP13 Lora transceiver on the same board. As far as I’m aware, the HDP13 is similar to the RFM95W (including pinout), and in my case it seems it has an original Semtech SX1276 (868Mhz radio transceiver) chip installed on the HDP13 module. This board is similar to the Adafruit 32U4 Lora feather, if not equal… (possible schematics for the Lora32u4 board)

The board hardware includes beside the Lora HDP13 module a LiPo connector with an 2 pin JST PH 2.0mm pin spacing connector and the power supporting electronics.
There are two leds, on orange for LiPo and charger status, that blinks very fast when no LiPo is connected, and a very bright white led that fades in and out when the bootloader is in the programming mode or programming is ongoing. After the bootloader exists and starts the main program, the led shuts off.
This led, as usual in Arduino boards is connected to I/O pin 13, so it is software controllable.

Also the only way to power up the board is either trough the USB port, LiPo battery or 5V to an input pin. No other voltages, like RAW voltages above 5V are supported.

As a final note, the board that I’ve bought also came with an uFL adapter cable for SMA, an antenna and a link for accessing documentation, so, excluding the LiPo battery, the complete kit.

Starting up using the board:

I’m testing the board to send data to the Things Network and doing so by using PlatformioIO as the developing IDE. Platformio IDE is much better than the Arduino IDE, since each project has it’s own depending libraries directory .piolibdeps which we can modify and edit the library code without breaking other projects.

The platformio.ini board definition for the Lora32u4II board is just a clone of Adafruit feather 32u4:

[env:feather32u4]
platform = atmelavr
board = feather32u4
framework = arduino

As the code to send data to the TTN network, I’ve just used ABP lorawan device connection that I’ve used on my previous hand build node.

I’m testing the node with both the IBM LMIC Library (ID: 852) and the Arduino LMIC Library (ID: 1729).

After setting the correct keys and device ID, all we need is to change the LMIC pins configuration for this board: LoRa32u4II pinout diagram

According to documentation the pins are:

  1. nss (SS – Chip Select): Pin 8
  2. rst (Reset): Pin 4
  3. DIO (Lora TX/RX indicator): Pin 7

So the Lmic Pins configuration is:

const lmic_pinmap lmic_pins = {
    .nss = 8,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = 4,
    .dio = {7, 6 , LMIC_UNUSED_PIN}
};

Regarding Pin 6, is the chosen pin to connect to the DIO1 pin. This pin signals receive timeouts generated by the radio module. The connection of this pin is required for LMIC and for the onEvent() function signaling of EV_TXCOMPLETE to be triggered/fired, otherwise the onEvent() funciton is never called.
Since this is a LoraWan Class A node, after the transmission, two receive windows are opened for any downlink data that might be sent to the node. The DIO1 pin signals the receive timeout, and at the end of the receive windows, triggers the EV_TXCOMPLETE event. I’ve tried to use other pins, for example, pin 3, but then the EV_TXCOMPLETE event was never fired… Strange. Anyway, with the above configuration and with DIO1 connected through a wire bridge to pin 6 works fine.

If we do not connect DIO1 by removing the DIO1 pin configuration:

 .dio = {7, LMIC_UNUSED_PIN , LMIC_UNUSED_PIN}

with the platformio IBM Lmic library (Id: 852), or with the Arduino LMIC Library the LMIC fails. An example:

pio device monitor --port /dev/ttyACM0 --baud 115200
[cortex@brightlight:TTN32u4ABP]$ pio device monitor --port /dev/ttyACM0 --baud 115200
--- Miniterm on /dev/ttyACM0  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Starting...
FAILURE
.piolibdeps/IBM LMIC framework_ID852/src/hal/hal.cpp:24

The line hal.cpp:24 point to an ASSERT that doesn’t allow a LMIC_UNUSED_PIN for DIO1.

Putting pin 6 and making sure that it is connected to DI1 is required. Otherwise if the pin is defined but not connected we have the following behaviour:

--- Miniterm on /dev/ttyACM0  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Starting...
Sending uplink packet...

As we can see the EV_TXCOMPLETE event is never fired, and the associated reschedule of another transmission never happens, since that code is inside the code for the EV_TXCOMPLETE event. The only way, in this case, is to reset the board so another transmission happens.

So if using the above LMIC pins configuration and connecting DIO1 to pin 6, sending data to the The Things Network works just fine:

Data received at the TTN side

Some final notes, tips and tricks:

The ATMega 32U4 USB Serial port:
The ATMega 32U4 USB serial port is a bit fiddly when using it from the Arduino framework. At reset or connection first the USB port is used by the bootloader (white led fading in and out). After a while the board starts to execute the flash program (white led off), but it resets the USB port. The host computer might have an issue with this and fails to assign an USB address.

The solution is just to add at the start of the setup function a delay:

void setup() {
  delay(2500);   // Give time to the ATMega32u4 port to wake up and be recognized by the OS.
  
  Serial.begin(115200);
...
...

Using minicom instead of PlatformIO serial monitor:
This one is quite simple to explain, since minicom survives to the USB port resets since they appear and disappear through the board reset.
Against it, is that we need to explicitly exit minicom to be able to program the board.

# minicom -D /dev/ttyACM0 -b 115200

The PlatformIO Arduino LMIC library is outdated:
This is solved now. Lib 852 is now updated.
The Arduino LMIC version (1729) on the PlatformIO is outdated, since, for example doesn’t have neither the LMIC_UNUSED_PIN definition and the LMIC_setClockError function needed for a successful OTAA TTN network join.

The solution is just clone the Arduino LMIC library and copy the src folder to .piolibdeps/IBM LMIC framework_ID852/ removing the original src folder version.

Comparing Library sizes:

Using the IBM LMIC Library (ID:852) with PINGS and BEACONS disabled on the config.h file, otherwise it doesn’t fit on the 32u4 32K flash space, our sketch uses the following space:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   26040 bytes (79.5% Full)
(.text + .data + .bootloader)

Data:       1014 bytes (39.6% Full)
(.data + .bss + .noinit)

Using the Arduino LMIC library (ID: 1729) with PINGS and BEACONS enabled, but a more efficient AES implementation, we get:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   22776 bytes (69.5% Full)
(.text + .data + .bootloader)

Data:        954 bytes (37.3% Full)
(.data + .bss + .noinit)

With PINGS and BEACONS disabled we get:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   19032 bytes (58.1% Full)
(.text + .data + .bootloader)

Data:        903 bytes (35.3% Full)
(.data + .bss + .noinit)

So we get, with this last change, and while keeping support for OTTA, at least 8K/9K for program space not related to the Lorawan/TTN code support.

Some thoughts on Lorawan

It cropped up on my news feed here on WordPress an article regarding three major flaws on the Lorawan protocol. Since I’m following the Lorawan topic I was interested to see what those “Major” flaws where, and found it rather interesting, that the text, also derived from a commercial vendor, looked like the conundrum story like if a glass is half full or half empty.

First there is an huge misconception regarding low power communications protocols (LPWAN), being either Lorawan, Sigfox or anything else. The key is power, and lower power usage, the less the better. Asking for LPWAN protocols to do the same that other high energy, higher bandwidth protocols can do, is mixing apples with oranges.

Anyway the “flaws”, if we can call then that, are as follow:

– All uplink messages are unacknowledged.
This not true. Lorawan supports three classes of devices. Class A, the less power hungry of them, opens two receive windows for downlink messages from the server, that can be used for acknowledging the uplink message. If it makes sense to have acknowledge, it depends on the business case… If it is required probably LoraWan neither Sigfox or other LPWAN protocols are adequate to be used…

Also any protocols that use the ISM radio bands must obey the defined rules by the government bodies that control the radio spectrum. These rules apply to any protocol on those bands, so it’s not a flaw specific to any protocol. The 1% duty cycle applies to Lorawan and SigFox. In fact Sigfox will enforce the 1% duty cycle by refusing messages that exceed that percentage, and the LoraWan backend provider The Things Network will do the same.

Also there is some confusion regarding the medium access protocol, in this case the radio spectrum. The medium is shared by anyone, so collisions and interference will happen. Sigfox adds some resilience to this this by transmitting each message three times in three different ISM band frequencies, for example. As Lorawan it also doesn’t check for the medium before transmitting but only transmits once, since the physical modulation CSS (Chirp Spread Spectrum) has more resilience to interference. Also due to the availability of what is called Spread Factor, several transmissions can happen at the same time at the same frequency, and successfully be decoded at the other end.

On radio protocols checking for medium occupancy before transmitting only makes sense for non constrained devices, since the process of checking the medium before transmitting will consume power (a lot by having the radio on) and without any sort of guarantee that interference will not start right away or the interferance is happening not at the node side, but on the gateway side. So since one of the engineering requirements for LPWAN is low power, then the exchange between power and medium access control is made, which means ALOHA and let’s transmit!. So now we can have devices that have batteries that last years.

– All gateways in range see all uplink traffic which is not safe
I find this one rather funny since, turning on my radio can catch any available radio broadcasters, or any radio scanner can receive anything. Just check out Web SDR for hours of amusement.

The fact that all gateways see all traffic is a direct consequence of the radio medium, not an issue with the protocol. Any protocol that uses radio has the same “flaw” and it applies to Lorawan, SigFox, UNB, Weightless, 3G, LTE, you name it.

To solve this, encryption is used and at least in Lorawan there are several encryption keys and ways of providing them.
Lorawan can use fixed provided keys (ABP – Activation by personalisation) or variable keys through OTAA (over the air activation).

Anyway the gateways can receive any Lorawan packets, but without at least the 128 bit Network key and 128 bit application key, can’t do anything with the data. Gateways only forward data to network backend servers, and there, if they have the correct keys, decryption can be done and data forward to the correct application servers.

Check out this for more information.

– LoRaWAN requires an enormous amount of bandwidth
Well, yes, it is a SPREAD spectrum technology and it makes part of the holy war between Narrow band supporters vs Wide band supporters. Ones say that UNB is better, others don’t, and so on. Spread spectrum technology exists since a long time ago. Lorawan bandwidth can be 150Khz, 250Khz and 500Khz vs the 200Hz of Sigfox.

While SST can be used and detected below the noise floor level and accepts variations on the frequencies (reflections, Doppler effects), UNB is, on the receiver side way more complicated since it requires very precise crystals for frequency reference and higher power levels on the spectrum.

So in the article that I read it seems that Lorawan SST is just bad, without any consideration of the advantages vs UNB, which by itself is a discussion on different technologies which have advantages and disadvantages each.

Conclusion:
Nothing is perfect in engineering. Trade offs need to be made to achieve the requested requirements, and then based on the initial implementations, improve on it. So Lorawan, as Sigfox, solve the same issues by different means with the associated advantages and disadvantages.

So the above flaws, are just engineering trade offs that can be applied to any protocol.

LPWAN – Starting up with LoraWAN and The Things Network

LPWAN Networks – A simple introduction
Low Power Wide Area Communications (LPWAN) classifies a group of communication protocols featuring low power usage and a high communication range. For Internet of Things communications, where battery powered devices and constrained devices (weak CPU, low RAM/ROM) are the norm, LPWAN use as the communication protocol for IoT makes sense, since it makes possible to have standalone devices with batteries that last years, instead of hours or days, and might be hundreds of meters to Kms away from a base station/gateway.

But LPWAN protocols, in contrast have low communication bandwidth, since from power, range and bandwidth, we can only have two of those, and while this might be a problem for certain applications, IoT devices don’t require high bandwidth and since most have sparse communication requirements, and when they do communicate they don’t need to transmit a lot of data.

Starting up with LoraWan and The Things Network
One of the possible ways of starting to use LPWAN networks in our IoT devices, is to use a LPWAN protocol named LoraWan, which is based on the Lora physical protocol, and if available at our area, the crowd sourced LPWAN network The Things Network as the backend network for receiving data.

Lora uses the 868Mhz ISM radio band and it uses a form of signal transmission called CSS. The ISM radio band can be used freely by everybody, but under certain rules that depends of the country where the devices are located. So to use that band, rules like maximum emission power and duty cycle (usage of the available spectrum) are defined, and in Europe, the maximum power is 20mW and 1% duty cycle per day. Additionally the back end operator can add other restrictions.

Over Lora, the Lorawan protocol is an open standard and source code for implementing it is available and already ported to several devices, including the Arduino, Raspberry PI and the ESP8266, among others.

The software for implementing the LoraWan protocol at the end devices (nodes) is IBM’s LMIC library, available at Github and on Platformio libs:

[ ID  ] Name             Compatibility         "Authors": Description
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[ 852 ] IBM LMIC framework arduino, atmelavr, atmelsam, espressif8266, intel_arc32, microchippic32, nordicnrf51, teensy, timsp430 "IBM, Matthijs Kooijman": Arduino port of the LMIC (LoraWAN-in-C, formerly LoraMAC-in-C) framework provided by IBM. | @1.5.0+arduino-1

Based on this library we can build code and nodes that are able to transmit data through LoraWan to a network backend. Specifically for the Arduino, the Atmega328 is the bare minimum to begin with, since the library, due to use of AES encryption functions occupies a lot of memory.

The backend can be provided by really anyone, including Telco operators, or private and crowd source operators like The Things Network (TTN). TTN provides the backend and management services, but depends on crowd sourced gateways, and might not be available at our area. Still it is possible, for testing, to build our own gateways, our buy them, and connect them to the Things Network. TTN doesn’t require any access fees (yet).

So with LoraWan an the Things Network, we can build our own nodes and gateways since code and hardware can be easily obtained, connect them and use it to build applications.

Regarding LoraWan we can also read this great introduction from Design Spark.

Lora hardware:

Anyway the easiest way for starting up using Lora and Lorawan, is to buy an Dragino Lora Shield and connect it to an Arduino Uno or Mega.

Dragino Lora Shield

Dragino Lora Shield

This is a great shield to startup since doesn’t need any soldering or complex configuration, just plug it on an Arduino shield and use the LMIC library and some code samples. Also the Dragino Shield can be used to build a simple gateway by connecting it to a Raspberry PI or ESP8266 to build what is called a single channel gateway, that allows to test gateway functionality, but it isn’t quite compatible with the LoraWan specifications. Anyway it gets the job done if there is no gateway nearby. Just make sure that you by version 1.3 or 1.4 of the shield. Mine also came with an SMA antenna.

Other alternatives to start using Lorawan are available at eBay/Aliexpress and other renowned name shops, namely Lora radio transceivers at around 8/16€, for example the HopeRF RFM95. Those we can also use them with Arduino or ESP8266 to build our nodes or single channel gateways.

Just make sure that the frequency for these modules and shields must match the allowed radio transmission frequency in your area. In my case, in Europe is 868Mhz, but for example at USA is 900Mhz.

Dragino Lora Shield Jumpers
The Shield has some jumpers and only looking at the schematic and cross referenced them with the RFM95 module (used by the shield as the Lora transceiver) I could see what they where for:

– There are two set of jumpers:

  • One defines the pins for SPI communication: SV# Jumpers;
  • The other set defines which data pins of the RFM95 module are exposed to the shield allowing them to be connected to the Arduino: JP# Jumpers.

The JP# jumpers connect the following if closed:

  • JP1 – RFM95 DI1 – Arduino pin D6
  • JP2 – RFM95 DI2 – Arduino pin D7
  • JP3 – RFM95 DI5 – Arduino pin D8

The RFM95 pin DI0 is permanently connected to Arduino pin D2.
The RFM95 pin RST (Reset) is permanently connected to Arduino pin D9.

Their function for Lora might be the following (The function depends how the RFM95 module is used)

  • RFM95 DI0 – Indicates if data was received and is ready to be read through the SPI bus. Indicates end of transmission for data that was previously sent: RXReady/TXReady
  • RFM95 DI1 – Receive timeout for Lora mode.
  • RFM95 DI2 – Receive timeout for FSK mode.
  • RFM95 DI5 – Used by Semtech Library. LMIC library doesn’t use it.

The SV# jumpers connect:

  • SV2 – SPI Clock line. Default on pin D13, otherwise on Arduino SPI CLK on the ICSP header.
  • SV3 – SPI Data line In (MOSI). Default on pin D11, otherwise on Arduino MOSI pin on the ICSP header.
  • SV4 – SPI Data line Out (MISO). Default on pin D12, otherwise on Arduino MISO pin on the ICSP header.

The SPI Chip Select line is always at pin D10.

So now we know that D10, D9 and D2 are used permanently connected and used by the shield, and the others can be connected or disconnected if needed or not.

LMIC software with Dragino Lora Shield:

To start using the Dragino Lora Shield so it connects to a LPWAN network, we can start using the following example: TTN node by ABP. ABP means Activation by Personalization, which means that all data for joining the network will be explicitly defined on the code. Other alternative would be OTAA: Over the air activation, where the gateway sends the configuration data to the node. Since we don’t know if we have a gateway in range, let’s start first with ABP mode.

The above code uses the LMIC library for implementing the LoraWan stack.
Since LMIC library doesn’t use DI05, we can remove the JP3 jumper, and free this IO line for other things, like another shield.

To use the LMIC library we must define first the pins that are used according to the shield configuration:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, 7}             //DIO pin mapping for DIO0, DIO1 and DIO2  
};

But DIO2 pin is only used for FSK modulation, and so if only using LoraWan we can also open up the JP2 jumper, and our LMIC pin configuration can be as follows:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, LMIC_UNUSED_PIN} //DIO pin mapping for DIO0, DIO1. DIO2 is not used  
};

Connecting to TTN
Connecting to The Things Network (TTN) depends of course of an available TTN gateway at the nodes range. Still we need to configure some parameters to allow the node to connect.

On this example the connection is done through Activation by Personalization. This means that we should put on our code the Network Session key and Application Session key on the code. To obtain these values we need to register on the TTN site, TTN Dashboard and add an ABP device..

From this site, then we can get the three configuration parameters that we need:

  • Device ID
  • Network Session key
  • Application Session key

Note by pressing the icon we can get the values in a form ready to paste it on the code:

/*************************************
 * TODO: Change the following keys
 * NwkSKey: network session key, AppSKey: application session key, and DevAddr: end-device address
 *************************************/
static u1_t NWKSKEY[16] = { 0xAA, 0x0F, 0x29, 0xD3, 0x9D, 0x7A, 0xAE, 0x3B, 0x54, 0xCF, 0xDF, 0x2F, 0x2A, 0x23, 0x55, 0xB5 };
static u1_t APPSKEY[16] = { 0x2F, 0x85, 0x43, 0x5B, 0x34, 0x9C, 0x80, 0xC6, 0xA8, 0xFA, 0x27, 0x49, 0x5A, 0x36, 0x82, 0x21 };
static u4_t DEVADDR = 0x95337738;

And that’s it, when running the sketch, and if in range of a gateway, the sent message should appear at the Dashboard:

Messages received for the device

Messages received for the device

Final thoughts:
In my area there are at least, supposedly, two active TTN gateways and I’m around 2Km’s away from them in a dense urban area.
But when running the sketch the first times, I had no results what so ever.

One of the configurations options for LoraWan is what is called a Spread Factor that, in a simplest way, exchanges range with on-air time for transmission, being SF7 the fastest speed and shorter range and SF12 the slowest speed and higher range. The sketch default value for the sprea factor was SF7, and changing it to SF12:

 ...
    // TTN uses SF9 for its RX2 window.
    LMIC.dn2Dr = DR_SF9;

    // Set data rate and transmit power for uplink (note: txpow seems to be ignored by the library)
    LMIC_setDrTxpow( DR_SF12 ,14);

    // Start job
    do_send(&sendjob);

With SF12 the message started be received and the sequence numbers of the messages received where continuous, so no lost messages.

Dropping to SF11, also worked fine, and the message sequence received and shown on the TTN Dashboard where still continuous.

At SF10, some of the messages where lost, almost 75% of them. In this case, arranging the antenna position (to exactly vertical) and placement did alter the reception successes.

Processing received data
After the data is at the TTN backend there are several ways of getting it. For reference on the TTN site there are instructions in how to access the data.

Crash course on Lora
For further information the next one hour video is a great starting point:

The slides for the presentation are available here:

https://www.dropbox.com/s/87htha4dq5b32wa/ttn-lora-crash-course.pdf?dl=0