The ESP32 Oled Lora TTGO LoRa32 board and connecting it to TTN

The TTGO LoRa32 board is an ESP32 based board that has both an Oled and a Lora transceiver, in my case, the SX1276 transceiver for the 868Mhz band. So it is very similar to some of the ESP32 Oled boards available on the Internet. The board looks like this:

And the interesting part of this board is the new Wifi antenna located in the back that is made of bend metal:

The board also has a LiPo connector, and probably a charger circuit, since I haven’t tried it yet, a user controlled blue led, and a very dim red power led. The led is so dim that at first I thought the board was broken/short circuited, but it is normal.
The Lora Antenna is connected by U.FL/IPEX connector. Both a U.FL to SMA adapter cable is provided and also a cable to connect to the LiPo connector.

An important information to use this board for the LMIC LoraWan based communication is the location of the Lora transceiver DI01 and DIO2 pins. Fortunately they are exposed and connected internally to the ESP32 processor GPIO33 and GPIO32 pins respectively. I’ve updated the pin out for this board:

TTGO LoRa32 Pinout

So the LMIC definition pins for using this board are:

const lmic_pinmap lmic_pins = {
    .nss = 18,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = 14,
    .dio = {26, 33, 32}  // Pins for the Heltec ESP32 Lora board/ TTGO Lora32 with 3D metal antenna
};

The Blue Led Pin is at Pin 2, and according to the sample code the Oled Display is at I2C address 0x3C. The I2C bus where the OLed is at SDA pin 4 and SCLK pin 15.

Also it seems there are at least two revisions for the ESP32 silicon, Revision 0 (Zero) for the initial one, and the latest, at the current date, Revision one.

By executing the Andreas Spiess revision check code it seems that my board is using the latest revision:

REG_READ(EFUSE_BLK0_RDATA3_REG) 1000000000000000
EFUSE_RD_CHIP_VER_RESERVE_S 1100
EFUSE_RD_CHIP_VER_RESERVE_V 111

Chip Revision (official version): 1
Chip Revision from shift Operation 1

Programming the board:
The board can be programmed easily with Platformio IDE by selecting as the target board the Heltec Wifi Lora board. Probably both boards are identical.

The platformio.ini file is as follows:

[env:heltec_wifi_lora_32]
platform = espressif32
board = heltec_wifi_lora_32
framework = arduino

For supporting the OLed and the Lora transceiver we also need to install the ESP8266_SSD1306 lib (ID: 562) and the IBM LMIC library (ID: 852) by either manually installing them on the project root or by adding the following line to the platformio.ini file:

[env:heltec_wifi_lora_32]
platform = espressif32
board = heltec_wifi_lora_32
framework = arduino
lib_install= 852, 562

With this, the sample TTN INO sketchs for connecting either through ABP or OTAA work flawlessly without any issue by using the above LMIC pins configuration.

The sample sketch for the board: Connecting to TTN and display the packet RSSI:
Since we have the OLed, we can use the RX window to display the received RSSI of our messages on the gateway. This only works if the downlink messages from the gateway can reach back our node, so it might not work always. In my case, I’m about 3Km from the gateway in dense urban area, and not always I can display the packet RSSI.

How this works? Simple, just send our packet, and on the backend we send back the received RSSI as downlink message by using Node-Red, the TTN nodes, and some code:

Since our packet can be received by several gateways, we iterate over the TTN message and calculate the better RSSI and SNR:

// Build an object that allows us to track
// node data better than just having the payload

//For the debug inject node. Comment out when in real use
//var inmsg = msg.payload;
var inmsg = msg;  // from the TTN node

var newmsg = {};
var devicedata = {};
var betterRSSI = -1000;  // Start with a low impossible value
var betterSNR = -1000;

// WARNING only works with String data
// Use TTN decode functions is a better idea
var nodercvdata = inmsg.payload.toString("utf-8");

devicedata.device = inmsg.dev_id;
devicedata.deviceserial = inmsg.hardware_serial;
devicedata.rcvtime = inmsg.metadata.time;
devicedata.nodedata = nodercvdata;

// Iterate over the gateway data to get the best RSSI and SNR data
var gws = inmsg.metadata.gateways;

for ( var i = 0 ; i  betterRSSI )
        betterRSSI = gw.rssi;
        
    if ( gw.snr > betterSNR )
        betterSNR = gw.snr;
}

devicedata.rssi = betterRSSI;
devicedata.snr = betterSNR;

newmsg.payload = devicedata;

return newmsg;

We build then the response object and send it back to the TTN servers that send it to our node. The received data is then displayed on the Oled.

The Node-Red code is as follows:

[{"id":"d4536a72.6e6d7","type":"ttn message","z":"66b897a.7ab5c68","name":"TTN APP Uplink","app":"b59d5696.cde318","dev_id":"","field":"","x":140,"y":220,"wires":[["facbde95.14894"]]},{"id":"facbde95.14894","type":"function","z":"66b897a.7ab5c68","name":"Calculate better RSSI","func":"// Build an object that allows us to track\n// node data better than just having the payload\n\n//For the debug inject node. Comment out when in real use\n//var inmsg = msg.payload;\nvar inmsg = msg;  // from the TTN node\n\nvar newmsg = {};\nvar devicedata = {};\nvar betterRSSI = -1000;  // Start with a low impossible value\nvar betterSNR = -1000;\n\n// WARNING only works with String data\n// Use TTN decode functions is a better idea\nvar nodercvdata = inmsg.payload.toString(\"utf-8\");\n\ndevicedata.device = inmsg.dev_id;\ndevicedata.deviceserial = inmsg.hardware_serial;\ndevicedata.rcvtime = inmsg.metadata.time;\ndevicedata.nodedata = nodercvdata;\n\n// Iterate over the gateway data to get the best RSSI and SNR data\nvar gws = inmsg.metadata.gateways;\n\nfor ( var i = 0 ; i  betterRSSI )\n        betterRSSI = gw.rssi;\n        \n    if ( gw.snr > betterSNR )\n        betterSNR = gw.snr;\n}\n\ndevicedata.rssi = betterRSSI;\ndevicedata.snr = betterSNR;\n\nnewmsg.payload = devicedata;\n\nreturn newmsg;","outputs":1,"noerr":0,"x":400,"y":220,"wires":[["1ac970ec.4cfabf","94515e56.904228"]]},{"id":"1ac970ec.4cfabf","type":"debug","z":"66b897a.7ab5c68","name":"","active":false,"console":"false","complete":"payload","x":670,"y":260,"wires":[]},{"id":"2bea15d8.18f88a","type":"ttn send","z":"66b897a.7ab5c68","name":"TTN APP Downlink","app":"b59d5696.cde318","dev_id":"","port":"","x":970,"y":100,"wires":[]},{"id":"94515e56.904228","type":"function","z":"66b897a.7ab5c68","name":"set Payload","func":"msg.dev_id  = msg.payload.device;\nmsg.payload = Buffer.from(\"RSSI: \" + msg.payload.rssi);\n\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":100,"wires":[["2bea15d8.18f88a","cd04abb9.ccd278"]]},{"id":"cd04abb9.ccd278","type":"debug","z":"66b897a.7ab5c68","name":"","active":true,"console":"false","complete":"true","x":930,"y":200,"wires":[]},{"id":"b59d5696.cde318","type":"ttn app","z":"","appId":"TTNAPPLICATIONID","region":"eu","accessKey":"ttn-account-v2.CHANGEMECHANGEME"}]

Just make sure that we have the TTN nodes installed, and change the credentials for your TTN Application.

On the TTGO ESP32 Lora32 board we just modify the event handling code to display the downlink message:

void onEvent (ev_t ev) {
    if (ev == EV_TXCOMPLETE) {
        display.clear();
        display.drawString (0, 0, "EV_TXCOMPLETE event!");


        Serial.println(F("EV_TXCOMPLETE (includes waiting for RX windows)"));
        if (LMIC.txrxFlags & TXRX_ACK) {
          Serial.println(F("Received ack"));
          display.drawString (0, 20, "Received ACK.");
        }

        if (LMIC.dataLen) {
          int i = 0;
          // data received in rx slot after tx
          Serial.print(F("Data Received: "));
          Serial.write(LMIC.frame+LMIC.dataBeg, LMIC.dataLen);
          Serial.println();

          display.drawString (0, 20, "Received DATA.");
          for ( i = 0 ; i < LMIC.dataLen ; i++ )
            TTN_response[i] = LMIC.frame[LMIC.dataBeg+i];
          TTN_response[i] = 0;
          display.drawString (0, 32, String(TTN_response));
        }

        // Schedule next transmission
        os_setTimedCallback(&sendjob, os_getTime()+sec2osticks(TX_INTERVAL), do_send);
        digitalWrite(LEDPIN, LOW);
        display.drawString (0, 50, String (counter));
        display.display ();
    }
}

For example we can now see on the serial port monitor:

EV_TXCOMPLETE (includes waiting for RX windows)
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Data Received: RSSI: -118
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Data Received: RSSI: -114
Sending uplink packet...
EV_TXCOMPLETE (includes waiting for RX windows)
Data Received: RSSI: -105

Thats it!

Some final notes:
Probably not related to the board, but when connecting it to an USB3 port, the Linux Operating system was unable to configure a device for the board. Connecting it to an USB2 port worked flawlessly:

usb 2-1: new full-speed USB device number 2 using xhci_hcd
usb 2-1: string descriptor 0 read error: -71
usb 2-1: can't set config #1, error -71      

As additional information the serial chip on this board is an umarked CP210x chip:

usb 4-1.3: new full-speed USB device number 6 using ehci-pci
cp210x 4-1.3:1.0: cp210x converter detected
usb 4-1.3: cp210x converter now attached to ttyUSB0

lsusb:

Bus 004 Device 006: ID 10c4:ea60 Cygnal Integrated Products, Inc. CP2102/CP2109 UART Bridge Controller [CP210x family]

I haven’t yet tried the WiFi and checked if the metal antenna is any good.

TTN ABP Sketch:

The Sketch that I’m using, for your reference is as follows:

// MIT License
// https://github.com/gonzalocasas/arduino-uno-dragino-lorawan/blob/master/LICENSE
// Based on examples from https://github.com/matthijskooijman/arduino-lmic
// Copyright (c) 2015 Thomas Telkamp and Matthijs Kooijman

#include 
#include "lmic.h"
#include 
#include 
#include 
#include "soc/efuse_reg.h"

#define LEDPIN 2

#define OLED_I2C_ADDR 0x3C
#define OLED_RESET 16
#define OLED_SDA 4
#define OLED_SCL 15

unsigned int counter = 0;

SSD1306 display (OLED_I2C_ADDR, OLED_SDA, OLED_SCL);

/*************************************
 * TODO: Change the following keys
 * NwkSKey: network session key, AppSKey: application session key, and DevAddr: end-device address
 *************************************/
static u1_t NWKSKEY[16] = { .... };  // MSB form

static u1_t APPSKEY[16] = { .... };  // MSB form

static u4_t DEVADDR = 0x...D;   // Put here the device id in hexadecimal form.

void os_getArtEui (u1_t* buf) { }
void os_getDevEui (u1_t* buf) { }
void os_getDevKey (u1_t* buf) { }

static osjob_t sendjob;

// Schedule TX every this many seconds (might become longer due to duty
// cycle limitations).
const unsigned TX_INTERVAL = 60;
char TTN_response[30];

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 18,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = 14,
    .dio = {26, 33, 32}  // Pins for the Heltec ESP32 Lora board/ TTGO Lora32 with 3D metal antenna
};

void do_send(osjob_t* j){
    // Payload to send (uplink)
    static uint8_t message[] = "Hello World!";

    // Check if there is not a current TX/RX job running
    if (LMIC.opmode & OP_TXRXPEND) {
        Serial.println(F("OP_TXRXPEND, not sending"));
    } else {
        // Prepare upstream data transmission at the next possible time.
        LMIC_setTxData2(1, message, sizeof(message)-1, 0);
        Serial.println(F("Sending uplink packet..."));
        digitalWrite(LEDPIN, HIGH);
        display.clear();
        display.drawString (0, 0, "Sending uplink packet...");
        display.drawString (0, 50, String (++counter));
        display.display ();
    }
    // Next TX is scheduled after TX_COMPLETE event.
}

void onEvent (ev_t ev) {
    if (ev == EV_TXCOMPLETE) {
        display.clear();
        display.drawString (0, 0, "EV_TXCOMPLETE event!");


        Serial.println(F("EV_TXCOMPLETE (includes waiting for RX windows)"));
        if (LMIC.txrxFlags & TXRX_ACK) {
          Serial.println(F("Received ack"));
          display.drawString (0, 20, "Received ACK.");
        }

        if (LMIC.dataLen) {
          int i = 0;
          // data received in rx slot after tx
          Serial.print(F("Data Received: "));
          Serial.write(LMIC.frame+LMIC.dataBeg, LMIC.dataLen);
          Serial.println();

          display.drawString (0, 20, "Received DATA.");
          for ( i = 0 ; i > (EFUSE_RD_CHIP_VER_RESERVE_S)&&EFUSE_RD_CHIP_VER_RESERVE_V) ;
}

void printESPRevision() {
  Serial.print("REG_READ(EFUSE_BLK0_RDATA3_REG) ");
  Serial.println(REG_READ(EFUSE_BLK0_RDATA3_REG), BIN);

  Serial.print("EFUSE_RD_CHIP_VER_RESERVE_S ");
  Serial.println(EFUSE_RD_CHIP_VER_RESERVE_S, BIN);

  Serial.print("EFUSE_RD_CHIP_VER_RESERVE_V ");
  Serial.println(EFUSE_RD_CHIP_VER_RESERVE_V, BIN);

  Serial.println();

  Serial.print("Chip Revision (official version): ");
  Serial.println(getChipRevision());

  Serial.print("Chip Revision from shift Operation ");
  Serial.println(REG_READ(EFUSE_BLK0_RDATA3_REG) >> 15, BIN);

}

void setup() {
    Serial.begin(115200);
    delay(1500);   // Give time for the seral monitor to start up
    Serial.println(F("Starting..."));

    printESPRevision();

    // Use the Blue pin to signal transmission.
    pinMode(LEDPIN,OUTPUT);

   // reset the OLED
   pinMode(OLED_RESET,OUTPUT);
   digitalWrite(OLED_RESET, LOW);
   delay(50);
   digitalWrite(OLED_RESET, HIGH);

   display.init ();
   display.flipScreenVertically ();
   display.setFont (ArialMT_Plain_10);

   display.setTextAlignment (TEXT_ALIGN_LEFT);

   display.drawString (0, 0, "Init!");
   display.display ();

    // LMIC init
    os_init();

    // Reset the MAC state. Session and pending data transfers will be discarded.
    LMIC_reset();

    // Set up the channels used by the Things Network, which corresponds
    // to the defaults of most gateways. Without this, only three base
    // channels from the LoRaWAN specification are used, which certainly
    // works, so it is good for debugging, but can overload those
    // frequencies, so be sure to configure the full frequency range of
    // your network here (unless your network autoconfigures them).
    // Setting up channels should happen after LMIC_setSession, as that
    // configures the minimal channel set.
    LMIC_setupChannel(0, 868100000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(1, 868300000, DR_RANGE_MAP(DR_SF12, DR_SF7B), BAND_CENTI);      // g-band
    LMIC_setupChannel(2, 868500000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(3, 867100000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(4, 867300000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(5, 867500000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(6, 867700000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(7, 867900000, DR_RANGE_MAP(DR_SF12, DR_SF7),  BAND_CENTI);      // g-band
    LMIC_setupChannel(8, 868800000, DR_RANGE_MAP(DR_FSK,  DR_FSK),  BAND_MILLI);      // g2-band
    // TTN defines an additional channel at 869.525Mhz using SF9 for class B
    // devices' ping slots. LMIC does not have an easy way to define set this
    // frequency and support for class B is spotty and untested, so this
    // frequency is not configured here.

    // Set static session parameters.
    LMIC_setSession (0x1, DEVADDR, NWKSKEY, APPSKEY);

    // Disable link check validation
    LMIC_setLinkCheckMode(0);

    // TTN uses SF9 for its RX2 window.
    LMIC.dn2Dr = DR_SF9;

    // Set data rate and transmit power for uplink (note: txpow seems to be ignored by the library)
    //LMIC_setDrTxpow(DR_SF11,14);
    LMIC_setDrTxpow(DR_SF9,14);

    // Start job
    do_send(&sendjob);
}

void loop() {
    os_runloop_once();
}


Advertisements

Using the BSFrance Lora32U4 board to connect to the Things Network Lorawan

The BSFrance Lora32u4 II (Lora32U4II for helping Google out) board is an Atmega32U4 processor with a HDP13 Lora transceiver on the same board. As far as I’m aware, the HDP13 is similar to the RFM95W (including pinout), and in my case it seems it has an original Semtech SX1276 (868Mhz radio transceiver) chip installed on the HDP13 module. This board is similar to the Adafruit 32U4 Lora feather, if not equal… (possible schematics for the Lora32u4 board)

The board hardware includes beside the Lora HDP13 module a LiPo connector with an 2 pin JST PH 2.0mm pin spacing connector and the power supporting electronics.
There are two leds, on orange for LiPo and charger status, that blinks very fast when no LiPo is connected, and a very bright white led that fades in and out when the bootloader is in the programming mode or programming is ongoing. After the bootloader exists and starts the main program, the led shuts off.
This led, as usual in Arduino boards is connected to I/O pin 13, so it is software controllable.

Also the only way to power up the board is either trough the USB port, LiPo battery or 5V to an input pin. No other voltages, like RAW voltages above 5V are supported.

As a final note, the board that I’ve bought also came with an uFL adapter cable for SMA, an antenna and a link for accessing documentation, so, excluding the LiPo battery, the complete kit.

Starting up using the board:

I’m testing the board to send data to the Things Network and doing so by using PlatformioIO as the developing IDE. Platformio IDE is much better than the Arduino IDE, since each project has it’s own depending libraries directory .piolibdeps which we can modify and edit the library code without breaking other projects.

The platformio.ini board definition for the Lora32u4II board is just a clone of Adafruit feather 32u4:

[env:feather32u4]
platform = atmelavr
board = feather32u4
framework = arduino

As the code to send data to the TTN network, I’ve just used ABP lorawan device connection that I’ve used on my previous hand build node.

I’m testing the node with both the IBM LMIC Library (ID: 852) and the Arduino LMIC Library (ID: 1729).

After setting the correct keys and device ID, all we need is to change the LMIC pins configuration for this board: LoRa32u4II pinout diagram

According to documentation the pins are:

  1. nss (SS – Chip Select): Pin 8
  2. rst (Reset): Pin 4
  3. DIO (Lora TX/RX indicator): Pin 7

So the Lmic Pins configuration is:

const lmic_pinmap lmic_pins = {
    .nss = 8,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = 4,
    .dio = {7, 6 , LMIC_UNUSED_PIN}
};

Regarding Pin 6, is the chosen pin to connect to the DIO1 pin. This pin signals receive timeouts generated by the radio module. The connection of this pin is required for LMIC and for the onEvent() function signaling of EV_TXCOMPLETE to be triggered/fired, otherwise the onEvent() funciton is never called.
Since this is a LoraWan Class A node, after the transmission, two receive windows are opened for any downlink data that might be sent to the node. The DIO1 pin signals the receive timeout, and at the end of the receive windows, triggers the EV_TXCOMPLETE event. I’ve tried to use other pins, for example, pin 3, but then the EV_TXCOMPLETE event was never fired… Strange. Anyway, with the above configuration and with DIO1 connected through a wire bridge to pin 6 works fine.

If we do not connect DIO1 by removing the DIO1 pin configuration:

 .dio = {7, LMIC_UNUSED_PIN , LMIC_UNUSED_PIN}

with the platformio IBM Lmic library (Id: 852), or with the Arduino LMIC Library the LMIC fails. An example:

pio device monitor --port /dev/ttyACM0 --baud 115200
[cortex@brightlight:TTN32u4ABP]$ pio device monitor --port /dev/ttyACM0 --baud 115200
--- Miniterm on /dev/ttyACM0  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Starting...
FAILURE
.piolibdeps/IBM LMIC framework_ID852/src/hal/hal.cpp:24

The line hal.cpp:24 point to an ASSERT that doesn’t allow a LMIC_UNUSED_PIN for DIO1.

Putting pin 6 and making sure that it is connected to DI1 is required. Otherwise if the pin is defined but not connected we have the following behaviour:

--- Miniterm on /dev/ttyACM0  115200,8,N,1 ---
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Starting...
Sending uplink packet...

As we can see the EV_TXCOMPLETE event is never fired, and the associated reschedule of another transmission never happens, since that code is inside the code for the EV_TXCOMPLETE event. The only way, in this case, is to reset the board so another transmission happens.

So if using the above LMIC pins configuration and connecting DIO1 to pin 6, sending data to the The Things Network works just fine:

Data received at the TTN side

Some final notes, tips and tricks:

The ATMega 32U4 USB Serial port:
The ATMega 32U4 USB serial port is a bit fiddly when using it from the Arduino framework. At reset or connection first the USB port is used by the bootloader (white led fading in and out). After a while the board starts to execute the flash program (white led off), but it resets the USB port. The host computer might have an issue with this and fails to assign an USB address.

The solution is just to add at the start of the setup function a delay:

void setup() {
  delay(2500);   // Give time to the ATMega32u4 port to wake up and be recognized by the OS.
  
  Serial.begin(115200);
...
...

Using minicom instead of PlatformIO serial monitor:
This one is quite simple to explain, since minicom survives to the USB port resets since they appear and disappear through the board reset.
Against it, is that we need to explicitly exit minicom to be able to program the board.

# minicom -D /dev/ttyACM0 -b 115200

The PlatformIO Arduino LMIC library is outdated:
The Arduino LMIC version (1729) on the PlatformIO is outdated, since, for example doesn’t have neither the LMIC_UNUSED_PIN definition and the LMIC_setClockError function needed for a successful OTAA TTN network join.

The solution is just clone the Arduino LMIC library and copy the src folder to .piolibdeps/IBM LMIC framework_ID852/ removing the original src folder version.

Comparing Library sizes:

Using the IBM LMIC Library (ID:852) with PINGS and BEACONS disabled on the config.h file, otherwise it doesn’t fit on the 32u4 32K flash space, our sketch uses the following space:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   26040 bytes (79.5% Full)
(.text + .data + .bootloader)

Data:       1014 bytes (39.6% Full)
(.data + .bss + .noinit)

Using the Arduino LMIC library (ID: 1729) with PINGS and BEACONS enabled, but a more efficient AES implementation, we get:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   22776 bytes (69.5% Full)
(.text + .data + .bootloader)

Data:        954 bytes (37.3% Full)
(.data + .bss + .noinit)

With PINGS and BEACONS disabled we get:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   19032 bytes (58.1% Full)
(.text + .data + .bootloader)

Data:        903 bytes (35.3% Full)
(.data + .bss + .noinit)

So we get, with this last change, and while keeping support for OTTA, at least 8K/9K for program space not related to the Lorawan/TTN code support.

Starting up with the Nordic NRF52 BLE chip

The nRF52 based chips are the latest version of the popular Bluetooth chip from Nordic that has an ARM Cortex based processor and Bluetooth communications support.
Major differences from the previous nRF51 version includes:

  1. Based on ARM Cortex M4F instead of ARM M0.
  2. Support for the latest Bluetooth 5 specification
  3. On chip NFC support for device bounding and probably something else

The following post centralizes the information that I gathered to start using the demo board that I bought based on the nRF52832 chip.

The eBay,Aliexpress nRF52832 based board:
I’ve bought my nRF52832 based board from AliExpress for around 13€. An higher price than the ESP32 which has both WifI and also blueetooth, but since I really needed to start using the nRF5X base chips I’ve bought what is called “NRF52832 Mini Development Board Gold Core board Wireless Bluetooth Transceiver Module”…

This board build is based on a two boards joined together: one daughter board holding the nRf52832 chip, and another, larger board, exposing the pins, JTAG/SWD connector, power regulator, two leds and two switches. As a bonus the main board was designed for something else and so all the pins silk screen are just plain wrong, but at least the power pins and the SWD pins are correctly identified.

For mapping out correctly the nRF pins to the out pins we need to see the board schematics vs the daughter board pins.

This board schematics are here at this link: NRF52832 Module Test Board V1.0.

And the daughter board pinout is here:

Checking the schematics vs the daughter board pin out we can see that on the pdf schematics file our nRF chip is located where would/should be a CC2640_RGZ module (!…). For example on that module the DIO0 pin corresponds to P25 pin, the DIO1 pin to P26, and so on. We also can check that by, probably sheer luck, the power pins and SWD pins TCLK-SWCLK and TDIO-SWDIO are just right… and so they just reused the main board to hold the nRF52.

Checking out the board and the schematic we can see also that we have a switch on nRF52 pin P04 and two red leds at P30 and P31. The leds can be disconnected by removing the soldering on the nearby solder bridges. The other pins seem free.

As a final note, at least the board that I’ve received, comes with the BLE peripheral Nordic UART example loaded as the running firmware.

Programming the board
The board can be programmed at least by two ways:

  1. Openocd On chip debugger – But a set of patchs are needed to support the nRF52
  2. Black Magic Probe – Running on a cheap stm32F103C8T6 board – Blue pill

Both ways allow to successfully program the board and debug the running code.

To avoid making this a very long post I’ve split it into further posts how to build the tools necessary to program the nRF52 chip.

  1. Setting up Openocd for programming the Nordic nRF52832 chip
  2. Building a Black Magic Probe using the “blue pill” STM32F103C8T6 based board

STM CubeMX and multiple definition errors

So I’m using the latest STM CubeMX program version 4.23.0 to bootstrap the code of a STM32F103 based blue pill board, using Makefiles as the Toolchain/IDE.

After generating the project and executing the make command at the project root directory, I have the following errors:

build/main.o: In function `_Error_Handler':
/opt/ARM/Projects/TTNBluePill/Src/main.c:259: multiple definition of `_Error_Handler'
build/main.o:/opt/ARM/Projects/TTNBluePill/Src/main.c:259: first defined here
build/main.o: In function `MX_GPIO_Init':
/opt/ARM/Projects/TTNBluePill/Src/main.c:240: multiple definition of `SystemClock_Config'
build/main.o:/opt/ARM/Projects/TTNBluePill/Src/main.c:240: first defined here
build/main.o: In function `MX_GPIO_Init':
/opt/ARM/Projects/TTNBluePill/Src/main.c:240: multiple definition of `main'
build/main.o:/opt/ARM/Projects/TTNBluePill/Src/main.c:240: first defined here
build/stm32f1xx_it.o: In function `NMI_Handler':
/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: multiple definition of `NMI_Handler'
build/stm32f1xx_it.o:/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: first defined here
build/stm32f1xx_it.o: In function `NMI_Handler':
/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: multiple definition of `HardFault_Handler'
build/stm32f1xx_it.o:/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: first defined here
build/stm32f1xx_it.o: In function `NMI_Handler':
/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: multiple definition of `MemManage_Handler'
build/stm32f1xx_it.o:/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: first defined here
build/stm32f1xx_it.o: In function `NMI_Handler':
/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: multiple definition of `BusFault_Handler'
build/stm32f1xx_it.o:/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: first defined here
build/stm32f1xx_it.o: In function `NMI_Handler':
/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: multiple definition of `UsageFault_Handler'
build/stm32f1xx_it.o:/opt/ARM/Projects/TTNBluePill/Src/stm32f1xx_it.c:52: first defined here
build/stm32f1xx_it.o: In function `NMI_Handler':
....
....
....

And the code linkage fails.

The issue is that the generated CubeMX Makefile has duplicated entries:

######################################
# source
######################################
# C sources
C_SOURCES =  \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_cortex.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_tim_ex.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_uart.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_spi.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_rcc_ex.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_dma.c \

Src/main.c \

Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_tim.c \
Src/system_stm32f1xx.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_spi_ex.c \

Src/stm32f1xx_it.c \
Src/stm32f1xx_hal_msp.c \

Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_gpio.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_gpio_ex.c \

Src/main.c \
Src/stm32f1xx_it.c \

Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_rcc.c \

Src/stm32f1xx_hal_msp.c \

Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_flash.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_i2c.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_pwr.c \
Drivers/STM32F1xx_HAL_Driver/Src/stm32f1xx_hal_flash_ex.c  

We can see that main.c, stm32f1xx_hal_msp.c and stm32f1xx_it.c are included twice on the C_SOURCES entry. For sure a CubeMX bug.

Anyway, just delete the duplicated entries, and now the compilation and linkage succeeds.

Happy coding!

TTN LoraWan Atmega32U4 based node – ABP version

TTN is the The Things Network provides the needed backend services and infra-structure for supporting IoT (Internet of Things) connectivity that uses the LORAWAN protocol.
Anybody can participate by either providing the radio gateways that feed the received data to the TTN backend that then delivers it to the user applications, and so increasing the coverage of the TTN network, or just use the network by building TTN Lorawan nodes.

This post is regarding the later case, the build of a simple node based on an Arduino board: the Arduino Micro Pro. So why the Micro PRO, these are quite more expensive than the normal Arduinos, but come in two versions: 5V and 3.3V. Since the Lora radio works with 3.3V, I’ve chosen the 3.3V version so that I do needed to use level shifters from a 5V based board. At least was what I thought a few months ago. Anyway, the Micro PRO chip, the Atmega32u4 has embedded USB connectivity, so no need for serial adapters and boards which might lead to lower power consumption.

At least right now, on sites like eBay and Aliexpress, boards like the Lora32u4 come at least in two versions: with the Atmega328p and with the Atmega32u4. Both suffer the same problem, they Atmel uProcessor used only has 32K of RAM which might be too short to be used for some applications since the Lorawan stack takes a huge amount of space. We will see about that. The great advantage of these boards is they also have connection and charger for a LiPo battery, so in reality all we need is to add sensors, battery and our code.

The node build:
While I’m waiting for my Atmega32U4 based Lora32u4 board, I’m using an Hoperf RFM95 radio soldered on board/shield designed for the Wemos ESP8266: Wemos RFM95 Lora shield. this way I can use the RFM radio either on the ESP8266 Wemos based set of boards, or, as in this case, with the Arduino 32u4.

The Hallard shield as one interesting feature that is that merges all the Lora transceiver status pins by using diodes and hence only use on Arduino pin for inquiring that status. this is due to the lack of pins on the Wemos ESP8266 board. For this to work, we need to add a pull-down resistor to the Arduino pin that connects to the merged output. In my case I used a 10K resistor.
The RFM95 radio is controlled using SPI, so we need to use also the SPI Arduino Pins, and also need to connect the Chip Select pin.
The schematics is as follows:

Arduino Pro Micro and RFM95 Wemos Shield

The node software:
After the node hardware build is done, from the software perspective the node needs now at least another two things: the LMIC stack for implementing the Lorawan protocol support over the Lora radio and, at the TTN site, the device configuration.

Since I’m using Platformio to develop, the LMIC library is the library 852: pio lib show 852. We need to install it and add the reference to it on the file platformio.ini. Also since there is no ATMega 32U4 board on the Platformio IDE available boards, we can use the Adafruit Feather 32u4 board, which is the same thing:

[env:feather32u4]
platform = atmelavr
board = feather32u4
framework = arduino
lib_install= 852

The device registration can be done so that the node device access the TTN network in two different ways:

  1. ABP – Activation by personalisation – This means that all set of keys required to join the Lorawan network are embedded into the software.
  2. OTAA – Over the Air Activation – The network session keys needed to join the Lorawan network are generated when the device tries to join the network.

On this post we will ABP first, since I have no nearby TTN gateway capable o OTTA (I’m using a single channel gateway without downlink support.).

Anyway, the node code is really nothing special, except the necessary configuration for the LMIC to communicate with our RFM95 board.

On the ABP device registration TTN page we need to register our device, so that, on main.cpp code file we can fill the required keys and device ID.

As a quick introduction, after registering onto the TTN site, we go to the console and choose Applications. We can there create or reuse an existing application and register the device, making sure we choose ABP as the method to join the network.

On the Device EUI field, either we fill it or press the crossing arrows to generate an ID. We let the system generate an ID, and then we can finally press the Register button.

The newly register device is configured as an OTAA device:

So we go to Settings and change the OTAA to ABP. After this step we have the required data to put on our code.

Since our node doesn’t have any memory to track frame counting that survives reboots or power cycles, we disable the frame counter checks.

Don’t forget to press save. Again on the main device screen we can now copy the keys to the code:

We can now copy the keys:

static u1_t NWKSKEY[16] = { 0xEE, ... ... ... ... }; // <- Put here the NETWORK KEY
static u1_t APPSKEY[16] = { 0x4E, 0x12, ... ... ... ... };  // <- Put here the APPLICATION KEY
static u4_t DEVADDR = 0x26304050;   // Put here the device id in hexadecimal form.

Testing:

Compiling the code with the pio run command, we have the following output:

Calculating size .pioenvs/feather32u4/firmware.elf
AVR Memory Usage
----------------
Device: atmega32u4

Program:   28542 bytes (87.1% Full)
(.text + .data + .bootloader)

Data:        957 bytes (37.4% Full)
(.data + .bss + .noinit)

And we can flash the firmware with the command: pio run -t upload.

The result is data on the TTN console referring to our device:

The problem… :
So, everything runs OK, and we can send data to the TTN Network, everything looks good, right?

As soon we start to add functionality to our code, for example reading some I2C sensors, our some serial debug messages, we hit this problem:

Linking .pioenvs/feather32u4/firmware.elf
Checking program size
text       data     bss     dec     hex filename
Error: The program size (28756 bytes) is greater than maximum allowed (28672 bytes)
28548       208     749   29505    7341 .pioenvs/feather32u4/firmware.elf
*** [.pioenvs/feather32u4/firmware.elf] Explicit exit, status 1

So in reality we can’t add much functionality to our code if using a full LMIC stack, since it occupies a lot of the available flash memory.

Trimming down the LMIC stack:
Since our node is ABP only we can strip out some LMIC functionality for OTAA an other Lorawan features. For this we need to edit the config.h file from the LMIC library. Since we are using platformio, this file is located at project_root/.piolibdeps/IBM LMIC framework_ID852/src/lmic

We only leave support for ABP by enabling the disable lines for other LMIC functionality:

...
...
// Any runtime assertion failures are printed to this serial port (or
// any other Print object). If this is unset, any failures just silently
// halt execution.
#define LMIC_FAILURE_TO Serial

// Uncomment this to disable all code related to joining
#define DISABLE_JOIN
// Uncomment this to disable all code related to ping
#define DISABLE_PING
// Uncomment this to disable all code related to beacon tracking.
// Requires ping to be disabled too
#define DISABLE_BEACONS

// Uncomment these to disable the corresponding MAC commands.
// Class A
//#define DISABLE_MCMD_DCAP_REQ // duty cycle cap
//#define DISABLE_MCMD_DN2P_SET // 2nd DN window param
//#define DISABLE_MCMD_SNCH_REQ // set new channel
// Class B
#define DISABLE_MCMD_PING_SET // set ping freq, automatically disabled by DISABLE_PING
#define DISABLE_MCMD_BCNI_ANS // next beacon start, automatical disabled by DISABLE_BEACON

By uncommenting the above lines, our code now takes (we can and should ignore the LMIC compile warnings):

AVR Memory Usage
----------------
Device: atmega32u4

Program:   23324 bytes (71.2% Full)
(.text + .data + .bootloader)

Data:        796 bytes (31.1% Full)
(.data + .bss + .noinit)

So around 5KB less without the OTAA and Class B support.

So we have a bit more memory to do something useful.

Enabling OTAA by commenting the line //#define DISABLE_JOIN:

AVR Memory Usage
----------------
Device: atmega32u4

Program:   25048 bytes (76.4% Full)
(.text + .data + .bootloader)

Data:        912 bytes (35.6% Full)
(.data + .bss + .noinit)

We still have around 3K free. Tight but might be enough.

Conclusion:
The availability of boards with the AtMega32u4 processor, Lora Radio and LiPo charge and battery connectivity, is a great step to start using the TTN (or other) Lorawan networks. But with only with 32K or flash memory, for some applications, these boards might not be the best solution.
Also the price for such boards are still a bit on the expensive side, since a discrete 32u4 + RFM95 + Lipo charger is a bit cheaper than the single board solution.
Anyway, the STM32F103 blue pill boards, cost half of the 32U4 price and have double the flash size and 9x the clock, so it would be great that such single boards, instead of the 328p or 32u4 used the STM32F103…
So my conclusion is, without power considerations taken into account, a STM32F103 + RFM95 and LiPo charger, is a better alternative than the one that I’ve used here.

Using the Blackmagic probe with Netbeans

Using the Blackmagic debug probe (BMP) with Netbeans is very similar when doing the same but with using Openocd.

The initial steps are the same to use the BMP probe and Openocd. First we set up the ARM toolchain or other, as needed. We also need the gdbserver plugin for Netbeans, and so, we also need to install it. After these two initial steps, the remaining configuration and how to use the probe differs a bit differently from when using Openocd.

One of the main differences is that with Openocd, externally we launch an instance of Openocd and then use the Netbeans gdbserver plugin to connect to that openocd instance. For that we use the Netbeans menu Debug and Attach Debugger and use connection string extended-remote localhost:3333 and the selected project:

For using the BMP probe we don’t need no intermediary software since the code supporting the gdbserver plugin is already on the probe firmware, which means that the debugger can connect directly to the target. In fact due to this we do not need now to attach to a debugger and we can start using the Debug Project (CTRL-F5) to program/flash the code output (optional) and to start debugging the code running on the target device.

For this to happen we need to configure the Debug Command for the project (this must be done for each project if needed) and if wanted we can also configure the Run Command.
Before we are able to do that, we need first to create a command file for the gdb debugger that will pre-execute some commands, namely to attach to the targets probe, before starting the debugging session.

For that I’ve created a file named BMPgdbinit located, in my case, in /opt/ARM:

target extended-remote /dev/ttyACM0
monitor swdp_scan
att 1
load
start
run

This file is invoked by the ARM debugger toolchain (like this: arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit file.out) and it will connect to the Blackmagic probe, that should be located at /dev/ttyACM0, scan the SWD searching for the device, attaches to it, loads/programs the output file, namely the firmware, and starts it. From this point the gdbserver from Netbeans takes control and should stop at the first line of code or the first break point.

So to configure the run command we edit the Project Properties and modify the Run command:

Just define the Run Command as arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit “${OUTPUT_PATH}”

With this configuration we can now press F6 to flash our firmware (because the load command is on the BMPgdbinit file) and run it.

For debugging we need the following configuration:

We must define the Debug Command as ${OUTPUT_PATH} since this will be passed as the arm-none-eabi-gdb file parameter.
We also need to point to the correct location of the BMPgdbinit file.

We can now just press CTRL-F5 to launch a debug session, and again, the firmware will be loaded, and the code will stop at the first break point.

An example of using Netbeans to build a simple program targeting the nRF52832 (the blinky example from pcbreflux ) with an added variable and a watch point set:

That’s it….

Issues:

There are some issues with the above process…

Can’t stop the debugger. Pressing Pause does nothing…
This is a long standing bug apparently so the solution is on a external terminal window send a SIGINT signal to the gdb process:

[pcortex@pcortex:~]$ ps -ef | grep arm-none
pcortex 9792  9405  0 20:22 ?        00:00:00 /tmp/dlight_fdam/5fe1c993/0397644155/pty --dir /opt/nordic/nRF52832/blinky --no-pty /opt/ARM/gcc-arm-none-eabi-6_2-2016q4/bin/arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit -nx --interpreter mi -tty /dev/pts/7
pcortex 9794  9792  0 20:22 ttyACM0  00:00:00 /opt/ARM/gcc-arm-none-eabi-6_2-2016q4/bin/arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit -nx --interpreter mi -tty /dev/pts/7
 

Just send a SIGINT signal to the process associated to the Blackmagic probe:

kill -SIGINT 9794

The control is returned to Netbeans.

The debugger doesn’t seem to follow my code…
Just Clean and build all the project and select run so a clean version of the program is flashed.

I unplug and plug the probe a lot and the tty ports change..

Because of this the BMPgdbinit script can be broken because the probe port changes, for example, to /dev/ttyACM2.

The solution is to create a rules file:

Mine, for Arch Linux is as follows:

# Black Magic Probe
# there are two connections, one for GDB and one for uart debugging
  SUBSYSTEM=="tty", ATTRS{interface}=="Black Magic GDB Server", SYMLINK+="ttyBMP"
  SUBSYSTEM=="tty", ATTRS{interface}=="Black Magic UART Port", SYMLINK+="ttyBMPUART"

This file is named 99-blackmagic.rules located at /etc/udev/rules.d.

We now need to change the port used on the BMPgdbinit file from /dev/ttyACM0 to /dev/ttyBMP. Note that after plugging in the probe to the USB port, it takes a while (2 to 3 seconds) to the ports appear.

This just doesn’t work…

Yes, some times the behaviour seems strange… I’ve worked around some of the issues by flashing again the softdevice and the program mannually from the GDB command line.

Also if something like this happens:

It is due to the fact that the Netbeans environment variables $OUTPUT_PATH is empty… The solution is to explicitly state the outputfile on the RUN and Debug commands instead of using the ENV var. For example change this:

arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit "${OUTPUT_PATH}"

to this

arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit /opt/Projects/nRF/proj1/program.out

or to this:

arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit "${PROJECT_DIR}/file.out"

Final note:
As a final note, some other errors might creep up like:

  • Program is not being run: Check the command line of arm-none-eabi-gdb to see if the BMPgdbinit file is correctly provided.
  • Don’t know how to run. Try \”help target\”.: Same issue. Check if the run and debug commands are correctly defined.
  • Sometimes my target isn’t detected.: Adding an extra monitor swdp_scan line to the BMPgdbinit file might help.

Building a Blackmagic Debug Probe

The Black Magic debug probe is in-application debugger, which allows IDE’s like NetBeans, Eclipse or others to connect directly to the hardware without using bridges like for example Openocd. The Blackmagic Probe software is open source and available on this Github link.

There are several posts/instructions on the internet regarding how to load the Black Magic firmware either on ST-Link hardware debuggers, ST-Link clones or using the “blue pill” STM32F103C8T6 based board.

There are several possible issues with this board:

  1. First they are quite cheap – not an issue
  2. It seems on some versions there is a problem with the USB connector regarding a resistor and bad contacts. In my version I had no issues. It just worked fine.
  3. The on board chip is the STM32F103C8T6 which has 64Kb of flash, but some claim that it can have 128Kb. In my case all the boards that I have, the flash size is 64Kb.

This last issue is a show stopper at the current release of the Black Magic firmware since it won’t fit on the 64Kb flash of the ST32F103C8T6. We must cut things from the firmware to be able to flash it on the STM32F103C8T6 board. But we will see how to do that.

Downloading and compiling the firmware:
This step is just straight forward, just clone the Blacmagic repository and compile. Since we are compiling to ST32F103 board we will assume it is a ST-Link clone.

[pcortex@pcortex:opt]$ git clone https://github.com/blacksphere/blackmagic
Cloning into 'blackmagic'...
remote: Counting objects: 5029, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 5029 (delta 1), reused 2 (delta 0), pack-reused 5020
Receiving objects: 100% (5029/5029), 2.01 MiB | 1.38 MiB/s, done.
Resolving deltas: 100% (3558/3558), done.
[pcortex@pcortex:opt]$ cd blackmagic/
README.md  Makefile  libopencm3/  HACKING  driver/  COPYING  scripts/  src/  upgrade/
[pcortex@pcortex:blackmagic|master]$ make
Initialising git submodules...
Submodule 'libopencm3' (https://github.com/libopencm3/libopencm3.git) registered for path 'libopencm3'
Cloning into '/opt/lixo/blackmagic/libopencm3'...
Submodule path 'libopencm3': checked out '67242de60dec0227739cd549e8a78e1a3c15dbf5'
  GENHDR  include/libopencm3/efm32/efm32gg/irq.json
  GENHDR  include/libopencm3/efm32/efm32g/irq.json
  GENHDR  include/libopencm3/efm32/efm32lg/irq.json
  GENHDR  include/libopencm3/efm32/efm32tg/irq.json
  GENHDR  include/libopencm3/stm32/f2/irq.json
...
...
...

Now we must target the firmware to our blue pill board, and so compiling the firmware as it was for a ST-Link clone:

[pcortex@pcortex:blackmagic|master]$ cd src
[pcortex@pcortex:src|master]$ make clean
 CLEAN
[pcortex@pcortex:src|master]$ make PROBE_HOST=stlink
GIT     include/version.h
  CC      target/adiv5.c
  CC      target/adiv5_jtagdp.c
  CC      target/adiv5_swdp.c
  CC      command.c
  CC      target/cortexa.c
  CC      target/cortexm.c
...
...
...
  CC      platforms/stlink/dfu_upgrade.c
  LD      dfu_upgrade
  OBJCOPY dfu_upgrade.bin
  OBJCOPY dfu_upgrade.hex

Done! On the src there is now at least two files:

  1. blackmagic_dfu.bin – The Device Firmware Upgrade loader.
  2. blackmagic.bin – The Blackmagic firmware.

Checking the file sizes we can see that the blackmagic.bin file (as of today) 57K, and the bootloader is 6.8K.

Flashing the firmware:
Flashing the firmware can be done by several forms:

  1. Using the STM32 embedded loader through a serial port
  2. Using the SWD connection, ST-Link probe and Openocd
  3. Using the SWD connection, ST-Link probe and Texane/st-link/st-util
  4. Using the DFU protocol, but it depends if the board already has a DFU firmware loaded.

Most of the posts on the internet regarding this step use the first aproach, namely using the STM32 boot loader and serial port. This needs a serial to USB converter to be connected and the STM32 boot pins to be modified.

Anyway, I’m flashing the firmware using Openocd and/or st-link utils through a ST-Link debug probe:

The blue pill pinout is as follows where the SWD connector is oposite the USB port:

From the ST-Link probe we need to connect the following pins:

  • Probe SWCLK Blue Pill DCLK
  • Probe SWDIO Blue Pill DIO
  • Probe GND Blue Pill GND

Regarding the power, I advise not to connect it and power up the blue pill board through the USB connector. This will allow to have access to both boards through USB without any possible power clashes.

We can now flash the firmware either with Texane St-Link programs or the AUR Arch Linux package stlink-git or using Openocd.

For using Openocd, create the following file and name it for example bluepill.cfg:

set CHIPNAME STM32F103C8T6

source [find interface/stlink-v2.cfg]
transport select hla_swd
source [find target/stm32f1x.cfg]
set WORKAREASIZE 0x2000

Also make sure that the ST-Link probe is detected so both st-link utils and openocd can work:

[pcortex@pcortex:src|master]$ lsusb
Bus 002 Device 003: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 003: ID 046d:c52f Logitech, Inc. Unifying Receiver
Bus 008 Device 002: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
...
Bus 003 Device 008: ID 0483:3748 STMicroelectronics ST-LINK/V2

We can now flash our board with the following command using Openocd:

openocd -f ../bluepill.cfg -c ‘init_reset halt; program blackmagic_dfu.bin 0x8000000 verify; reset;exit’

[pcortex@pcortex:src|master]$ openocd -f ../bluepill.cfg -c 'init_reset halt; program blackmagic_dfu.bin 0x8000000 verify; reset;exit'
Open On-Chip Debugger 0.10.0
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
0x2000
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v24 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.208372
Info : STM32F103C8T6.cpu: hardware has 6 breakpoints, 4 watchpoints
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x080017b4 msp: 0x20005000
** Programming Started **
auto erase enabled
Info : device id = 0x20036410
Info : flash size = 64kbytes
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000003a msp: 0x20005000
wrote 7168 bytes from file blackmagic_dfu.bin in 0.656939s (10.655 KiB/s)
** Programming Finished **
** Verify Started **
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x20005000
verified 6900 bytes in 0.187018s (36.030 KiB/s)
** Verified OK **
[pcortex@pcortex:src|master]$

Using the st-flash command as documented on STM as BMP

[pcortex@pcortex:src|master]$ st-flash write blackmagic_dfu.bin 0x8000000
st-flash 1.3.1
2017-06-13T20:26:01 INFO src/common.c: Loading device parameters....
2017-06-13T20:26:01 INFO src/common.c: Device connected is: F1 Medium-density device, id 0x20036410
2017-06-13T20:26:01 INFO src/common.c: SRAM size: 0x5000 bytes (20 KiB), Flash: 0x10000 bytes (64 KiB) in pages of 1024 bytes
2017-06-13T20:26:01 INFO src/common.c: Attempting to write 6888 (0x1ae8) bytes to stm32 address: 134217728 (0x8000000)
Flash page at addr: 0x08001800 erased
2017-06-13T20:26:01 INFO src/common.c: Finished erasing 7 pages of 1024 (0x400) bytes
2017-06-13T20:26:01 INFO src/common.c: Starting Flash write for VL/F0/F3 core id
2017-06-13T20:26:01 INFO src/flash_loader.c: Successfully loaded flash loader in sram
  6/6 pages written
2017-06-13T20:26:02 INFO src/common.c: Starting verification of write complete
2017-06-13T20:26:02 INFO src/common.c: Flash written and verified! jolly good!
[pcortex@pcortex:src|master]$

So far so good, but the above is just DFU firmware that allows to use DFU utils later to upgrade the firmware through the USB port without using any other tools.

If unplug and plug our blue pill board and check for the USB ports we have now:

[pcortex@pcortex:src|master]$ lsusb
Bus 002 Device 003: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 003: ID 046d:c52f Logitech, Inc. Unifying Receiver
Bus 008 Device 002: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
...
Bus 003 Device 012: ID 1d50:6017 OpenMoko, Inc. Black Magic Debug Probe (DFU)
Bus 003 Device 008: ID 0483:3748 STMicroelectronics ST-LINK/V2
...

And we can now also use the dfu-util command to upload the firmware.

To upload the firmware we will see that it fails due to the firmware size and the 64Kb flash size. It might not happen to you if your board has more than 64Kb of flash available.

With Openocd command:

openocd -f ../bluepill.cfg -c 'init_reset halt; program blackmagic.bin verify; reset;exit'    
Open On-Chip Debugger 0.10.0
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
0x2000
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v24 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.208372
Info : STM32F103C8T6.cpu: hardware has 6 breakpoints, 4 watchpoints
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x080017b4 msp: 0x20005000
** Programming Started **
auto erase enabled
Info : device id = 0x20036410
Info : flash size = 64kbytes
Warn : no flash bank found for address 0
wrote 0 bytes from file blackmagic.bin in 0.016043s (0.000 KiB/s)
** Programming Finished **
** Verify Started **
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x20005000
Error: checksum mismatch - attempting binary compare
diff 0 address 0x08010000. Was 0xff instead of 0x00
diff 1 address 0x08010001. Was 0xff instead of 0x00
diff 2 address 0x08010002. Was 0xff instead of 0x00
diff 3 address 0x08010003. Was 0xff instead of 0x00
diff 4 address 0x08010004. Was 0xff instead of 0x00
diff 5 address 0x08010005. Was 0xff instead of 0x00
diff 6 address 0x08010006. Was 0xff instead of 0x00

....
diff 123 address 0x0801007b. Was 0xff instead of 0x00
diff 124 address 0x0801007c. Was 0xff instead of 0x00
diff 125 address 0x0801007d. Was 0xff instead of 0x00
diff 126 address 0x0801007e. Was 0xff instead of 0x00
diff 127 address 0x0801007f. Was 0xff instead of 0x00
More than 128 errors, the rest are not printed.
embedded:startup.tcl:476: Error: ** Verify Failed **
in procedure 'program' 
in procedure 'program_error' called at file "embedded:startup.tcl", line 520
at file "embedded:startup.tcl", line 476

With the st-flash command:

[pcortex@pcortex:src|master]$ st-flash --reset write blackmagic.bin 0x8002000
st-flash 1.3.1
2017-06-13T20:37:27 INFO src/common.c: Loading device parameters....
2017-06-13T20:37:27 INFO src/common.c: Device connected is: F1 Medium-density device, id 0x20036410
2017-06-13T20:37:27 INFO src/common.c: SRAM size: 0x5000 bytes (20 KiB), Flash: 0x10000 bytes (64 KiB) in pages of 1024 bytes
2017-06-13T20:37:27 INFO src/common.c: Attempting to write 58084 (0xe2e4) bytes to stm32 address: 134225920 (0x8002000)
2017-06-13T20:37:27 ERROR src/common.c: addr too high
stlink_fwrite_flash() == -1
[pcortex@pcortex:src|master]$ 

And with the dfu util forcing the write above 64Kb:

[pcortex@pcortex:src|master]$ sudo dfu-util -a 0 -s 0x08002000:leave:force -D blackmagic.bin
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

dfu-util: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
Opening DFU capable USB device...
ID 1d50:6017
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #0 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 1024
DfuSe interface name: "Internal Flash   "
Downloading to address = 0x08002000, size = 58084
dfu-util: Last page at 0x080102e3 is not writeable
[pcortex@pcortex:src|master]$ 

So no luck with my board version, and as a solution we need to strip down the firmware.

Striping down the firmware:
To make the BMP firmware able to fit the 64Kb flash we have to cut things. In my case I’ve removed some processors support since my work is with the STM32 and nRF families:

Edit the file named cortexm.c located at target subdirectory, and delete the following lines around line 254 (or search by PROBE) to remove the unneeded support:

#define PROBE(x) \
        do { if ((x)(t)) return true; else target_check_error(t); } while (0)

        PROBE(stm32f1_probe);
        PROBE(stm32f4_probe);
        PROBE(stm32l0_probe);   /* STM32L0xx & STM32L1xx */
        PROBE(stm32l4_probe);
        PROBE(lpc11xx_probe);
        PROBE(lpc15xx_probe);
        PROBE(lpc43xx_probe);
        PROBE(sam3x_probe);
        PROBE(sam4l_probe);
        PROBE(nrf51_probe);
        PROBE(samd_probe);
        PROBE(lmi_probe);
        PROBE(kinetis_probe);
        PROBE(efm32_probe);

Still at the source directory we build again the firmware:

[pcortex@pcortex:src|master]$ make clean
[pcortex@pcortex:src|master]$ make PROBE_HOST=stlink
[pcortex@pcortex:src|master]$ ls -l blackmagic*
-rwxr-xr-x 1 pcortex users 753K jun 13 20:53 blackmagic*
-rwxr-xr-x 1 pcortex users  55K jun 13 20:53 blackmagic.bin*
-rwxr-xr-x 1 pcortex users 325K jun 13 20:53 blackmagic_dfu*
-rwxr-xr-x 1 pcortex users 6,8K jun 13 20:53 blackmagic_dfu.bin*
-rw-r--r-- 1 pcortex users  20K jun 13 20:53 blackmagic_dfu.hex
[pcortex@pcortex:src|master]$

We can seen now that the firmware was “slim down” around 2K, which is enough now to fit the 64Kb flash:

Using Openocd:

[pcortex@pcortex:src|master]$ openocd -f ../bluepill.cfg -c 'init_reset halt; program blackmagic.bin 0x8002000 verify; reset;exit'
Open On-Chip Debugger 0.10.0
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
0x2000
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v24 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.208372
Info : STM32F103C8T6.cpu: hardware has 6 breakpoints, 4 watchpoints
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x080017b4 msp: 0x20005000
** Programming Started **
auto erase enabled
Info : device id = 0x20036410
Info : flash size = 64kbytes
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000003a msp: 0x20005000
wrote 54272 bytes from file blackmagic.bin in 3.792976s (13.973 KiB/s)
** Programming Finished **
** Verify Started **
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x20005000
verified 53744 bytes in 0.854957s (61.388 KiB/s)
** Verified OK **

Using st-flash:

[pcortex@pcortex:src|master]$ st-flash --reset write blackmagic.bin 0x8002000
st-flash 1.3.1
2017-06-13T20:58:02 INFO src/common.c: Loading device parameters....
2017-06-13T20:58:02 INFO src/common.c: Device connected is: F1 Medium-density device, id 0x20036410
2017-06-13T20:58:02 INFO src/common.c: SRAM size: 0x5000 bytes (20 KiB), Flash: 0x10000 bytes (64 KiB) in pages of 1024 bytes
2017-06-13T20:58:02 INFO src/common.c: Attempting to write 55720 (0xd9a8) bytes to stm32 address: 134225920 (0x8002000)
Flash page at addr: 0x0800f800 erased
2017-06-13T20:58:05 INFO src/common.c: Finished erasing 55 pages of 1024 (0x400) bytes
2017-06-13T20:58:05 INFO src/common.c: Starting Flash write for VL/F0/F3 core id
2017-06-13T20:58:05 INFO src/flash_loader.c: Successfully loaded flash loader in sram
 54/54 pages written
2017-06-13T20:58:09 INFO src/common.c: Starting verification of write complete
2017-06-13T20:58:09 INFO src/common.c: Flash written and verified! jolly good!

Using the DFU firmware upload:

[pcortex@pcortex:src|master]$ sudo dfu-util -a 0 -s 0x08002000:leave:force -D blackmagic.bin
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

dfu-util: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
Opening DFU capable USB device...
ID 1d50:6017
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #0 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 1024
DfuSe interface name: "Internal Flash   "
Downloading to address = 0x08002000, size = 55720
Download        [=========================] 100%        55720 bytes
Download done.
File downloaded successfully
Transitioning to dfuMANIFEST state

And now if we unplug the blue pill board and connect it again we should have two new tty USB based ports:

[35014.003313] usb 3-2.4: new full-speed USB device number 20 using uhci_hcd
[35019.346373] cdc_acm 3-2.4:1.0: ttyACM0: USB ACM device
[35019.349364] cdc_acm 3-2.4:1.2: ttyACM1: USB ACM device

Success!

Starting using the BMP probe:

To start using our blue pill board as a Blackmagic probe we need the following information and configuration:

  1. Connect a LED (with a resistor…) to pin PA8 to show the probe activity.
  2. The target SWD pins are CLK at pin PA5 and DIO at PB14.
  3. The support for the USB to UART bridge (the second ttyACM1 port) is PA3 – RX and PA2 – TX

So we now can use the BM probe instead of a ST-Link probe, and connect the target board SWD pins to PA5 e PB14: CLK and DIO respectively.

The led will blink when the program is running or it will fast blink when loading a program into the target board.

More information on this video from no other than the builder of the Magic Probe to see more of the probe capabilities. I also recommend the Blackmagic probe wiki for further information.

A quick example with an nRF52 based board connected to the BMP by using the ARM GDB debugger arm-none-eabi-gdb directly:

(gdb) target extended-remote /dev/ttyACM0
Remote debugging using /dev/ttyACM0
(gdb) monitor swdp_scan
Target voltage: unknown
Available Targets:
No. Att Driver
 1      Nordic nRF52
(gdb) att 1
(gdb) monitor
Black Magic Probe (Firmware v1.6.1-25-gaaa7b0e-dirty) (Hardware Version 0)
Copyright (C) 2015  Black Sphere Technologies Ltd.
License GPLv3+: GNU GPL version 3 or later 


Interesting links:

Most of this post wouldn’t be possible without gathering information from around the web. Here is a set of information that might be useful to gather more or missing information from this post:

  1. Black Magic Probe (BMP) on ST-Link v2 clones
  2. STM Discovery as Black Magic Probe
  3. STM32Duino forum