MQTT-SN Paho gateway and UDP6

Sometimes simple things can take a lot of time, such as this: How to make the Paho MQTT-SN gateway to listen to UDP6 instead of the standard UDP4 ports…

Introduction:
A lot can be written about MQTT-SN and it’s counterpart MQTT. To keep it simple and short, MQTT-SN (SN for Sensor Networks) uses UDP instead of TCP/IP and is streamlined to be more efficient in transmitting information, and so more adequate to be used on constrained devices than MQTT on WSN (Wireless Sensor Networks).
One important key element in using MQTT-SN is that MQTT-SN clients use UDP broadcasts to find gateways, and so they have no need to hardcode a broker address on the firmware code, since gateways can be dynamically discover at run time. Still MQTT-SN depends on MQTT broker to connect to the outside world (and that’s why MQTT-SN is a gateway, not a broker), but that is configured at the gateway level, not at the node level.

Compiling and configuring:
If we just clone the MQTT-SN Github repository, we can for UDP version 4, just do the following:

git clone https://github.com/eclipse/paho.mqtt-sn.embedded-c
cd MQTTSNGateway/
make 

And that’s it. On the Build/ directory there are the executables, among them the MQTT-SNGateway, and on the current directory there is the configuration file gateway.conf that is used by the gateway to startup. Out of the box the only thing that we need to change is the upstream MQTT broker address that by default points to the iot.eclipse.org MQTT broker, the gateway name, and maybe the ports:

BrokerName=iot.eclipse.org
BrokerPortNo=1883
BrokerSecurePortNo=8883

GatewayID=1
GatewayName=PahoGateway-01
KeepAlive=900

# UDP
GatewayPortNo=10000
MulticastIP=225.1.1.1
MulticastPortNo=1883

So if we start with this config, the output is:

20190612 142837.189 PahoGateway-01 has been started.

 ConfigFile: ./gateway.conf
 PreDefFile: ./predefinedTopic.conf
 SensorN/W:  UDP Multicast 225.1.1.1:1883 Gateway Port 10000
 Broker:     172.17.0.4 : 1883, 8883
 RootCApath: (null)
 RootCAfile: (null)
 CertKey:    (null)
 PrivateKey: (null)

We can see that in this case my upstream broker is at 172.17.0.4 (a docker image of the Mosquitto broker).

Checking which ports the gateway is listening:

root@b74f5ad3fd8e:/app/mqttsn# lsof -c MQTT-SNGateway -a -i4
COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
MQTT-SNGa 923 root    3u  IPv4 5017719      0t0  UDP *:10000 
MQTT-SNGa 923 root    4u  IPv4 5017720      0t0  UDP *:1883 
root@b74f5ad3fd8e:/app/mqttsn# lsof -c MQTT-SNGateway -a -i6
root@b74f5ad3fd8e:/app/mqttsn# 

We can see that the gateway is listening on UDP4 and not on UDP6.

Supporting UDP6:
Sensor networks such as Openthread networks work only with IPv6… and so the MQTT-SN gateway must listen to UDP6 instead of UDP4. It can still communicate with the MQTT broker over IPv4, and hence bridging the IPv4 network with the IPv6 sensor network.

To add support to UDP6 for the MQTT-SN gateway we need to change the Makefile (and hence the time to find information for this little bit… ) to change the protocol used by the sensor network and recompile.

So again on the source folder for the MQTT-SN gateway, edit the Makefile and change the following line from

SENSORNET := udp

to

SENSORNET := udp6

Or just change the parameter at the make call: make SENSORNET=udp6

And then clean and build:

make clean
make SENSORNET=udp6

We have now a MQTT-SN gateway version for UDP6. We need now to add the configuration entries to the gateway configuration file so that the UDP6 configuration is set:

#UDP v6
GatewayUDP6Broadcast = ff03::1
GatewayUDP6Port = 47193
GatewayUDP6If = wpan0

We can now start the UDP6 MQTT-SN enabled version:

20190612 144124.280 PahoGateway-01 has been started.

 ConfigFile: ./gateway.conf
 PreDefFile: ./predefinedTopic.conf
 SensorN/W:   Gateway Port: 47193 Broadcast Address: ff03::1 Interface: wpan0
 Broker:     172.17.0.4 : 1883, 8883
 RootCApath: (null)
 RootCAfile: (null)
 CertKey:    (null)
 PrivateKey: (null)

Specifically in this case the GatewayUDP6If interface that is defined is the NCP OpenThread interface, but it should be the interface where we want the gateway to listen. The broadcast address is the address defined on the Nordic SDK for Thread, namely on the file mqttsn_packet_internal.h.

Checking the ports:

root@b74f5ad3fd8e:/app/mqttsn# lsof -c MQTT-SNGateway -a -i4
root@b74f5ad3fd8e:/app/mqttsn# lsof -c MQTT-SNGateway -a -i6
COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
MQTT-SNGa 947 root    4u  IPv6 5050130      0t0  UDP *:47193 
root@b74f5ad3fd8e:/app/mqttsn# 

And that’s it.

Conclusion:
A simple lack of documentation regarding MQTT-SN and UDP6 took a while to solve since it was so “simple”.

Still one still needs two compiled versions of MQTT-SN for supporting UDP4 and UDP6 based networks, but since MQTT-SN is light there is no big issue with this.

Advertisements

When updating breaks projects…

A quick post regarding updating platforms and libraries for projects, specifically projects for the ESP8266 platforms:

The PZEM004T is device available on eBay and other sites, that allows to measure energy consumption. The PZEM004T has a serial output port and when connecting it to, for example, an ESP8266, we can access the collected data trough WIFI and process it for finding out how much electricity we are using, and so on.

Anyway, when using the ESP8266 WEMOS D1 mini, to be able to still use the USB serial port, we need to use Software serial emulation. In fact the Arduino PZEM004T library available on Github and on the Platformio library registry allows the use of the Software Serial to communicate with the PZEM004T (and it works just fine).

So what is the issue?

I’m using Platformio to develop the ESP8266 application, and normally when running, it checks and offers any updates that might be available. So, I’ve updated to the latest Espressif ESP8266 platform and ESPSoftwareSerial, and then everything just break down:

  • ESPSoftwareSerial last version just completely breaks the previous existing API which made the PZEM004T library also broken.
  • The new ESP8266 platform removes Esp8266 a SDK attachInterruptArg function which renders the ESPSoftwareSerial library unbuildable

The solution?

The solution with Platformio is quite easy: use semantic versioning.

In fact something like this on the platformio.ini file:

[env:d1_mini]
platform = espressif8266
...
...

can be locked to a working previous version:

[env:d1_mini]
platform = espressif8266@2.0.3
...
...

The same can be done with the project libraries. While the ESPSoftwareSerial (the PZEM004T dependency loads it) does not need to be defined, specifying it allows to use a specific version:

[env:d1_mini]
platform = espressif8266@2.0.3
board = d1_mini_lite
framework = arduino
upload_speed = 921600
monitor_speed = 115200

lib_deps =  ESPSoftwareSerial@5.0.3
            PZEM004T
            MQTT
            LiquidCrystal_I2C
            SimpleTimer
            ESPAsyncTCP
            ESP Async WebServer
            Time

And with this, the ESP8266 platform and ESPSoftwareSerial versions locked, the issues with the newer versions are avoided, and the code compiles and works as it should.

So, updating is fine, but when it breaks it can be an issue. Fortunately Platformio allows the usage of specific version for building our projects, and even allows to deploy our specific library version under the project lib directory.

Have fun!

Programming the Taida Century NRF52832 developer board

Its been a long hiatus regarding this post. At the time I’ve bought a generic eBay/Aliexpress nRF52832 development board (the Taida Century NRF52832 based board) for performing some BLE testing, did some development with Nordic SDK and the NRF Arduino firmware using Platformio, but posted nothing on my blog regarding the tests.
The main trigger for picking up again this board and do some writing about it, is that I’m evaluating RIOT-OS and its capability to target several hardware platforms, namely the NRF52 based chips. Anyway, first the use of the Arduino framework for the NRF52 and platformio:

Programming the NRF52832:
For programming the Taida Century NRF52 board need a programmer supporting the NRF52 chip, that can be either a SEGGER JLink programmer probe (highly recommended), or with a STLink V2 coupled with modified Openocd software, that also does work fine (Recent versions of Openocd already support the NRF52 out of the box). The Blackmagic probe is also an alternative, but since I have now a Jlink probe, never bother again with it.

The hardware connections for programming and debugging:
Since that the board does not have a serial to USB chip and a USB connection, an external TTL to USB serial convert must be used to see any serial output on the computer from the Serial.println commands. Furthermore the USB serial adapter must also must work with the 3.3V used by the NRF52. The 3.3v power can be provided either by the USB serial converter or by the debug probe (but not both at the same time!). I normally use a 6 pin CP2102 USB serial adapters: they work with 3.3v and have all the serial pins exposed.

The Arduino framework for the NRF52 board and specifically for the Taida board variant uses the following pins for serial communication:

RXD – Pin 18
TXD – Pin 19
GND – GND

So the TXD labelled pin on the USB adapter connects to the Pin 18 labelled header on the board (RXD), and the RXD pin on the USB adapter, connects to Pin 19 on the pin header on the board. Ground connects to ground. That settles the serial connection for debugging. The ground pin should also be connected either on the header pin labelled GND on the top left when the IDC connector is on the left, or on the IDC connector itself.

The connections for programming are a bit tricker depending if we are using an STLink programmer or a Segger Jlink programmer. In both cases the power can be provided by the programmer, but with the JLink one, a special connection is also needed so that the SWD debug lines voltage reference used is 3.3v instead of 5v.

Connecting the ST-Link programmer:
Since we are powering up the board from the programmer, and using one of those cheap st-link usb dongles, only four connections are needed:

The USB programmer has a 3.3v output and so can power up the board directly to the 3.3v board pin.

Connecting the JLink probe:
The Jlink probe can only provide 5V output on the JTAG pins, and so while the JTAG pins TCK and TMS can be connected to the TCK and TMS pins on the JTAG connector, the 5V pin must be connected to the VBUS pin on the IDC connector. This pin will provide power to the board and the internal regulator drops it down from 5V to 3.3V. To adjust the logic levels from the probe to 3.3V the JTAG pin VREF must be connected to the IDC pin 3.3V pin, that informs the JLink probe the device target working voltage.
So in total one more connection that with the ST-Link programmer.

The connection diagram is as follows, where the Taida board 12 pin IDC connector is used:

After the connection is done, we can test if we can communicate with the board using the JLinkExe:

JLinkExe -if SWD -device NRF52 -speed 4000
SEGGER J-Link Commander V6.44 (Compiled Mar  1 2019 17:36:52)
DLL version V6.44, compiled Mar  1 2019 17:36:42

Connecting to J-Link via USB...O.K.
Firmware: J-Link V10 compiled Mar  1 2019 16:57:49
Hardware version: V10.10
S/N: XXXXXXXXX
License(s): FlashBP, GDB
OEM: SEGGER-EDU
VTref=3.264V

Type "connect" to establish a target connection, '?' for help
J-Link>connect
Device "NRF52" selected.

Connecting to target via SWD
Found SW-DP with ID 0x2BA03466
Found SW-DP with ID 0x2BA03466
Scanning AP map to find all available APs
AP[2]: Stopped AP scan as end of AP map has been reached
AP[0]: AHB-AP (IDR: 0x24770011)
AP[1]: JTAG-AP (IDR: 0x02880000)
Iterating through AP map to find AHB-AP to use
AP[0]: Core found
AP[0]: AHB-AP ROM base: 0xE00FF000
CPUID register: 0x410FC241. Implementer code: 0x41 (ARM)
Found Cortex-M4 r0p1, Little endian.
FPUnit: 6 code (BP) slots and 2 literal slots
CoreSight components:
ROMTbl[0] @ E00FF000
ROMTbl[0][0]: E000E000, CID: B105E00D, PID: 000BB00C SCS-M7
ROMTbl[0][1]: E0001000, CID: B105E00D, PID: 003BB002 DWT
ROMTbl[0][2]: E0002000, CID: B105E00D, PID: 002BB003 FPB
ROMTbl[0][3]: E0000000, CID: B105E00D, PID: 003BB001 ITM
ROMTbl[0][4]: E0040000, CID: B105900D, PID: 000BB9A1 TPIU
ROMTbl[0][5]: E0041000, CID: B105900D, PID: 000BB925 ETM
Cortex-M4 identified.
J-Link>

So the JLink probe works just fine.

The above did work, because the JLink probe is programmed to provide 5V power with the power on perm command.

Using Platformio to programm the board:

To program the board, it is necessary to specify correctly the Nordic SoftDevice and for that, the platformio.ini file needs special configuration, namely on the build_flags and build_unflags.

[env:stct_nrf52_minidev]
platform = nordicnrf52
board = stct_nrf52_minidev
framework = arduino
lib_deps = 259
build_flags = -DS132, -DNRF52_S132, -g3, -Og, -DNRF_52822_DEBUG -DENABLE_DEBUG_LOG_SUPPORT -DNRF_LOG_USES_RTT=1 -DENABLE_SWO
build_unflags = -Os, -O1, -O2, -O3, -g1
;upload_protocol = stlink
debug_tool = jlink

With this file, any example from the NRF Arduino Library compiles.

fldigi slow startup on Linux

I installed fldigi on my ne laptop computer, with Arch Linux, and I hit trouble right away: The startup of fldigi on this computer was around 130s until the start command and the ui display. During that time one of the CPU cores hit 100% and stayed there. Not acceptable.
On my desktop computer, that is much older and has at least less than half the processing power of the new laptop, fldigi started up right away and I had no issues with CPU occupation.

It took me a while to find out the cause, and with the source code available (yay for open source) I’ve found the issue at the confdialog.cxx file specifically here:

Fl_Double_Window* ConfigureDialog() {
  Fl_Double_Window* w; 
  font_browser = new Font_Browser;
  ....

A lot of the time was lost inside the call to the new Font_Browser.

So I’ve checked the fonts installed on my new computer, and low and behold: 1248 more or less fonts !!! compared to 234 installed fonts on my desktop computer.

Comparing the Arch Linux packages that I had installed on one computer and on the other, the laptop had the ttf-google-fonts-git AUR package installed that added the bulk of all the fonts.

Removing this package solved the issue with fldigi startup delay. With the fonts removed fldigi started right away as soon I execute the command to start it up.

Self hosted mobile phone push notifications

Google cloud messaging (GCM), Pushbullet, among others, allows to notify users though their mobile phone or portable device for events that where trigger either, programmatically or by any other mean.

A nice self hosted solution that does not depend in form of third party services, and as far as I can see is being actively developed, is Gotify that offers a self hosted server, and mobile application for Android.

Gotify offers a docker images (and unfortunately in the last few days they stopped producing the Docker image for Raspberry Pi), that allows to have a running Gotify server in less than 2 minutes.

Anyway, I’m still running the latest docker image for the RPI. Since I have already running something at the HTTP port 80, I’ve started the Gotify container with an alternate port:

docker run -p 1080:80 -v /var/gotify/data:/app/data gotify/server-arm7

After docker pulls the image and starts it up, we can access the Notify Web Server interface at http://rpi:1080/ and access it with the default user/password admin/admin.

The first thing that we need to do is to change the default admin password and add an application. Both steps are straight forward.

The key aspect for the application after it is created is to take notice of the application token. This token will be necessary for inject messages on Gotify and associate the injected message to the application. Also there is the possibility to associate an icon to the application, that also be shown on the mobile application, by pressing the pen icon and adding an image.

After this, to inject a message from anywhere we can use CURL:

curl -X POST "http://rpi:1080/message?token=AhV-43qgmkKSj07" -F "title=RPI" -F "message=Status Alert"
{"id":16,"appid":2,"message":"Status Alert","title":"RPI","priority":0,"date":"2019-02-25T12:13:35.777531495Z"}

Anyway, to make things short, Gotify works more or less well. I still have some issues with the web interface and with the mobile application, but those issues might be associated to being using an older version of the server.

Synology reverse proxy and websockets
Gotify uses websockets as the transport medium for the push notifications.

As a final note, since I’m using the Synology reverse proxy to access the Gotify server from the internet, some special configuration must be done so that the Synology reverse proxy let’s websockets to “pass through” to the Gotify server.

Basically we need on to go to Control Panel -> Application Portal -> Reverse Proxy and create our rule that maps out the Gotify server to the external URL. I’ve create something like http://notify.myserver.myds.me to be pointing to the internal Gotify server http://rpi:1080:

The important bit is that after creating the reverse proxy rule, we must edit and add the support for websockets by going to Custom Header and press the Create -> WebSocket button.

And that’s it. We have now a self hosted notification system. We can now, for examble, by using Node-Red or bash scripts, to start sending notifications to our clients when important events happen.

Lorawan/TTN with the TTGO ESP32 board

So I’m testing my TTGO Lora32 board with the RIOT-OS operating system to connect to the The Things Network, and while at first I had some success, then out of the blue things didn’t worked anymore.

The code that I’m using is based on the standard RIOT-Os Loramac example but with supporting code for ABP instead of the OTA activation.

The code is available at this Github repository.

Since I saw no activity on any channel for the Lora transmission by using the RTLSDR dongle, I suspected that something was up.
The Arduino based code from for connecting to TTN using Lorawan worked fine, and so it excluded an issue with the hardware.

By going to RIOT-OS installation directory, under drivers and on the sx127x directory, I’ve enabled on th sx127x.c file the debug to 1, since enabling debug on the semtech_loramac.c file held no useful information:

#define ENABLE_DEBUG (1)

With debug enabled, lo and behold, at startup:

2019-01-22 14:11:02,993 - INFO # [sx127x] error: failed to initialize SPI_0 device (code -1)
2019-01-22 14:11:02,993 - INFO # [sx127x] error: failed to initialize SPI

This is rather strange since at the initial tests the sx127x was detected and worked.
The only thing that changed is that I periodically update the RIOT local repository, and checking my board definition with other ESP32 boards that have a sx127x transceiver, the way that the pins where declared were quite different from what I had initially.

So I’ve changed the way that the pins for the SPI and the sx127x peripheral where defined, and the result:

2019-01-22 14:28:00,847 - INFO #  -> Node activation by: ABP
2019-01-22 14:28:00,847 - INFO # [sx127x] SPI_0 initialized with success
2019-01-22 14:28:00,848 - INFO # Set ABP information.

And the data arrives at TTN without any issue:

So either I had initially had the pins for SPI and the sx127x defined wrong or the latest RIOT build changed the way pins where defined.

A more complete example, working with multiple threads, is also available at this Github repository.

The ESP32 TTGO board using RIOT-OS

In the last few months, RIOT-OS operating system gained support for the ESP32 and ESP8266 based boards. RIOT-OS also supports several communication protocols, namely Lorawan through the usage of the Semtech implementation.

My post connecting the ESP32 TTGO board to TTN (The Things Network) describes (more or less) how to connect this board to the The Things Network using the Arduino and LMIC stack for the ESP32.

So why don’t give it a go with RIOT-OS?

Board definition:
RIOT-OS works with board definitions that describes several key features of the targeting board. In the case of the TTGO ESP32 Version 1 that I have several things needed to be changed from the base ESP32 board definitions that are available now on the RIOT-OS code base.

The most important issue was the clock speed definition for this ESP32 board, since it seems it uses a 26MHz clock instead of the standard(?) 40Mhz clock. Without the change for the clock speed all the serial output on the USB port was garbled and unreadable.

With the clock speed correctly set, important information regarding the pins for the I2C, SPI buses and the sx1276 chip needed to be defined. That was also correctly defined, and preliminary tests show that the Oled and sx1276 is detected using the Semtech Loramac stack.

I haven’t yet change the other pin definitions, so for the basic I2C, SPI, Button, and sx1276 are correctly defined.

The board definition can be found in my repository located at RIOT TTGO ESP32 repository under the directory boards.

To use this board definition, we just need to copy the board definition directory to the boards directory of the cloned RIOT-OS repository.

Testing:
For testing we can just compile the examples under the example directory for the RIOT TTGO ESP32 repository.

There are two examples: One for testing the OLED screen and other to test the sx1276 and the Semtech Loramac stack.
This last example is based on the provided RIOT-OS example for Lorawan, but I’ve added support for ABP and debug prints to see what is going on.
On the Make file, we should select the type of activation and fill out the required data from the TTN device console. All data just needs to be copied and pasted directly and it should work.

To build the examples I’m using Docker according to this instructions.

Additionally, for the ESP32 flashing to work, the ESP-IDF repository must be cloned and its location set for RIOT build by setting the ESP32_SDK_DIR variable.

Results:
The following things work fine:

– The serial console works fine.
– The blue led, on and off
– The button, detecting it release and pressed.

The OLED works fine through I2C, but the I2C speed must be set exactly as defined on the board definition, otherwise it won’t work. It took me several hours to debug this issue.

The sx1276 works, it is detected and the Semtech stack seems to be able to use it. Adding some debugging to Loramac stack shows that the DIO# interrupts are detected but there are issues. For example I have two boards and with the same code I have different behaviours which seems rather strange… I was able to send data to the TTN gateway, but when waiting for the Semtech stack it hangs waiting for a stack status message. The other board is always complaining regarding MAC timeout.

So, all in all, as a first experience with RIOT-OS on the TTGO board, excepting the Lorawan stack, everything seems to work as expected. Regarding the Lorawan stack, probably going to try the LMIC Arduino implementation to see if it is possible to port it to RIOT-OS.