CubicSDR and Arch Linux (Again…)

Again quick note if installing CubicSDR in Arch Linux:

Arch Linux LiquidDSP package current version on the repository is 1.3.2.

Using this version of liquiddsp or liquiddsp-git breaks CubicSDR compilation, and hence if installing CubicSDR from AUR fails:

cannot convert ‘double’ to ‘liquid_ampmodem_type’

The solution to solve this issue is to use the package manager to remove the offending 1.3.2 version and manually install the 1.3.1 version, and then also install manually the CubicSDR according to the CubicSDR install instructions..

Not ideal, but it will get things done.

Simple instructions:

pacman -R liquiddsp liquiddsp-git
mkdir liquid
cd liquid
wget https://github.com/jgaeddert/liquid-dsp/archive/v1.3.1.tar.gz
tar xvzf v1.3.1.tar.gz
cd liquid-dsp-1.3.1/
./bootstrap.sh
make -j4
sudo make install
sudo ldconfig

Until CubicSDR upgrades to use LiquidDSP version 1.3.2, this is the interim solution.

Advertisements

PZEM-004T ESP8266 software

Following up the home energy meter post based on an ESP8266 and PZEM-004T hardware, this post describes succinctly the software for using the energy meter.

There are at least two components to the solution:

  1. ESP8266 software for driving the power meter and make the measurements.
  2. The backend software for receiving and processing data.

The ESP8266 software:
The power meter software for the ESP8266 available on this GitHub repository, uses an available PZEM-004T library for accessing the power meter, and sends the collected data through MQTT to any subscribers of the power meter topic.
I’m using the convention that is also used on Thingsboard, namely an MQTT attributes topic to publish the device status, and a telemetry topic to post the data in JSON format.
Around lines 80 on main.cpp of PowerMeter sources, the topics are defined as:

  1. Attributes: “iot/device/” + String(MQTT_ClientID) + “/attributes”
  2. Telemetry: “iot/device/” + String(MQTT_ClientID) + “/telemetry”

MQTT_ClientID is defined on the secrets.h file, where we also define a list of available WIFI connections for our ESP8266. The attributes topic periodically sends the current device status (RSSI, HEAP, wifi SSID), while the data on the telemetry topic is fed into a timeseries database such as InfluxDB where then a Grafana Dashboard shows and allows to see the captured data across time.

As also my previous post regarding framework and libraries versions, I needed to block the ESP8266 framework version and the SoftwareSerial library because the combination of these with the PZE-004T library was (is ?) broken of more recent versions. As is currently defined on the platformio.ini file, the current set of versions, work fine.

A lot of people had problems working with the use of SoftwareSerial library for the PZEM library to communicate with the hardware. The issue, that I accidentally found out, are related with timing issues to communicate with the PZEM hardware. There are periods of time that the PZEM is not responsive, probably because is making some measurement.

The solution to this issue is at start up to try the connection during some time, at 3 seconds interval until it succeeds. After the connection is successful, we need to keep an interval around one minute between reads to encounter no issues/failures . If this interval is kept, the connection to the PZEM hardware works flawlessly, at least with the hardware that I have.

So the connection phase is checked and tried several times to synchronize the ESP8266 with the PZEM, and them every single minute there is a data read. If the interval is shorter, lets say, 30s, it will fail, until the elapsed time to one minute is completed.

The firmware solves the above issue, and after reading the data, it posts it to a MQTT broker. The firmware also makes available a web page with the current status and measurements:

Power Meter Web Page

Then there are other bits, namely since the meter will be on the electric mains board, an UDP logging facility that allows on the computer to run an UDP server and see what is going on.

The back-end software:
I’ve not done much on this area, since most of it is just standard stuff. An MQTT broker and Node-Red flow. The flow just receives the data, saves it into an InfluxDB database and creates a Node-Red UI dashboard.

This screenshot doesn’t show much, but shows more or less what information is available, including the current power factor.

Future work:
Basically what is missing is two things:

  1. Grafana Dashboard based on the InfluxDB data.
  2. Some kind of exporter to CSV or Spreadsheat to allow further data analysis such as the daily power consumption totals.

Docker Portainer WEB Interface on SBC (RPI, Odroid)

A quick, fast note, following up on the previous post regarding installing Portainer web interface for handling Docker operations through an web browser.

If deploying Portainer on Docker that is running either on a Raspberry PI or Odroid, the instructions are exactly the same as specified on the previous post, with the only difference being the image that is pulled: So instead of doing docker pull portainer/portainer, the correct image for ARM based SBC (Single board computers) such as the RPI or Odroid is: docker pull portainer/portainer:linux-arm. And that’s it:

$ docker volume create portainer_data
$ docker pull portainer/portainer:linux-arm
$ docker run -d -p 9000:9000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer:linux-arm

The interface is now available at the docker machine port 9000.

Measuring home energy consumption with the PZEM004T and ESP8266

First of all a very BIG WARNING: This project works with AC mains current, which, where I live, is 220V AC, meaning that extra precautions must be taken, since risk of serious injuries and/or death is possible.

The PZEM004T
The Peacefair PZEM004T device (available at the usual far east shopping web sites) is a device that can measure energy consumption by monitoring a live AC mains wire using an inductor as the measuring sensor. One of the wires that carries the current (normally the AC power phase) goes through the inductor so that the current that flows through it can be measured and hence the other measurements, including power consumption, can be also measured.

The PZEM004T can be bought with two types of inductor, one that opens up and can clip on the wire of interest, and the other type that requires to disconnect and connect the wire of interest, so that it passes through the inductor core. I’ve chosen the former, since in this case I do not need to do any disconnection/connections on the electric mains board, and so it is way safer and easier to add and remove the measurement device.

PZEM 004T

The PZEMM04T outputs the collected data through an opto-coupled isolated serial port that allows to retrieve values for voltage, current/intensity, current power consumption and energy accumulated consumption.

The device that connects to the PZEM serial port must provide power to it (5V), and so the serial port data lines are 5V level, which means that we should use a 5v to 3.3V level converter to connect to the ESP8266. While there are several hacks to make the PZEM004T serial port to use 3.3V on the serial port, and hence have 3.3V data lines, I just used a simple level converter to connect the serial port to the ESP8266, and avoid in this way any modifications to the PZEM-004T. The serial port connector is a 4 Pin JST-XH connector.

So the basic schematics for using the PZEM004T is as simple as the following highly professionally drawn schematic shows:

PZEM004T And Wemos D1 connection schematic

Two things of notice:

  • The Ground connection – The serial port uses the same ground as the Wemos D1.
  • The Power supply – Wemos D1 is powered through the 5V pin, NOT through the 3.3v pin, since we need 5V to power up the PZEM serial port.

The level converter is just a simple, cheap I2C level converter, used in this case to level convert the serial data lines.
Also the above schematic shows that the TX and RX pins connect to the Wemos D6 and D5 pins, since I’ll be using software serial, but the depicted connections are just an example, since the pins to be used can be software defined.
In my code I use the connections the other way around ( D5-TX, D6-RX) so beware to how the pins are connected and how they are defined at the software level.

Powering up the ESP8266 Wemos D1
I’ll be using the Wemos D1 ESP8266 based boards, as we can see on the above schematics (associated to a prototype shield to solder the connections and the level converter), we need to power it up using 5V. The ESP8266 uses 3.3v, but the Wemos board has a 5V input and a 5V to 3.3V converter, so no issues there. The PZEM004T on the other hand uses 220V, and since the ESP8266 will be near the PZEM004T, it makes sense to get the 5V CC from the 220V AC to power up the Wemos D1 board.

The 220V AC to 5V CC can be achieved in several ways, and since I’ll be installing all this in a DIN case on my home electricity mains board, the easiest solution is just to buy a 5V output 220V based DIN power supply for around 10/15€. This is the easiest and safest solution.

There are other solutions, including the one that I’m using that is based on the 5V HLK-PM01 based modules. This requires some assembly and also be aware that there are fake HLK modules around.

Do not connect the HLK-PM01 without the associated protection components, namely fuses, VDR, and the most important component the thermal fuse of 72ºC (Celsius!) that will cut off the power to anything after it (including the VDR) if the temperature of the HLK module or it’s surroundings rises above the 72ºC temperature. I’ve not soldered the thermal fuse, since the heat from the soldering iron can destroy it, just used a two terminal with screws to connect it.

The schematic used is the following one:

5V Power supply

The PZEM-004T, the HLK based power supply and the Wemos D1 ESP8266 module are inside a double length project DIN case so that all components can be safely installed on the mains electricity board.

Since all is self contained on the DIN case, all is needed is to clip the inductor on the main phase wire entering the mains board (and it is easy since the inductor is an open clip on type), and connect the components to the 220V AC power. I’ve derived the power from one of the circuit breakers that already protects a house circuit, which adds an additional layer of protection.

The software
On the next post I’ll discuss the software for driving the ESP8266 to gather data from the PZEM004T and how it works.
The firmware for driving this is already available at: https://github.com/fcgdam/PowerMeter

MQTT-SN Paho gateway and UDP6

Sometimes simple things can take a lot of time, such as this: How to make the Paho MQTT-SN gateway to listen to UDP6 instead of the standard UDP4 ports…

Introduction:
A lot can be written about MQTT-SN and it’s counterpart MQTT. To keep it simple and short, MQTT-SN (SN for Sensor Networks) uses UDP instead of TCP/IP and is streamlined to be more efficient in transmitting information, and so more adequate to be used on constrained devices than MQTT on WSN (Wireless Sensor Networks).
One important key element in using MQTT-SN is that MQTT-SN clients use UDP broadcasts to find gateways, and so they have no need to hardcode a broker address on the firmware code, since gateways can be dynamically discover at run time. Still MQTT-SN depends on MQTT broker to connect to the outside world (and that’s why MQTT-SN is a gateway, not a broker), but that is configured at the gateway level, not at the node level.

Compiling and configuring:
If we just clone the MQTT-SN Github repository, we can for UDP version 4, just do the following:

git clone https://github.com/eclipse/paho.mqtt-sn.embedded-c
cd MQTTSNGateway/
make 

And that’s it. On the Build/ directory there are the executables, among them the MQTT-SNGateway, and on the current directory there is the configuration file gateway.conf that is used by the gateway to startup. Out of the box the only thing that we need to change is the upstream MQTT broker address that by default points to the iot.eclipse.org MQTT broker, the gateway name, and maybe the ports:

BrokerName=iot.eclipse.org
BrokerPortNo=1883
BrokerSecurePortNo=8883

GatewayID=1
GatewayName=PahoGateway-01
KeepAlive=900

# UDP
GatewayPortNo=10000
MulticastIP=225.1.1.1
MulticastPortNo=1883

So if we start with this config, the output is:

20190612 142837.189 PahoGateway-01 has been started.

 ConfigFile: ./gateway.conf
 PreDefFile: ./predefinedTopic.conf
 SensorN/W:  UDP Multicast 225.1.1.1:1883 Gateway Port 10000
 Broker:     172.17.0.4 : 1883, 8883
 RootCApath: (null)
 RootCAfile: (null)
 CertKey:    (null)
 PrivateKey: (null)

We can see that in this case my upstream broker is at 172.17.0.4 (a docker image of the Mosquitto broker).

Checking which ports the gateway is listening:

root@b74f5ad3fd8e:/app/mqttsn# lsof -c MQTT-SNGateway -a -i4
COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
MQTT-SNGa 923 root    3u  IPv4 5017719      0t0  UDP *:10000 
MQTT-SNGa 923 root    4u  IPv4 5017720      0t0  UDP *:1883 
root@b74f5ad3fd8e:/app/mqttsn# lsof -c MQTT-SNGateway -a -i6
root@b74f5ad3fd8e:/app/mqttsn# 

We can see that the gateway is listening on UDP4 and not on UDP6.

Supporting UDP6:
Sensor networks such as Openthread networks work only with IPv6… and so the MQTT-SN gateway must listen to UDP6 instead of UDP4. It can still communicate with the MQTT broker over IPv4, and hence bridging the IPv4 network with the IPv6 sensor network.

To add support to UDP6 for the MQTT-SN gateway we need to change the Makefile (and hence the time to find information for this little bit… ) to change the protocol used by the sensor network and recompile.

So again on the source folder for the MQTT-SN gateway, edit the Makefile and change the following line from

SENSORNET := udp

to

SENSORNET := udp6

Or just change the parameter at the make call: make SENSORNET=udp6

And then clean and build:

make clean
make SENSORNET=udp6

We have now a MQTT-SN gateway version for UDP6. We need now to add the configuration entries to the gateway configuration file so that the UDP6 configuration is set:

#UDP v6
GatewayUDP6Broadcast = ff03::1
GatewayUDP6Port = 47193
GatewayUDP6If = wpan0

We can now start the UDP6 MQTT-SN enabled version:

20190612 144124.280 PahoGateway-01 has been started.

 ConfigFile: ./gateway.conf
 PreDefFile: ./predefinedTopic.conf
 SensorN/W:   Gateway Port: 47193 Broadcast Address: ff03::1 Interface: wpan0
 Broker:     172.17.0.4 : 1883, 8883
 RootCApath: (null)
 RootCAfile: (null)
 CertKey:    (null)
 PrivateKey: (null)

Specifically in this case the GatewayUDP6If interface that is defined is the NCP OpenThread interface, but it should be the interface where we want the gateway to listen. The broadcast address is the address defined on the Nordic SDK for Thread, namely on the file mqttsn_packet_internal.h.

Checking the ports:

root@b74f5ad3fd8e:/app/mqttsn# lsof -c MQTT-SNGateway -a -i4
root@b74f5ad3fd8e:/app/mqttsn# lsof -c MQTT-SNGateway -a -i6
COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
MQTT-SNGa 947 root    4u  IPv6 5050130      0t0  UDP *:47193 
root@b74f5ad3fd8e:/app/mqttsn# 

And that’s it.

Conclusion:
A simple lack of documentation regarding MQTT-SN and UDP6 took a while to solve since it was so “simple”.

Still one still needs two compiled versions of MQTT-SN for supporting UDP4 and UDP6 based networks, but since MQTT-SN is light there is no big issue with this.

When updating breaks projects…

A quick post regarding updating platforms and libraries for projects, specifically projects for the ESP8266 platforms:

The PZEM004T is device available on eBay and other sites, that allows to measure energy consumption. The PZEM004T has a serial output port and when connecting it to, for example, an ESP8266, we can access the collected data trough WIFI and process it for finding out how much electricity we are using, and so on.

Anyway, when using the ESP8266 WEMOS D1 mini, to be able to still use the USB serial port, we need to use Software serial emulation. In fact the Arduino PZEM004T library available on Github and on the Platformio library registry allows the use of the Software Serial to communicate with the PZEM004T (and it works just fine).

So what is the issue?

I’m using Platformio to develop the ESP8266 application, and normally when running, it checks and offers any updates that might be available. So, I’ve updated to the latest Espressif ESP8266 platform and ESPSoftwareSerial, and then everything just break down:

  • ESPSoftwareSerial last version just completely breaks the previous existing API which made the PZEM004T library also broken.
  • The new ESP8266 platform removes Esp8266 a SDK attachInterruptArg function which renders the ESPSoftwareSerial library unbuildable

The solution?

The solution with Platformio is quite easy: use semantic versioning.

In fact something like this on the platformio.ini file:

[env:d1_mini]
platform = espressif8266
...
...

can be locked to a working previous version:

[env:d1_mini]
platform = espressif8266@2.0.3
...
...

The same can be done with the project libraries. While the ESPSoftwareSerial (the PZEM004T dependency loads it) does not need to be defined, specifying it allows to use a specific version:

[env:d1_mini]
platform = espressif8266@2.0.3
board = d1_mini_lite
framework = arduino
upload_speed = 921600
monitor_speed = 115200

lib_deps =  ESPSoftwareSerial@5.0.3
            PZEM004T
            MQTT
            LiquidCrystal_I2C
            SimpleTimer
            ESPAsyncTCP
            ESP Async WebServer
            Time

And with this, the ESP8266 platform and ESPSoftwareSerial versions locked, the issues with the newer versions are avoided, and the code compiles and works as it should.

So, updating is fine, but when it breaks it can be an issue. Fortunately Platformio allows the usage of specific version for building our projects, and even allows to deploy our specific library version under the project lib directory.

Have fun!

Programming the Taida Century NRF52832 developer board

Its been a long hiatus regarding this post. At the time I’ve bought a generic eBay/Aliexpress nRF52832 development board (the Taida Century NRF52832 based board) for performing some BLE testing, did some development with Nordic SDK and the NRF Arduino firmware using Platformio, but posted nothing on my blog regarding the tests.
The main trigger for picking up again this board and do some writing about it, is that I’m evaluating RIOT-OS and its capability to target several hardware platforms, namely the NRF52 based chips. Anyway, first the use of the Arduino framework for the NRF52 and platformio:

Programming the NRF52832:
For programming the Taida Century NRF52 board need a programmer supporting the NRF52 chip, that can be either a SEGGER JLink programmer probe (highly recommended), or with a STLink V2 coupled with modified Openocd software, that also does work fine (Recent versions of Openocd already support the NRF52 out of the box). The Blackmagic probe is also an alternative, but since I have now a Jlink probe, never bother again with it.

The hardware connections for programming and debugging:
Since that the board does not have a serial to USB chip and a USB connection, an external TTL to USB serial convert must be used to see any serial output on the computer from the Serial.println commands. Furthermore the USB serial adapter must also must work with the 3.3V used by the NRF52. The 3.3v power can be provided either by the USB serial converter or by the debug probe (but not both at the same time!). I normally use a 6 pin CP2102 USB serial adapters: they work with 3.3v and have all the serial pins exposed.

The Arduino framework for the NRF52 board and specifically for the Taida board variant uses the following pins for serial communication:

RXD – Pin 18
TXD – Pin 19
GND – GND

So the TXD labelled pin on the USB adapter connects to the Pin 18 labelled header on the board (RXD), and the RXD pin on the USB adapter, connects to Pin 19 on the pin header on the board. Ground connects to ground. That settles the serial connection for debugging. The ground pin should also be connected either on the header pin labelled GND on the top left when the IDC connector is on the left, or on the IDC connector itself.

The connections for programming are a bit tricker depending if we are using an STLink programmer or a Segger Jlink programmer. In both cases the power can be provided by the programmer, but with the JLink one, a special connection is also needed so that the SWD debug lines voltage reference used is 3.3v instead of 5v.

Connecting the ST-Link programmer:
Since we are powering up the board from the programmer, and using one of those cheap st-link usb dongles, only four connections are needed:

The USB programmer has a 3.3v output and so can power up the board directly to the 3.3v board pin.

Connecting the JLink probe:
The Jlink probe can only provide 5V output on the JTAG pins, and so while the JTAG pins TCK and TMS can be connected to the TCK and TMS pins on the JTAG connector, the 5V pin must be connected to the VBUS pin on the IDC connector. This pin will provide power to the board and the internal regulator drops it down from 5V to 3.3V. To adjust the logic levels from the probe to 3.3V the JTAG pin VREF must be connected to the IDC pin 3.3V pin, that informs the JLink probe the device target working voltage.
So in total one more connection that with the ST-Link programmer.

The connection diagram is as follows, where the Taida board 12 pin IDC connector is used:

After the connection is done, we can test if we can communicate with the board using the JLinkExe:

JLinkExe -if SWD -device NRF52 -speed 4000
SEGGER J-Link Commander V6.44 (Compiled Mar  1 2019 17:36:52)
DLL version V6.44, compiled Mar  1 2019 17:36:42

Connecting to J-Link via USB...O.K.
Firmware: J-Link V10 compiled Mar  1 2019 16:57:49
Hardware version: V10.10
S/N: XXXXXXXXX
License(s): FlashBP, GDB
OEM: SEGGER-EDU
VTref=3.264V

Type "connect" to establish a target connection, '?' for help
J-Link>connect
Device "NRF52" selected.

Connecting to target via SWD
Found SW-DP with ID 0x2BA03466
Found SW-DP with ID 0x2BA03466
Scanning AP map to find all available APs
AP[2]: Stopped AP scan as end of AP map has been reached
AP[0]: AHB-AP (IDR: 0x24770011)
AP[1]: JTAG-AP (IDR: 0x02880000)
Iterating through AP map to find AHB-AP to use
AP[0]: Core found
AP[0]: AHB-AP ROM base: 0xE00FF000
CPUID register: 0x410FC241. Implementer code: 0x41 (ARM)
Found Cortex-M4 r0p1, Little endian.
FPUnit: 6 code (BP) slots and 2 literal slots
CoreSight components:
ROMTbl[0] @ E00FF000
ROMTbl[0][0]: E000E000, CID: B105E00D, PID: 000BB00C SCS-M7
ROMTbl[0][1]: E0001000, CID: B105E00D, PID: 003BB002 DWT
ROMTbl[0][2]: E0002000, CID: B105E00D, PID: 002BB003 FPB
ROMTbl[0][3]: E0000000, CID: B105E00D, PID: 003BB001 ITM
ROMTbl[0][4]: E0040000, CID: B105900D, PID: 000BB9A1 TPIU
ROMTbl[0][5]: E0041000, CID: B105900D, PID: 000BB925 ETM
Cortex-M4 identified.
J-Link>

So the JLink probe works just fine.

The above did work, because the JLink probe is programmed to provide 5V power with the power on perm command.

Using Platformio to programm the board:

To program the board, it is necessary to specify correctly the Nordic SoftDevice and for that, the platformio.ini file needs special configuration, namely on the build_flags and build_unflags.

[env:stct_nrf52_minidev]
platform = nordicnrf52
board = stct_nrf52_minidev
framework = arduino
lib_deps = 259
build_flags = -DS132, -DNRF52_S132, -g3, -Og, -DNRF_52822_DEBUG -DENABLE_DEBUG_LOG_SUPPORT -DNRF_LOG_USES_RTT=1 -DENABLE_SWO
build_unflags = -Os, -O1, -O2, -O3, -g1
;upload_protocol = stlink
debug_tool = jlink

With this file, any example from the NRF Arduino Library compiles.