ESP8266, NodeMcu and MQTT: Event subscription load testing

I’ve being doing some testing with the latest NodeMcu regarding MQTT support. My experience is, more or less, documented here: https://primalcortex.wordpress.com/2015/02/06/nodemcu-and-mqtt-how-to-start/ .

I wanted to check if concurrent subscription seems to work fine, since I know that publishing is not straightforward. In the code from the above link, the esp8266 subscribes to four topics, named topic1, topic2 and so on.

This simple bash script, calls repeatedly the mosquitto broker publishing command:

#!/bin/bash
for i in {1..60}
do
/home/pcortex/01.Develop/mosquitto/mosquitto-1.3.5/client/mosquitto_pub -h 192.168.1.16 -m "Mensagem $i" -t topic1
/home/pcortex/01.Develop/mosquitto/mosquitto-1.3.5/client/mosquitto_pub -h 192.168.1.16 -m "Mensagem $i" -t topic2
done

But with low values on the loop count, the MQTT code on the esp8266 seems to work fine, but with higher values, as above (60), the esp8266 hangs and reboots. One interesting thing was that after some testing not even doing a reset or completely powering it off, I was able to bring the esp8266 back online. It just put out garbage on the serial output and that was it. Strange indeed.

So I’ve reflash it again with the latest NodeMcu version (lots of reflashing hey :) ), but still the esp8266 did not recover! Only garbage out when connecting to the serial port (And no, it wasn’t the serial port rate issue). So I’ve reflashed again the original AT Firmware, and it came back to life again, and then reflash it again with the latest Nodemcu version.

Adding a delay on the bash script, before the done line, like sleep 1 seems to keep the esp8266 running without issue.

So indeed this shows, that bombarding the esp8266 with MQTT messages in burst mode is not a good idea…

Nodemcu and MQTT: How to start

The latest nodemcu version has MQTT protocol support already built in. MQTT is one of the supporting protocols for building IoT, Internet of Things devices,

So I’ve done some tests using the new MQTT functions, with code based from this thread on ESP8266 forum http://www.esp8266.com/viewtopic.php?f=19&t=1278&start=40 ( My code is based on the helix user code. Thanks!). I’ve made some changes to make things work in a more robust way, that I’m documenting the on this post.

My initial tests where done with nodemcu version 20150123 but I’ll write this post based on version NodeMCU 0.9.5 build 20150127  so if your are using an older or newer version of the firmware, the code on this post can behave a bit different of what is expected. Also I’m using a MQTT broker named RSMB ( Real small message broker) running in my Synology NAS, but we can run any MQTT broker. Also for testing I’m using the great MQTT-SPY program.

The init.lua code that I use is as documented in this post: https://primalcortex.wordpress.com/2014/12/30/esp8266-nodemcu-and-lua-language-and-some-arduino-issues/ . That code makes sure that an WiFi connection is made before continuing.  The only difference is that on the init.lua code, namely the launch function, if a successful connection is made, instead of calling a ping.lua file repeatedly, a mqtt.lua file is called once. To do this just change the CMDFILE to be CMDFILE = “mqtt.lua”. Note that the timer is one shot now. It calls the mqtt.lua file and that’s it. Then the mqtt.lua file takes control.

CMDFILE = "mqtt.lua"   -- File that is executed after connection
-- Change the code of this function that it calls your code.
function launch()
  print("Connected to WIFI!")
  print("IP Address: " .. wifi.sta.getip())
  tmr.alarm(0, 1000, 0, function() dofile(CMDFILE) end )  -- Zero as third parameter. Call once the file.
end  -- !!! Increase the delay to like 10s if developing mqtt.lua file otherwise firmware reboot loops can happen

The MQTT API is used by basically creating an MQTT Client with mqtt.Client(), then connecting, and finally we can subscribe or post to topics. Subscriptions will call a callback function when data is received, and Publishing will also call a callback function when data is sent.

It also seems that only at one message at the same time can be publish, otherwise we get an error like this unprotected error in call to Lua API (pir.lua:34: sending in process) and the module reboots. I’ve solved the issue on a first approach by using a semaphore that blocks a new call before the older one is finished.

The following code uses two different functions to publish to a topic, do to the use of the publish semaphore. Please read  the code comments to follow the way the code works:

-- Configuration to connect to the MQTT broker.
BROKER = "192.168.1.16"   -- Ip/hostname of MQTT broker
BRPORT = 1883             -- MQTT broker port
BRUSER = "user"           -- If MQTT authenitcation is used then define the user
BRPWD  = "pwd"            -- The above user password
CLIENTID = "ESP8266-" ..  node.chipid() -- The MQTT ID. Change to something you like

-- MQTT topics to subscribe
topics = {"topic1","topic2","topic3","topic4"} -- Add/remove topics to the array

-- Control variables.
pub_sem = 0         -- MQTT Publish semaphore. Stops the publishing whne the previous hasn't ended
current_topic  = 1  -- variable for one currently being subscribed to
topicsub_delay = 50 -- microseconds between subscription attempts, worked for me (local network) down to 5...YMMV
id1 = 0
id2 = 0

-- connect to the broker
print "Connecting to MQTT broker. Please wait..."
m = mqtt.Client( CLIENTID, 120, BRUSER, BRPWD)
m:connect( BROKER , BRPORT, 0, function(conn)
     print("Connected to MQTT:" .. BROKER .. ":" .. BRPORT .." as " .. CLIENTID )
     mqtt_sub() --run the subscription function
end)

function mqtt_sub()
     if table.getn(topics) < current_topic then
          -- if we have subscribed to all topics in the array, run the main prog
          run_main_prog()
     else
          --subscribe to the topic
          m:subscribe(topics[current_topic] , 0, function(conn)
               print("Subscribing topic: " .. topics[current_topic - 1] )
          end)
          current_topic = current_topic + 1  -- Goto next topic
          --set the timer to rerun the loop as long there is topics to subscribe
          tmr.alarm(5, topicsub_delay, 0, mqtt_sub )
     end
end

-- Sample publish functions:
function publish_data1()
   if pub_sem == 0 then  -- Is the semaphore set=
     pub_sem = 1  -- Nop. Let's block it
     m:publish("temperature","hello",0,0, function(conn) 
        -- Callback function. We've sent the data
        print("Sending data1: " .. id1)
        pub_sem = 0  -- Unblock the semaphore
        id1 = id1 +1 -- Let's increase our counter
     end)
   end  
end

function publish_data2()
   if pub_sem == 0 then
     pub_sem = 1
     m:publish("ts","hello",0,0, function(conn) 
        print("Sending data2: " .. id2)
        pub_sem = 0
        id2 = id2 + 1
     end)
   end  
end

--main program to run after the subscriptions are done
function run_main_prog()
     print("Main program")
     
     tmr.alarm(2, 5000, 1, publish_data1 )
     tmr.alarm(3, 6000, 1, publish_data2 )
     -- Callback to receive the subscribed topic messages. 
     m:on("message", function(conn, topic, data)
        print(topic .. ":" )
        if (data ~= nil ) then
          print ( data )
        end
      end )
end

This seems to work fine for some time, but then, even with some free heap space, it stops working.  No idea why. Even the subscriptions fail to work, and at the broker I get a ending connection event due to time-out.

20150206 213132.389 CWNAN0033I Connection attempt to listener 1883 received from client ESP8266-10350678 on address 192.168.1.82:32456 
20150206 214010.033 CWNAN0024I 120 second keepalive timeout for client ESP8266-10350678, ending connection

When at this stage that seems locked the semaphore is “red”: pub_sem = 1…

Another interesting notes: if at the main function the timer 2 and 3 have the same interval,  publish_data2 is never called. For example:

     tmr.alarm(2, 5000, 1, publish_data1 )
     tmr.alarm(3, 5000, 1, publish_data2 )

Interesting… probably both timers are triggered at the same time and only the first one is honoured.

Second the way this is done, it means that for every topic a publish_data function must be built, and this will increase the semaphore demand.

I’ll keep this post short and end it here, but I’ll write a follow up post where the publish_data function is unified for any topic, and, I hope, we will be using queues for queueing the messages to be published.

NodeMcu and Heap utilization: Great improvements

Since I saw that the MQTT client was ported into the NodeMCU firmware that allows the ESP8266 to be programmed in the Lua language, and knowing the Heap issues that this firmware has had in the past, I gave it a again a try to see how things have improved.

As we can see in the bellow graph, things, are not totally solved but it’s waaaay better…

Selection_052

This means that, at least for my sample code, things are quite stable now. Not a single reboot since the 24th of January. Not bad. A new stability milestone has been achieved making the NodeMcu firmware a firm contender to building applications on the esp8266 without the steep native SDK learning curve.

Upgrading to an SSD, moving from Kubuntu to Arch Linux and setting up KDE

This is a single post just to make notice of some points:

My home desktop PC, was running since 2009 the Kubuntu distribution. I’ve never reinstalled it and did always the upgrades from 8.10 to 9.04, 9.10 and so on until 14.04. Almost six (!) years with no virus, trojan horses, and a single initial installation. I’ve upgraded through the years the KDE desktop since the ill famed initial 4.0 release, to the latest 4.6 release.

But things where starting to go awry… At the beginning I had a ATI 4870 graphic board, but ATI dropped support for it on Linux (and Windows) and since I had so many troubles with the proprietary driver, and to add it up I had no support for the board, I sold it and bought an Nvidia GTX660.

But with both graphic boards, I had suffered from a long time problem: I could only do a sleep/resume a number of times before the system hanged. The symptoms starting from loosing network connectivity and then restoring right back, and some cycling of these symptoms to a complete lockup. At first I thought that was due to the network driver used by my motherboard, a Asus P6T with  a Realtek RTL8111/8168/8411 board, but with the correct driver for it, the issue continued.  Booting up was a painful two/three minute wait from BIOS to desktop.

But after a while I could pin point my issues not to the network board but to the Xorg and video driver combination. Some EQ buffer overflow, hanged X, and finally the machine.

Ubuntu and Kubuntu based distributions have PPA’s repositories where some alternative software and versions of mainstream software are available. I had trouble with some of these alternative PPA’s when upgrading, and had to remove them (with ppa-purge) to be able to upgrade the Kubuntu distribution. Anyway, there is an PPA, xorg-edgers,  where alternative and more recent versions of Xorg Software and Nvidia drivers are provided and that almost solved my sleep/resume problem. The problem was that after adding this PPA the Ubuntu/Kubuntu nvidia driver and the Xorg-edgers driver, after updates the X server was unable to find the nvidia driver, despite being installed, and had to reinstall the driver again to get my desktop working.

That was it. I had enough with Ubuntu/Kubuntu distributions. My bad and my issues, sure…

I’ve being testing Arch Linux on my work laptop for almost an year with great success. I’ve chosen Arch Linux because I didn’t had to loose an unknown number of hours per year upgrading to the next distribution version, and an rolling release version makes more sense. And Arch has the greatest Linux Wiki out there even solving problems that happen on the Ubuntu/Kubuntu line, and also mainly there are only one (well two) repositories. No PPA’s mess.

So I completely moved all my systems to Arch Linux, and my sleep/resume issue is 100% solved.

SSD on a SATA II based motherboard

My desktop computer has an Asus P6T motherboard. This motherboard only has SATA II (SATA 2) ports. Is it worth to use an SSD on such a board without upgrading to a SATA III board?

The answer is YES. If you have a SATA II based motherboard AND not buying an SSD because of that, do not wait any more. Buy it now. It is totally worth it.

Arch Linux is of course way lighter than Kubuntu, and with an SSD ( by the way it’s a Crucial MX100 256GB) my two/three minute boot to desktop time came down to 15s…

hdparm -Tt /dev/sdc
/dev/sdc:
Timing cached reads:   16710 MB in  2.00 seconds = 8361.40 MB/sec
Timing buffered disk reads: 728 MB in  3.00 seconds = 242.59 MB/sec

Note the Samsung EVO 120GB that I have on my work computer, that has the flawed firmware that slows down with time, on a SATA III based motherboard:

/dev/sda:
Timing cached reads: 23864 MB in 2.00 seconds = 11943.41 MB/sec
Timing buffered disk reads: 164 MB in 3.01 seconds = 54.47 MB/sec

Compare this with my desktop old boot disk, an WD 640GB blue:
/dev/sdb:
Timing cached reads: 16580 MB in 2.00 seconds = 8296.87 MB/sec
Timing buffered disk reads: 356 MB in 3.00 seconds = 118.59 MB/sec

I hadn’t yet had the time to upgrade the firmware on the EVO…

Making KDE looking good on Arch Linux

To end this rather long and tedious post, and also to my future reference, to make KDE looking good:

    • Make sure that the infinality modifications are installed: https://wiki.archlinux.org/index.php/Infinality. Make sure that you update.
    • Make sure that the DPI detected by your X server are correct for your monitor. Install xdpyinfo with pacman -S xorg-xdpyinfo. Figure out what might be your monitor dpi using this site: http://dpi.lv/ . Execute xdpyinfo | grep -B2 resolution and see if are similar. If not you need to set the correct dpi either in xorg.conf or in the Kde settings.
      In my case I have a dual monitor setup, and hence the weird dpi 95×96.screen

      #0: dimensions: 3200×1080 pixels (856×286 millimeters)

      resolution: 95×96 dots per inch

      So my font settings are:

Selection_002

  • While the above settings make my fonts look good, the taskbar fonts where awful and took me a while to figure out that was not the dpi settings but the Desktop Theme.
  • To solve this first install Krita with pacman -S calligra-krita . This will install the Krita color themes that are, in my opinion very good. And on KDE System Settings -> Workspace Appearance -> Desktop Themes select Get New Themes and get the AndroidBit theme.
  • That’s it:Selection_004
  • I’ve selected the Produkte Desktop Theme
  • And the colors:Selection_005

1.8 SPI TFT display for Arduino

If you need to buy a display to connect to Arduino (or other boards as the Raspberry PI) don’t waste your money on the Nokia 5110 display. While these Nokia displays are quite cheap, around 2.5€, they are monochrome, and suffer from an assembly issue (at least mine) that the display contacts to the supporting board become loose (it uses some kind of adhesive) and the display looses contrast. Pressing the display against the supporting board makes it ok, but only for a while. The only valid reason that I can see for using one of these displays, if you need one, is memory size available for your program, since driving it is quite easy.

So, for a bit more money, and at less than 4€, we can buy these gorgeous 1.8 SPI TFT display. It is a world apart from the Nokia 5110 display. It has a higher resolution, it has color, it is bright, it is much better.

I’ve bought mine from eBay but there are several versions of this display regarding the controller chip. These versions from the exterior are virtually indistinguishable. The eBay seller of mine, shows a QDTech picture and also provided a Arduino library, that off course didn’t work…

Anyway , to use this display:

Connect the VCC and BL (backlight) pin to your power source (in my case, 5V), and connect the GND to ground.

Connect the RST pin to the RST Arduino Pin directly.

The other data lines should be connected to Arduino through 1K resistors.

We can drive the display by using any pin from Arduino, but to have the best performance we should use the hardware SPI pins 13 and 11.

Note that the pins names on the TFT board can change, depending in what version was shipped, but in my case was:

  • TFT SCLK -> Arduino Pin 13  (Serial Clock)
  • TFT DIN -> Arduino Pin 11 (Data)
  • TFT D/C -> Arduino Pin 8 (Data/Command)
  • TFT CS -> Arduino Pin 10 (Chip Select)

All these pins are connected in serial with an 1K resistor.

After the connection it took a while to find out the correct library to use to drive my TFT version.

The eBay description provided the QDTech driver and UTFT library which didn’t work. In fact the UTFT library was too big that I couldn’t even upload the code to my Arduino Uno without cutting out some of the sample code.

In my case the correct library for my display was the Adafruit ST7735 library available here: https://github.com/adafruit/Adafruit-ST7735-Library  But this doesn’t mean that it is the correct library for your display. It seems there are some different chips for this display like the above ST7735, the S6D02A1 and the QDTech. We need to test several of the libraries to get the right version.

esp8266: Setting up Netbeans to compile to the esp8266

A quick post in how to setup Netbeans 8 to be able to program native C/C++ programs for the esp8266. Most of the people use Eclipse, but sorry, I’m a Netbeans guy :)

1st) Start up Netbeans and go to the menu Tools->Options

2nd) Select C/C++. You might have to install the C/C++ plugin if you haven’t done it yet or if you don’t have a full installation of Netbeans.
Selection_003

3rd) Select the Add button bellow Tool Collection list, and create a new Compiler tools for the esp8266:

Selection_004

Make sure that Tool Collection Family is GNU and give it a cool name. As we can see the tool directory is where the XTensa compiler tools are installed. Please note that I’ve installed the C++ version of tools that have the C and C++ compiler.

Press Apply.

4th) Add the complete path to the C and C++ compiler:
Selection_005

5th) Check configuration. Press the Versions button to show the current versions of the Xtensa compiler and check out the Code Assistance tab to see that the files are automatically defined:

Selection_006

Press Apply and OK and we are ready.

Example: Importing and compiling esp_mqtt project:

Select File->New Project and choose C/C++ Project with existing sources:

Selection_007

Press Next and select the directory where the project files are. (I’ve already have git “cloned” them).

Selection_008

Just make sure that the Tools collection selected is now the previously defined tool set: ESP8266_XTensa.

The new project will run make clean and make automatically and should compile things just fine:

Selection_009

Success!

ESP8266 NodeMCU: Heap issues

The latest version of NodeMCU firmware (8/9 Jan 2015) is way better in providing more free heap at startup. Now it starts up with around 19/20K instead of around 15/16K.

But unfortunately it seems far more unstable than the previous versions regarding usage, and as a consequence it reboots more often:

Selection_001

Before the end of 8th of January, heap usage decreased as time went by and had at the end a reboot.

Now it just “flaps” out:
Selection_002

Because I only own at the moment a single esp-01 board, I can’t keep track of NodeMcu heap management improvements, and moved out building things with the native SDK.

I’ll try this firmware again when some improvements are made.