Sming – MQTT (part 2)

While doing some test with MQTT using the Sming Framework for the ESP8266 I had some issues that are worthy to be documented.
The Sming Framework uses the libemqtt library for MQTT support, and it seems it works fine. Connection, subscribing and publishing work out of the box, and one important thing, the keep alive mechanism also works. We can see how MQTT can be used with the Sming framework in here.

But what is this keep alive mechanism on the MQTT protocol? The MQTT protocol specification defines this:

It is the responsibility of the Client to ensure that the interval between Control Packets being sent does not exceed the Keep Alive value. In the absence of sending any other Control Packets, the Client MUST send a PINGREQ Packet.

This means that before a keep alive time is reached the client must do a request to the MQTT broker, and in case it doesn’t, it should send a PINGREQ.

For example, on a Mosquitto broker with logs enabled I can have this:

1450189246: New client connected from 192.168.1.200 as ESP8266 (c1, k10)

It shows that a client connected from 192.168.1.200 with the client identification ESP8266 and it defined as the keep alive interval 10s.

This means that at least before a ten second period is reached, the esp8266 MQTT client should send a PINGREQ or do something, like publishing to a topic.

The above requirements are probably the root of my problems and the fact that the ESP8266 chip/firmware can’t do too many things at once….

The test that I was doing was that I didn’t published nothing from the ESP8266 to the MQTT broker. All I was doing was to subscribe some MQTT topic and print the messages on the I2C 16×02 LCD that I have connected to the ESP8266.

Without any activity the esp8266 stayed connected to broker without any problems.

If I published something to the topic subscribed by the esp8266, it worked fine.

But if I published something massively to the topic subscribed, then the broker would drop the connection:

1450190071: Socket error on client ESP8266, disconnecting.

And that was a bit odd…

So at a first approach, I added a check timer to see if the MQTT connection was valid, and to reconnect if it wasn’t:

// Connect to the MQTT broker.
// If BRUSER is not empty, connect with user and password.
void startMqttClient()
{
    Serial.println("Mqtt server connect & subscribe..");   
    String mqttId = "esp8266-" + String(system_get_chip_id());

    if ( strlen( BRUSER ) == 0 )   
        mqtt.connect( mqttId );
    else
        mqtt.connect( mqttId, BRUSER , BRPWD );
    
    // Let's subscribe topics    
    mqtt.subscribe("/data");
    
}

void checkMQTT() {
        if (mqtt.getConnectionState() != eTCS_Connected) {
            Serial.println("MQTT Server: Lost connection... ");
            Serial.println("MQTT Server: Reconnect");
            startMqttClient(); // Auto reconnect
        }
}

And the timer that is defined to check the MQTT connection:

Timer checkTimer;

void startMain() {
  //...
  //...
  
  startMqttClient();  // Connect to the MQTT broker.
  checkTimer.initializeMs( 60 * 1000 , checkMQTT).start();
}

So every minute the connection to the broker is checked, and the reconnection, if needed, happens automatically.

But this solution doesn’t explain why the connection is lost when the burst of messages arrive at the topic.

Anyway to keep the long story short, the burst duration of messages where superior to the keep alive interval. Since that during that period I didn’t publish nothing, AND, the Mqtt library was busy to receive the incoming data burst, the broker assumed that the esp8266 was dead, and drop the connection.

Since the default keep alive was 10s, that value was too short to keep the connection stable under load.

The real solution is to increase the MQTT keep alive interval, to something that is bigger that any duration of probable data bursts. So what is needed is to specify a higher keep alive interval:

void startMqttClient()
{
    Serial.println("Mqtt server connect & subscribe..");   
    String mqttId = "esp8266-" + String(system_get_chip_id());

    mqtt.setKeepAlive(180);  // Keep alive of 180 seconds...

    if ( strlen( BRUSER ) == 0 )   
        mqtt.connect( mqttId );
    else
        mqtt.connect( mqttId, BRUSER , BRPWD );
    
    // Let's subscribe topics    
    mqtt.subscribe("/data");
    
} 

And with this change we got at the broker:

1450190936: New client connected from 192.168.1.200 as ESP8266 (c1, k180)

I can now burst messages to the esp8266 MQTT client without the connection being dropped.

So as long that incoming burst of messages are under 180 seconds ( which is easy) or some publishing from the ESP8266 happens, the connection stays up. Anyway, if it fails, the check timer will reconnect again to the broker.

Simple Dashboard for HomeLab

I’ve found by chance a great source for alternative software for common applications on Reddit: Self Hosted. It’s a “sub” where people discuss, ask or report alternatives to things like Dropbox, online editors, and so on.

One of the posts that caught my attention was a discussion of Simple Dashboards for things like basic URL links to access common applications and so on. Nothing fancy.

Still the basic dashboard is very useful, it allowed me to centralize all links and information on a web page without depending on my browser bookmarks, avoiding the need to sync all the bookmarks across all devices.

Anyway, one of the striking examples is this one: Dashboard example. The discussion is here.

Another example is on this post: from where I used the code at with a mix of the above JsFiddle example for weather data.

So, some fiddling around HTML code is needed, but a simple and effective dashboard can be set on a web server running on Rpi, for example (mine is running on Odroid C1+), to centralize links, information and what ever our imagination allows to, on a web page. As I said, nothing revolutionary here :)

ESP8266 – Sming Framework, issues and tips.

So some issues and tips regarding the Sming framework for programming the ESP8266:

The Arduino Json library on Sming master branch
Version 4.0 of the Arduino Json Library is present, at least at today’s date (12/2015) on the Master branch of the Sming framework.
The develop branch of the framework uses the version 5.0 of the Arduino Json library.

To do any useful work with JSON on the Sming framework we should use the develop branch and not the master branch.

It might be related, or not, didn’t investigated too much, but I had huge memory leaks that exhausted my heap very rapidly when using the master branch when processing JSON data. When using the development branch, with the same code base (just a HTTP client and Json decoding) those issues where gone.

Other issues with version 4.0 of the Arduino Json Library included the processing of double/float values that had no floating point, which is a great problem on this version. For example (double)15 evaluates as zero, while (double)15.0 evaluates correctly. So it is really important to upgrade and use the version 5 of the JSON library.

So instead of just cloning the master branch, clone the development branch: git clone -b develop https://github.com/SmingHub/Sming.git

Building images
On the master branch I had no trouble creating and flashing the resulting images to the ESP8266, but on the development brach I had serious problems. The issue is solved now, and it is related to the esptool.py tool that is unable to generate the image files to be flashed. The latest version of the esptool.py available on the Github repository complains regarding the errno definition, also something that is related, to the Json Library.

Anyway, the latest commits on the development branch uses the esptool2, just clone the repository onto somewhere (in my case /opt):

[pcortex@pcortex: opt]$ git clone https://github.com/raburton/esp8266
[pcortex@pcortex: opt]$ cd esp8266/esptool2
[pcortex@pcortex: esptool2]$ make

And make sure that the esptool2 binary is on the path, or change the Makefile-project.mk located, in my case, in /opt/Sming/Sming:

From

# esptool2 path
ESPTOOL2 ?= esptool2

to

# esptool2 path
ESPTOOL2 ?= /opt/esp8266/esptool2/esptool2

After this, the usual cycle of make and make flash should work. If we already have a project we should first run make clean just that the esptool2 can create the new image files.
Note that the esptool.py tool is still used for the flashing process.

So the final conclusion is, despite its bleeding edge status, to use the development brach, and not the master branch as of today. The issues that are solved on the development branch are important to the make the framework useful.

ESP8266 Home made developer board and I2C

Finally I had the time to finish out the developer board that I’m using to program the ESP-01 chips as I described on this link: ESP8266 Developer board.

One of the remaining doubts that I had was if the I2C bus would work at all on the developer board. The issue is that while GPIO2 pin is not connected to anything, it just has a week pull up resistor, the GPIO0 pin is also used to control the firmware flashing through the CP2102 USB-Serial DTR line as we can see on the below schema:

Selection_012

So the GPIO0 pin has two functions, as the SCL (clock line) and as the flashing enable line.

For the Sming framework, if not specified, the GPIO0 is the SCL line, and GPIO2 is the SDA line.

Well with or without the USB cable from the CP2102 connected to the computer, the I2C bus works fine as expected :) and I can detect and control an I2C connected HD44780 lcd display, that worked right out of the box.

The HD44780 works at 5V and so its connected to the ESP8266 I2C bus through a voltage level shifter.

ESP8266 Sming framework – MQTT

The Sming framework supports the MQTT protocol out of the box. Comparing the MQTT protocol implementation on Sming with Nodemcu Lua, it appears that Sming MQTT implementation is faster and more stable and at least, in a first sight, doesn’t require any buffering code and semaphores to be implemented as I did in MQTT on Nodemcu to work.

So, out of the box code, without any other supporting code, I can have two timers “shooting” out MQTT massages at 200ms/250ms out of the ESP8266 and at the same time receiving MQTT data without losing any message! And this with the Mosquitto broker running on a low power Synology NAS ARM based device… Off course WIFI conditions are fine, but it’s an impressive achievement.
With Lua, I had to implement buffering and semaphoring for publishing, otherwise, I would get exceptions and/or loose messages.

How to start:
The Sming framework already has one example for using the MQTT protocol named MqttClient_Hello.
It is a simple example but shows all that is needed to use the MQTT protocol on the Sming Framework based code.
The API is simple, and it’s based on MqttClient object, where we define the broker address and the callback function for when receiving subscribed topics messages.

#define BROKER "192.168.1.16"   // We can use a hostname! -> "mqtt.myserver.com"
#define BRPORT 1883
#define BRUSER ""               // Empty string if broker doesn't use authentication
#define BRPWD  "pwd"

// Forward declaration, so that the compiler knows the MQTT callback function name before definition.
void onMqttReceived(String topic, String message);

// Define the MQTT object
MqttClient mqtt( BROKER , BRPORT , onMqttReceived );

As we can see the mqtt variable holds now a MQTT broker definition to be used, and also, defines the callback function that is called when a message arrives for a subscribed topic.

The sample function for displaying received messages is simple. It only prints the Topic name and message.
The function is the same for all subscribed topics, so in a real case scenario, some if/then/else logic is probably necessary.

// Callback for messages that arrived for subscribed topics
void onMqttReceived(String topic, String message)
{
	Serial.print(topic);
	Serial.print(":\r\n\t"); // Prettify alignment for printing
	Serial.println(message);
}

But after the above definition we are not still connected. To connect we need to call the connect function and define our client ID.

Connecting

The connect code is as follow:

// Connect to the MQTT broker.
// If BRUSER is not empty, connect with user and password.
void startMqttClient()
{
    String mqttID = "esp8266-" + String( system_get_chip_id() );
    if ( strlen( BRUSER ) == 0 )
	mqtt.connect( mqttID);
    else
        mqtt.connect( mqttID, BRUSER , BRPWD );
    
    // Let's subscribe topics    
    mqtt.subscribe("/status/#");
}

The mqtt.subscribe subscribes the MQTT topic passed as parameter.
Off course we can do as in the NodeMcu example, that is to define an topic array and subscribe each of the array elements.

Subscribe and unsubscribe:
After connecting we can subscribe and unsubscribe with the mqtt.subscribe( topic ) and mqtt.unsubscribe( topic ) functions. In both cases the functions receive the topic name.

As standard with MQTT we can use the # and * specifiers in topic names to subscribe one or more or zero or more.

Publishing:
There are two publishing functions, one that is the standard without Quality of Service and retention defined, and another one where we define those parameters:

  • publish( topic , value ) -> Standard publish
  • publishWithQoS( topic , value , QoS , Retain) -> Extended publish where QoS is defined and the boolean retention flag defines if the broker retains the message in case of non delivery.

The example function shows the use of the publish function, but also checks if a valid Mqtt broker connection is valid.

void publishMessage()
{
	if (mqtt.getConnectionState() != eTCS_Connected)
		startMqttClient(); // Auto reconnect

	Serial.println("Let's publish message now!");
	mqtt.publish("/sming", "Published! :)");
}

Testing:
My testing was done using the Mqtt-Spy tool for monitoring the messages published by the ESP8266, and a simple script that run on the broker machine that mass flood the subscribed topics…
Since I have the broker running on the Synology NAS, that as the ash shell, such script is as follows:

for i in `seq 500` ; do
 /volume1/@appstore/mosquitto/bin/mosquitto_pub -m "Hi there" -t "/status/sming"
done

At the same time I’ve changed the Sming example code for publish in two different topics:


int mq1 = 0;
int mq2 = 200;

Timer pub1;
Timer pub2;

void pubMqtt1() {
     	if (mqtt.getConnectionState() != eTCS_Connected)
		startMqttClient(); // Auto reconnect
    
        mq1 ++;
        if ( mq1 > 100 ) mq1 = 0;

        mqtt.publish("/data1", String(s));
}

void pubMqtt2() {
 	if (mqtt.getConnectionState() != eTCS_Connected)
		startMqttClient(); // Auto reconnect
    
        mq2 ++;
        if ( mq2 > 250 ) mq2 = 200;
    
        
        mqtt.publish("/data2", String(s));
}

....
....
// Will be called when WiFi station was connected to AP
void connectOk()
{
	Serial.println("I'm CONNECTED");

	// Start the MQTT client and subscribe to any topics
	startMqttClient();

	// Start publishing loop
	procTimer.initializeMs(20 * 1000, publishMessage).start(); // every 20 seconds
        
        pub1.initializeMs(200, pubMqtt1).start(true); // True is default , repeat timer.
        pub2.initializeMs(250, pubMqtt2).start(true);
}

....

So with above code that publish on two topics at very short intervals and with the Mosquitto server loop based script that also publishes data subscribed by the Sming framework, I’ve not witness any drops or failures that I’m aware off. Impressive.

Off course in other environments the results can be different, but the code is simple and needs no work arounds to be effectively used.

ESP8266 Sming – Accessing files in SPIFFs filesytem

This short post is just to document some functions of the file system API when using the Sming framework. The Sming framework has a simple API to access the SPIFF file system for opening, reading/writing and deleting files.

The file system can be used for holding web server files, configuration files, log files, anything that needs a file system…

So my quick notes:

Files at flashing time
As I notice in Sming – How to Start we can, at flash time of the firmware, to add files in the Spiffs file system. The only thing needed is to create a sub-drectory, add the files there and then refer the directory on the Makefile-user.mk: For a directory named files:

SPIFF_FILES = files

Also it seems that on the latest version, this directory is detected if exists, and if so, the file system is created.

One important thing is to use this version of the spiffy tool that comes with the release of Sming from Sming Hub. Just go to the /opt/Sming/Sming/spiffy directory and execute make to build and make available the spiffy binary. Other option is when building Sming libraries just execute make followed by make spiffy on /opt/Sming/Sming.
Don’t use spiffy from other sources as I previously was using. Otherwise it won’t work.

Using the Spiffs file system
If your application uses the Spiffy file system it must now must explicitly mount the file system at run time:

void init() {
     spiffs_mount();
     ...
     ...

}

If the file system is not mounted we will get an error -10024 that means the SPIFFs file system is not available/configured…

Listing files
At the application level we can list the contents of the SPIFFs file system with the following function:

void listFileSystem() {
    Vector filelist = fileList();
    
    Serial.println(" ")  ;
    Serial.println("Spiffs file system contents ");
    for ( int i = 0 ; i < filelist.count() ; i++ ) {
        Serial.print(" ");
        Serial.println(filelist.get(i) + " (" + String( fileGetSize(filelist.get(i))) + ")"     );
    }
        
}

We can see here the fileList() function that returns a Vector of Strings with the available list of files. We can then get the file size of a file with the fileGetSize( filename ) function.

We don’t need to do nothing to start using these functions right away, except the inclusion of SmingCore.h.

Error List
Before going further, we need to know the meaning of the errors that the file system functions may return.
These errors are described here: spiffs.h
We can see if trying to open a file we get -10002 it means that the file was not found, or for another example, -10021 we are trying to write into a file open in read mode.

File Flags
The file flags are flags passed to fileOpen() API call to open files.

At least one of the following three MUST be present:

  • eFO_ReadOnly  – Opens the file in READ ONLY mode
  • eFO_WriteOnly   – Opens the file in WRITE ONLY mode
  • eFO_ReadWrite  – Opens the file for reading and writing.

The remaining flags can be associated to change the fileOpen behaviour. For example:

  file_t fh = fileOpen("config.json", eFO_ReadOnly)

The variable fh, if positive holds the file handle to be used by other functions for accessing/manipulating the opened file.
If negative, it holds the error, for example -10002, the file doesn’t exist.

So this flags:

  • eFO_CreateIfNotExist – If the file doesn’t exist it creates it.
  • eFO_Truncate – Before opening the file, it resets its contents to nothing.
  • eFO_CreateNewAlways – Identical it seems to the above, but first it deletes the file. It seems a bug workaround
  • eFO_Append – Opens the file and moves the file pointer to the end of the file.

With the above flags we can do this:

   file_t fh = fileOpen("config.json", eFO_ReadWrite | eFO_CreateIfNotExist  );

In this case if the file doesn’t exist, it is created and no error is returned.

Another set of flags exist for the seek operation where we can seek from the start, current position or the file end:

  • eSO_FileStart – The seek offset is from the file start position (0).
  • eSO_CurrentPos – The seek offset is from the current file pointer position.
  • eSO_FileEnd – The seek offset is from the end of the file.

File operations
So the with the above information, using the SPIFFs file system has the same logic as in other language and platforms: open, read/write, close:

        char data[20] = "" ;
        file_t f = fileOpen("teste.json", eFO_ReadWrite | eFO_CreateIfNotExist );
        
        if ( f > 0  ){
            // Let's write
            fileWrite( f , "ola", 3 );
            // Let's return to the beginning of the file
            fileSeek ( f , 0 , eSO_FileStart );
            // Let's read what we have written.
            int br = fileRead ( f , &data , 3);   // br holds the bytes read.
            // Let's close the file.
            fileClose(f);
        } else {
            Serial.println("Couldn't create file");
        }

Other functions
The API offers other functions, that in a IDE like NetBeans can be seen at code completion.
Worth mention are the setContent() function where all the logic above is done internally, for example:

  setContent("log.txt", "System started");

The above instruction creates and writes on the log.txt file the content “System Started”.

There are other functions, like fileRename(), fileDelete(), getContent() that allow a quick and easy way to use the flash file system on the ESP8266 using the Sming framework.

Dropbox doesn’t start (Linux)

So a quick note regarding Dropbox running on ArchLinux with Nvidia drivers. Since I use KDE Plasma 5 that doesn’t show system tray icons of some applications that have not migrated to the new system tray protocol, I didn’t notice that after upgrading Dropbox it stopped working.

A quick investigation, I’ve found out that starting Dropbox from the command line did nothing. It started and stopped. Something is going on…

So I’ve used the strace command to see if I could catch something meaningful:

[pcortex@pcortex:~]$ cd .dropbox-dist/dropbox-lnx.x86_64-3.10.8/
[pcortex@pcortex:dropbox-lnx.x86_64-3.10.8]$ strace ./dropbox 2> stracedump.txt

And sure enough in the stracedump.txt I had the following error:

open("/usr/lib/dri/tls/swrast_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/dri/swrast_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
rt_sigaction(SIGSEGV, {0x7f09e1dea010, [SEGV], SA_RESTORER|SA_RESTART, 0x7f09e1a88d60}, NULL, 8) = 0
write(4, "Fatal Python error: ", 20)    = 20
write(4, "Segmentation fault", 18)      = 18
write(4, "\n\n", 2)                     = 2
write(4, "Traceback (most recent call firs"..., 36) = 36
write(4, "  File ", 7)                  = 7
write(4, "\"", 1)                       = 1
write(4, "d", 1)                        = 1

So the error is related to the Software Raster library, that I don’t have because I’m running Nvidia proprietary drivers.

The confirmation came with the following command:

[pcortex@pcortex:dropbox-lnx.x86_64-3.10.8]$ LIBGL_DEBUG=verbose ./dropbox
libGL: OpenDriver: trying /usr/lib/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /usr/lib/dri/swrast_dri.so
libGL error: dlopen /usr/lib/dri/swrast_dri.so failed (/usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: swrast_dri.so
libGL error: reverting to indirect rendering

The solution?
Quite simple, just remove (backup) the libGL.so.1 library that exists on the Dropbox directory.
After the removal, Dropbox starts just fine.