ESP8266 Home made developer board and I2C

Finally I had the time to finish out the developer board that I’m using to program the ESP-01 chips as I described on this link: ESP8266 Developer board.

One of the remaining doubts that I had was if the I2C bus would work at all on the developer board. The issue is that while GPIO2 pin is not connected to anything, it just has a week pull up resistor, the GPIO0 pin is also used to control the firmware flashing through the CP2102 USB-Serial DTR line as we can see on the below schema:


So the GPIO0 pin has two functions, as the SCL (clock line) and as the flashing enable line.

For the Sming framework, if not specified, the GPIO0 is the SCL line, and GPIO2 is the SDA line.

Well with or without the USB cable from the CP2102 connected to the computer, the I2C bus works fine as expected :) and I can detect and control an I2C connected HD44780 lcd display, that worked right out of the box.

The HD44780 works at 5V and so its connected to the ESP8266 I2C bus through a voltage level shifter.

ESP8266 Sming framework – MQTT

The Sming framework supports the MQTT protocol out of the box. Comparing the MQTT protocol implementation on Sming with Nodemcu Lua, it appears that Sming MQTT implementation is faster and more stable and at least, in a first sight, doesn’t require any buffering code and semaphores to be implemented as I did in MQTT on Nodemcu to work.

So, out of the box code, without any other supporting code, I can have two timers “shooting” out MQTT massages at 200ms/250ms out of the ESP8266 and at the same time receiving MQTT data without losing any message! And this with the Mosquitto broker running on a low power Synology NAS ARM based device… Off course WIFI conditions are fine, but it’s an impressive achievement.
With Lua, I had to implement buffering and semaphoring for publishing, otherwise, I would get exceptions and/or loose messages.

How to start:
The Sming framework already has one example for using the MQTT protocol named MqttClient_Hello.
It is a simple example but shows all that is needed to use the MQTT protocol on the Sming Framework based code.
The API is simple, and it’s based on MqttClient object, where we define the broker address and the callback function for when receiving subscribed topics messages.

#define BROKER ""   // We can use a hostname! -> ""
#define BRPORT 1883
#define BRUSER ""               // Empty string if broker doesn't use authentication
#define BRPWD  "pwd"

// Forward declaration, so that the compiler knows the MQTT callback function name before definition.
void onMqttReceived(String topic, String message);

// Define the MQTT object
MqttClient mqtt( BROKER , BRPORT , onMqttReceived );

As we can see the mqtt variable holds now a MQTT broker definition to be used, and also, defines the callback function that is called when a message arrives for a subscribed topic.

The sample function for displaying received messages is simple. It only prints the Topic name and message.
The function is the same for all subscribed topics, so in a real case scenario, some if/then/else logic is probably necessary.

// Callback for messages that arrived for subscribed topics
void onMqttReceived(String topic, String message)
	Serial.print(":\r\n\t"); // Prettify alignment for printing

But after the above definition we are not still connected. To connect we need to call the connect function and define our client ID.


The connect code is as follow:

// Connect to the MQTT broker.
// If BRUSER is not empty, connect with user and password.
void startMqttClient()
    if ( strlen( BRUSER ) == 0 )
	mqtt.connect("esp8266-" + system_get_chip_id() );
        mqtt.connect("esp8266-" + system_get_chip_id(), BRUSER , BRPWD );
    // Let's subscribe topics    

The mqtt.subscribe subscribes the MQTT topic passed as parameter.
Off course we can do as in the NodeMcu example, that is to define an topic array and subscribe each of the array elements.

Subscribe and unsubscribe:
After connecting we can subscribe and unsubscribe with the mqtt.subscribe( topic ) and mqtt.unsubscribe( topic ) functions. In both cases the functions receive the topic name.

As standard with MQTT we can use the # and * specifiers in topic names to subscribe one or more or zero or more.

There are two publishing functions, one that is the standard without Quality of Service and retention defined, and another one where we define those parameters:

  • publish( topic , value ) -> Standard publish
  • publishWithQoS( topic , value , QoS , Retain) -> Extended publish where QoS is defined and the boolean retention flag defines if the broker retains the message in case of non delivery.

The example function shows the use of the publish function, but also checks if a valid Mqtt broker connection is valid.

void publishMessage()
	if (mqtt.getConnectionState() != eTCS_Connected)
		startMqttClient(); // Auto reconnect

	Serial.println("Let's publish message now!");
	mqtt.publish("/sming", "Published! :)");

My testing was done using the Mqtt-Spy tool for monitoring the messages published by the ESP8266, and a simple script that run on the broker machine that mass flood the subscribed topics…
Since I have the broker running on the Synology NAS, that as the ash shell, such script is as follows:

for i in `seq 500` ; do
 /volume1/@appstore/mosquitto/bin/mosquitto_pub -m "Hi there" -t "/status/sming"

At the same time I’ve changed the Sming example code for publish in two different topics:

int mq1 = 0;
int mq2 = 200;

Timer pub1;
Timer pub2;

void pubMqtt1() {
     	if (mqtt.getConnectionState() != eTCS_Connected)
		startMqttClient(); // Auto reconnect
        mq1 ++;
        if ( mq1 > 100 ) mq1 = 0;

        mqtt.publish("/data1", String(s));

void pubMqtt2() {
 	if (mqtt.getConnectionState() != eTCS_Connected)
		startMqttClient(); // Auto reconnect
        mq2 ++;
        if ( mq2 > 250 ) mq2 = 200;
        mqtt.publish("/data2", String(s));

// Will be called when WiFi station was connected to AP
void connectOk()
	Serial.println("I'm CONNECTED");

	// Start the MQTT client and subscribe to any topics

	// Start publishing loop
	procTimer.initializeMs(20 * 1000, publishMessage).start(); // every 20 seconds
        pub1.initializeMs(200, pubMqtt1).start(true); // True is default , repeat timer.
        pub2.initializeMs(250, pubMqtt2).start(true);


So with above code that publish on two topics at very short intervals and with the Mosquitto server loop based script that also publishes data subscribed by the Sming framework, I’ve not witness any drops or failures that I’m aware off. Impressive.

Off course in other environments the results can be different, but the code is simple and needs no work arounds to be effectively used.

ESP8266 Sming – Accessing files in SPIFFs filesytem

This short post is just to document some functions of the file system API when using the Sming framework. The Sming framework has a simple API to access the SPIFF file system for opening, reading/writing and deleting files.

The file system can be used for holding web server files, configuration files, log files, anything that needs a file system…

So my quick notes:

Files at flashing time
As I notice in Sming – How to Start we can, at flash time of the firmware, to add files in the Spiffs file system. The only thing needed is to create a sub-drectory, add the files there and then refer the directory on the For a directory named files:


Also it seems that on the latest version, this directory is detected if exists, and if so, the file system is created.

One important thing is to use this version of the spiffy tool that comes with the release of Sming from Sming Hub. Just go to the /opt/Sming/Sming/spiffy directory and execute make to build and make available the spiffy binary. Other option is when building Sming libraries just execute make followed by make spiffy on /opt/Sming/Sming.
Don’t use spiffy from other sources as I previously was using. Otherwise it won’t work.

Using the Spiffs file system
If your application uses the Spiffy file system it must now must explicitly mount the file system at run time:

void init() {


If the file system is not mounted we will get an error -10024 that means the SPIFFs file system is not available/configured…

Listing files
At the application level we can list the contents of the SPIFFs file system with the following function:

void listFileSystem() {
    Vector filelist = fileList();
    Serial.println(" ")  ;
    Serial.println("Spiffs file system contents ");
    for ( int i = 0 ; i < filelist.count() ; i++ ) {
        Serial.print(" ");
        Serial.println(filelist.get(i) + " (" + String( fileGetSize(filelist.get(i))) + ")"     );

We can see here the fileList() function that returns a Vector of Strings with the available list of files. We can then get the file size of a file with the fileGetSize( filename ) function.

We don’t need to do nothing to start using these functions right away, except the inclusion of SmingCore.h.

Error List
Before going further, we need to know the meaning of the errors that the file system functions may return.
These errors are described here: spiffs.h
We can see if trying to open a file we get -10002 it means that the file was not found, or for another example, -10021 we are trying to write into a file open in read mode.

File Flags
The file flags are flags passed to fileOpen() API call to open files.

At least one of the following three MUST be present:

  • eFO_ReadOnly  – Opens the file in READ ONLY mode
  • eFO_WriteOnly   – Opens the file in WRITE ONLY mode
  • eFO_ReadWrite  – Opens the file for reading and writing.

The remaining flags can be associated to change the fileOpen behaviour. For example:

  file_t fh = fileOpen("config.json", eFO_ReadOnly)

The variable fh, if positive holds the file handle to be used by other functions for accessing/manipulating the opened file.
If negative, it holds the error, for example -10002, the file doesn’t exist.

So this flags:

  • eFO_CreateIfNotExist – If the file doesn’t exist it creates it.
  • eFO_Truncate – Before opening the file, it resets its contents to nothing.
  • eFO_CreateNewAlways – Identical it seems to the above, but first it deletes the file. It seems a bug workaround
  • eFO_Append – Opens the file and moves the file pointer to the end of the file.

With the above flags we can do this:

   file_t fh = fileOpen("config.json", eFO_ReadWrite | eFO_CreateIfNotExist  );

In this case if the file doesn’t exist, it is created and no error is returned.

Another set of flags exist for the seek operation where we can seek from the start, current position or the file end:

  • eSO_FileStart – The seek offset is from the file start position (0).
  • eSO_CurrentPos – The seek offset is from the current file pointer position.
  • eSO_FileEnd – The seek offset is from the end of the file.

File operations
So the with the above information, using the SPIFFs file system has the same logic as in other language and platforms: open, read/write, close:

        char data[20] = "" ;
        file_t f = fileOpen("teste.json", eFO_ReadWrite | eFO_CreateIfNotExist );
        if ( f > 0  ){
            // Let's write
            fileWrite( f , "ola", 3 );
            // Let's return to the beginning of the file
            fileSeek ( f , 0 , eSO_FileStart );
            // Let's read what we have written.
            int br = fileRead ( f , &data , 3);   // br holds the bytes read.
            // Let's close the file.
        } else {
            Serial.println("Couldn't create file");

Other functions
The API offers other functions, that in a IDE like NetBeans can be seen at code completion.
Worth mention are the setContent() function where all the logic above is done internally, for example:

  setContent("log.txt", "System started");

The above instruction creates and writes on the log.txt file the content “System Started”.

There are other functions, like fileRename(), fileDelete(), getContent() that allow a quick and easy way to use the flash file system on the ESP8266 using the Sming framework.

Dropbox doesn’t start (Linux)

So a quick note regarding Dropbox running on ArchLinux with Nvidia drivers. Since I use KDE Plasma 5 that doesn’t show system tray icons of some applications that have not migrated to the new system tray protocol, I didn’t notice that after upgrading Dropbox it stopped working.

A quick investigation, I’ve found out that starting Dropbox from the command line did nothing. It started and stopped. Something is going on…

So I’ve used the strace command to see if I could catch something meaningful:

[pcortex@pcortex:~]$ cd .dropbox-dist/dropbox-lnx.x86_64-3.10.8/
[pcortex@pcortex:dropbox-lnx.x86_64-3.10.8]$ strace ./dropbox 2> stracedump.txt

And sure enough in the stracedump.txt I had the following error:

open("/usr/lib/dri/tls/", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/dri/", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
rt_sigaction(SIGSEGV, {0x7f09e1dea010, [SEGV], SA_RESTORER|SA_RESTART, 0x7f09e1a88d60}, NULL, 8) = 0
write(4, "Fatal Python error: ", 20)    = 20
write(4, "Segmentation fault", 18)      = 18
write(4, "\n\n", 2)                     = 2
write(4, "Traceback (most recent call firs"..., 36) = 36
write(4, "  File ", 7)                  = 7
write(4, "\"", 1)                       = 1
write(4, "d", 1)                        = 1

So the error is related to the Software Raster library, that I don’t have because I’m running Nvidia proprietary drivers.

The confirmation came with the following command:

[pcortex@pcortex:dropbox-lnx.x86_64-3.10.8]$ LIBGL_DEBUG=verbose ./dropbox
libGL: OpenDriver: trying /usr/lib/dri/tls/
libGL: OpenDriver: trying /usr/lib/dri/
libGL error: dlopen /usr/lib/dri/ failed (/usr/lib/dri/ cannot open shared object file: No such file or directory)
libGL error: unable to load driver:
libGL error: reverting to indirect rendering

The solution?
Quite simple, just remove (backup) the library that exists on the Dropbox directory.
After the removal, Dropbox starts just fine.

ESP8266 Sming – How to start

After dumping Lua Nodemcu for doing any useful work/tinkering on the ESP8266(it works fine for simple things and that’s it), from the available options for programming the ESP8266, native C/C++ SDK programming, Arduino, Sming, Smart.js and Espruino, I’ve selected Sming. One of the main reasons for selecting Sming is the huge range of working examples comprising HTTP clients, servers, AJAX, sensors, MQTT, you name it. Also I can use Netbeans IDE to build the ESP8266 applications and since Netbeans offers code completion, It can help a lot with the apparent lack of API documentation.

Netbeans Sming Code Compltetion

Netbeans Sming Code Compltetion

Another advantage that is not stated enough is the SPIFFS file system support. Install spiffy and under your project directory, create a files sub-directory, put the files there, and those files are flashed onto the ESP8266 as the SPIFFS file system…

How to start
Install first the esp-open-sdk and then Sming. The Sming instructions are pretty clear, in my opinion, on the GitHub site. Also if using Netbeans instructions are here: Setting up Sming with Netbeans. Make sure that the include paths are configured. It will enable code completion and get rid of the “errors” on the IDE.

After that we can export the variables ESP_HOME and SMING_HOME with the esp-open-sdk and Sming paths, or change the of each project to point to the correct locations.

Installing and configuring Spiffy.

The following instructions are deprecated! -> See below how to install from SmingHub.
As usual, download spiffy from Github, compile, and make Sming aware of its location:

[pcortex@dune:opt]$ git clone
Cloning into 'spiffy'...
remote: Counting objects: 101, done.
remote: Total 101 (delta 0), reused 0 (delta 0), pack-reused 101
Receiving objects: 100% (101/101), 78.41 KiB | 0 bytes/s, done.
Resolving deltas: 100% (54/54), done.
Checking connectivity... done.
[pcortex@dune:opt] cd spiffy
[pcortex@dune:spiffy master]$ mkdir build
mkdir: created directory ‘build’
[pcortex@dune:spiffy master]$ make

Under the directory build should be now a binary named spiffs.

Goto now to the Sming/Sming subdirectory and edit the and edit the SPIFFY ?= spiffy line to include the full path of spiffy. In my case:

# Full path spiffy command, if not in path.
SPIFFY ?= /opt/spiffy/build/spiffy

With this change when we clone one of the available projects we can just create a files subdirectory under the project directory to enable files that are put there to be flashed onto the ESP8266. Just make sure that you add to the project file the line: SPIFF_FILES = files

Spiffy from SmingHub

The latest version of Sming that is available from Sming Hub already has spiffy available. We just need to build it.

[pcortex@dune:opt] cd /opt/Sming/Sming/spiffy
[pcortex@dune:spiffy] make

And that’s it, spiffy binary is now available.
Another way to build is just to make spiffy at the /opt/Sming/Sming directory.

First project
For first project I recommend to clone/copy one of the provided examples to a directory OUT of the Sming folders. Why? Because in this case we won’t have any conflicts when doing a git pull to update the Sming framework, and we keep the original projects intact, so we can use them as reference.

After copying the project we can see that the file structure is simple, and coding isn’t that hard ( but it’s needed to have some C and C++ proficiency ) to build applications. Coding on Sming framework is very similar to programming for an Arduino function and library wise, but with some differences, namely the callback “way” of programming and the nonexistence of the setup()/loop() paradigm (it doesn’t makes sense on a callback way of programming).

The code folder structure should mimic the example projects, namely a root folder where the make files reside, and under this folder the app folder for application files (.cpp, .c), the include folder for include files, and a files folder (optional) for holding files for the spiff file system.
Still these folders can be changed by modifying the file at the root directory, namely the MODULES entry and the SPIFF_FILES entry:

#Add your source directories here separated by space
# Sources under the ./app and ./libcode1 and ./libcode2
MODULES = app libcode1 libcode2

## MacOS / Linux:
ESP_HOME = /opt/esp-open-sdk

# MacOS / Linux
SMING_HOME = /opt/Sming/Sming

# MacOS / Linux:
# COM_PORT = /dev/ttyUSB0

# Com port speed
COM_SPEED	= 115200

# SPIFFs files Location


All Sming framework based programs must provide a init() function, that is called at ESP8266 startup. There is no setup() or loop() functions. The later function doesn’t make sense since for programming the ESP8266 we register a function for an event, and that function is called when the event triggers. No need for pooling and/or looping. So no loop() function. The Arduino sketch setup() function is in our Sming framework our init() function…

On the init() function we can setup a lot of things like, pin modes, serial port, and of course Wifi connectivity. Programming with an IDE that offers code completion allows us to see the available API and guess some of the functionality even without documentation ( note that most of the API is identical/similar to the Arduino).

So for programming with the Sming framework these are some of my notes that might be helpful:

Serial port
The always helpful serial port… allows the output of data and debugging information and keyboard input, at least when the device is connected to the computer.
The serial port usage from a coding stand point is identical to the Arduino library, but ESP8266 outputs some debugging information also for the serial port:

Serial.systemDebugOutput( flag ) - Enables/disables framework debugging.

When the value of the input parameter flag is true, we can see a lot of information, including HTTP headers, and so on. Quite verbose, which means that our output might be lost in the middle of this debugging information.
Still with the above flag set as false we can still see some debug output that can be almost completely shut off with the following instruction: system_set_os_print(0);.

So a standard init function would be something like:

void init() {
        system_set_os_print(0);  // Zero to disable, 1 to enable. Disable SDK output to serial 
  	Serial.begin(SERIAL_BAUD_RATE); // 115200 by default. SERIAL_BAUD_RATE constant is defined in include/user_config.h
	Serial.systemDebugOutput(false); // Disable debug output to serial
        Serial.println("Starting up!...");


A final note regarding printing to the serial port and debugging, we can use the higher level Serial output instructions, but with Sming we also can use/mix the standard SDK instructions: this means that we can also use the functions debugf and os_printf.

System ready
We can on the init function register a callback for when the system is ready:

void systemIsReady() {
   // do stuff.
   // Start servers
   // Note that wifi connection here is not assured. 


void init() {
        system_set_os_print(0);  // Zero to disable, 1 to enable. Disable SDK output to serial
  	Serial.begin(SERIAL_BAUD_RATE); // 115200 by default. SERIAL_BAUD_RATE constant is defined in include/user_config.h
	Serial.systemDebugOutput(false); // Disable SMING debug output to serial
        Serial.println("Starting up!...");


Wifi Connection
Almost all the examples have the setting up of a wifi connection, some as a client, others as access point.
The important thing is that the all the callback functionality either on success or failure is already implemented, and so it’s quite easy to check for success or failure.

// Will be called when WiFi station was connected to AP
void connectOk()
        String devIP;
        String mac;
	Serial.println("I'm CONNECTED");

        devIP = WifiStation.getIP().toString();
        Serial.println("Device IP : " + devIP );

        mac = WifiStation.getMAC();
        Serial.println("Mac addrs: " + mac );

// Will be called when WiFi station timeout was reached
void connectFail()
	Serial.println("I'm NOT CONNECTED. Need help :(");

        WifiStation.waitConnection(connectOk, 20, connectFail); // We recommend 20+ seconds for connection timeout at start

void init() {

    // ....
	WifiStation.config(WIFI_SSID, WIFI_PWD);

	// Run our method when station was connected to AP (or not connected)
	WifiStation.waitConnection(connectOk, 20, connectFail); // We recommend 20+ seconds for connection timeout at start



Note that is way easier than Nodemcu Lua where we need to implement the connection check by ourselfs by doing a timer based pooling.
With Sming is quite easy, connectOK() is called when a successful connection is made, and connectFail() is called if the timeout (the second parameter) is reached and a connection isn’t made.

On connectFail() we can just again repeat the same instruction or list the available networks and try to connect to some of them.

Still, in my opinion, we should set up the WIFI parameters and connection initiation under the system ready callback.

Some code snippets
List the available wifi networks:

// callback function for WifiStation.startScan(networkScanCompleted);
void networkScanCompleted(bool succeeded, BssList list)

	if (!succeeded)
		Serial.println("Failed to scan networks");

	for (int i = 0; i < list.count(); i++)
		Serial.print("\tWiFi: ");
		Serial.print(", ");
		if (list[i].hidden) Serial.print(" (hidden)");

void init() {

List system info.

void ShowInfo() {
    Serial.printf("\r\nSDK: v%s\r\n", system_get_sdk_version());
    Serial.printf("Free Heap: %d\r\n", system_get_free_heap_size());
    Serial.printf("CPU Frequency: %d MHz\r\n", system_get_cpu_freq());
    Serial.printf("System Chip ID: 0x%x\r\n", system_get_chip_id());
    Serial.printf("SPI Flash ID: 0x%x\r\n", spi_flash_get_id());
    Serial.printf("SPI Flash Size: %d\r\n", (1 <> 16) & 0xff)));

The above snippet was kind of important because it shows how to get the system or chip id that can used to identify the device, for example, in MQTT device names.

Just a final note, that all the above code and snippets are collected from several sources:

– Sming Examples
– ESP8266.Com forum
– GitHub: and mainly

SparkFun Phant server data stream graphing

This application Phant Graphs is a simple AngularJs and ChartJs single page web application that allows to configure and create a dashboard of graphs that display stream data available on Sparkfun Phant servers, either localy, or on the internet ( for example).

The datasources ( servers and streams) can be configured rapidly, the graphs types can be chosen and also the fields that make up the graphs can also be chosen, so we can have some nice graphs without coding a single line. The application also pools periodically the data streams for the latest data and updates the graphs if needed.
For example, on the following image, for the same data stream there are several graphs, where each one show a different set of fields on different graph type. This is useful, and needed, for example for graphing temperature and pressure. Due to the scale differences between unit values a single graph won’t show properly the temperature for a graph that also has the pressure field.

PhantGraphs Dashboard

PhantGraphs Dashboard

How to deploy
Deploy the supporting rest server:, and start the server. The provided database already has some configuration done.
Deploy PhantGraphs on a web server, and change the REST server endpoint on the restapi.js file to point to the correct location of the Rest Server.

To configure, goto Config->Phant Servers where we can add, edit and delete servers. We should provide a name and the base URL for the Phant Server.
Please notice, that at least at this stage, the REST server doesn’t provide referential integrity on the data, so it’s a bad idea to delete a server that has streams and graphs associated…

Phant Servers

Phant Servers

After a server is configured, we can associate data streams to it. Just goto Config->Phant Streams and create a stream. The stream public key is needed, so we have to get it on the Phant server web interface. The PhantGraphs application will try to fetch from the stream the available fields.

Phant Streams

Phant Streams

If it fails, an error will appear at the top of the page.

After all the streams are configured, we can now create the graphs for them, one or more, with different graph types and for different stream fields.
Some things are not quite ready here, namely the graph display order, that must be defined, but it’s not used for now.

Phant Graphs configuration

Phant Graphs configuration

We can play around with the Graph type, like Line, Bar, Pie, and so on. The option Single Bars only applies to Bar Graphs, that for these type of graphs instead of using a time series for the X axis, it uses only bars for the fields defined.

After this, we just go to Dashboard and enjoy our graphs.

On Config->Dashboard we can change the refresh period for getting the latest data. This value shouldn’t be to low because it will overload the browser and the servers. We can also define the number of datapoints shown on the graphs, the number of errors that the application will endure before stopping the automatic refresh. To resume, just refresh the page or click on any graph. The label interval is used in the Line graph charts so that the X axis doesn’t become too overcrowded. Anyway, just play with the values.

The MQTT is just a simple MQTT over web sockets monitor, that can be configured in Config->MQTT Server. The MQTT broker must support MQTT over web sockets, for example Mosquitto 1.4 or Mosca.

The code is based on the flm-ui application available on Github.

IoT Device Provisioning
The application also allows to list IoT devices, namely the ESP8266, that registered on the RestServer, check their status, and make a simple JSON based properties configuration that is requested and loaded by the devices:

IoT Device Provisioning

IoT Device Provisioning

Configuring properties for the device:


Check out for more information: IoT device provisioning and tracking

And that’s it. Enjoy your new graphs.

IoT device provisioning and tracking

If we are deploying several IoT devices, above a certain number it becomes a hard task to know which ones are alive and working, their location, and since they may have different functions/purposes, to keep track of their configuration. This last problem is exacerbated if we need to keep and maintain different code bases and firmwares for each device.

Device provisioning and some cleaver firmware code, can help to bring these issues under control. Device provisioning is nothing more than a fancy name for a central server that holds information for each device, it’s configuration and status and even might allow some form of monitoring. In advanced usage it might let an operator to push firmware or configuration updates to certain devices, without the need to physical get them.

So this post is about my simple device provisioning that allows to keep track of my ESP8266 devices, running the Sming firmware, and control, in this initial stage, some of it’s configuration.

The solution is based in three components:

  • NodeJs based REST server that holds information in a SQLite database
  • An Angular.Js based web application to managed the device configuration
  • Sming based firmware for the ESP8266

The REST server
The REST server is based in javascript running on a NodeJs server, which uses Express and Body-Parser to create a simple and quick REST server. The REST protocol uses the HTTP operations GET, POST and PUT to define operations like get data, create data or update data. Due to the Express modules, we can build a simple interface to a database that translates REST operations to database operations. The Body-Parser module allows to communicate purely using JSON, and so takes care of all the details of coding and decoding JSON data communicated over HTTP.
Finally there is one important thing, at least for the browser front-end that is the CORS protection. CORS, Cross Origin Resource Sharing is a security feature that browsers use to not connect to anything else other than the originating domain of the page that are showing. To allow our REST server to be called by a running application on a browser (in our case, our Angular.Js frontend), the server must explicitly allow this. So we also need a CORS module for our REST server.
Finally I’ve chosen a SQLITE database due to it’s simpler configuration. No need to setup servers…
To allow our REST server to run we need to install locally to our application the folowing modules: npm install express node-sqlite3 body-parser cors

// Define and call the packages necessary for the building the REST API.
var sqlite3    = require('sqlite3').verbose();
var db         = new sqlite3.Database('data/restDB.sqlite');
var express    = require('express');
var app        = express();
var bodyParser = require('body-parser');
var cors       = require('cors');

After the module instantiation, and note that the database is created in the data subdirectory, we can create if needed our tables and fill them up with default data if needed:

db.serialize(function() {  // All database operations will be executed in series.
    // some code here"CREATE TABLE IF NOT EXISTS devices ( deviceid TEXT , name TEXT , lastseen TEXT , ssid TEXT, ipaddr TEXT, cfgsn INTEGER, flags INTEGER, data TEXT, datec TEXT )");
    // some more code here

The db.serialize makes the database functions to run in series, one after each other.
The will execute a SQL command, and in the above case the creation of the device provisioning table.

The schema is the following:

  • deviceid -> the device id as sent by the device. It should be unique. I use the MAC address of the ESP8266 that should be unique.
  • name -> a human friendly name for the device. It can be anything.
  • lastseen -> The device should periodically call the REST server to let us know that’s is alive and connected. The lastseen field is the last time that information was received.
  • ssid -> Shows the WiFi SSID to which the device is connected.
  • ipaddr -> Shows the device IPv4 address in use. Might not be reachable if the device is behind a NAT router.
  • cfsn -> Config serial number. To let the device know if the configuration it holds is older than the configuration on the table row.
  • flags -> For future use, but the idea is to allow pagination, and command/control/data separation.
  • data -> Configuration data for the device. In my case a JSON array of properties and values user defined.
  • datec -> The date that the record was create, which means when the device was first seen.

Now knowing what information we are using, we can build the REST server:

//Configure Express
app.use(bodyParser.urlencoded({ extended: true }));

var router = express.Router();

// Root URI.
router.get('/', function(req, res) {
        res.json( {message: 'REST API' });

///// OUR CODE here

// Our base url is /api
app.use('/api', router);

console.log("Server started at http://localhost:3000/");

Excluding the ///// OUR CODE here this is the standard skeleton for the REST server that runs at port 3000 and hears for REST request on the /api base url.

For testing I recommend the FireFox plugin HTTPRequester that allows to simulate all the REST calls.

The devices REST interface

For the device provisioning I’ve implemented the REST end point devices which mean that we should call our REST server with the following URL’s: http://rest_server:3000/api/devices and http://rest_server:3000/api/devices/deviceid where deviceid is the device generated id.

For the first URL, so far I only implement the GET verb, which returns all registered devices:

        .get ( function(req, res) {
           console.log("Devices list");
           db.all("SELECT ROWID, deviceid, name, lastseen, ssid, ipaddr, cfgsn, flags, data, datec  FROM devices",
              function( err , row ) {
                if (err) {
                  res.json( { status:'NOK', message: 'ERROR selecting data'} );
                } else {            
                  console.log('API: GET devices -> (' + row.length + ') records -> ' + JSON.stringify(row));
                   res.json( row );

The function db.all returns an empty array if no results, or an array with row set. The row set is returned directly as a JSON array object to the calling application/device.


For the last URL, the implemented verbs are GET, to get the latest configuration and update the lastseen field, POST to register the device and/or returning the configuration, and PUT to update device information (used by the front end):

    .get( function(req, res) {
        var currdate = new Date();"UPDATE devices SET lastseen='" + currdate + "' WHERE deviceid='" + + "'");

        db.get("SELECT cfgsn, data FROM devices WHERE deviceid='""'",
          function( err , row ) {
            if (err) {    
              res.json( {status: 'NOK', message: 'Device get information failed' } );
            } else {
              if ( (typeof row == 'undefined') ) { 
                res.json( {status: 'NOK', message: 'Device NOT Registered' } );
              } else {
                res.json( row );
    .put( function(req, res) {"UPDATE devices SET name='""', cfgsn="+req.body.cfgsn.toString()+", data='""' WHERE deviceid='""'");  
        res.json( {status: 'OK', message: 'Device Updated' } );
    .post( function(req, res) {
        console.log("DEVICES -> POST Device request received: " +;
        console.log("Body: " + JSON.stringify(req.body) );
        console.log("IP address: " + req.body.ipaddr );
        console.log("SSID: " + req.body.ssid );

        db.get("SELECT ROWID , deviceid, name, lastseen, ssid, ipaddr, cfgsn, flags, data, datec FROM devices WHERE deviceid='""'",
          function( err , row ) {
            if (err) {
              res.json( {status: 'NOK', message: 'Device Registration failed' } );
            } else {
              if ( (typeof row == 'undefined') ) {    // Device is new... if row is undefined (no results)
                var deviceid  =;
                var name      =;
                var lastseen  = new Date();
                var ssid      = req.body.ssid;
                var ipaddr    = req.body.ipaddr;
                var cfgsn     = "1";
                var flags     = "0";
                var data      = "";
                var datec     = lastseen;
      "INSERT INTO devices ('deviceid','name','lastseen','ssid','ipaddr','cfgsn','flags','data','datec') VALUES ('" + deviceid + "', '" + name + "', '" + lastseen + "', '" + ssid + "', '" + ipaddr + "', " + cfgsn + ", " + flags + ", '" + data + "', '" + datec + "')");
                res.json( {status: 'OK', message: 'Device Registed' } );
             } else {
                res.json( row ); // Device already registered

The above REST verbs, namely the POST and PUT need a request body, to pass data from the source to the database.

In case of the POST verb the, body should be the following JSON: { “ipaddr”:”x.x.x.x”, “ssid”:”wifissid” }. For example:


We save the above code on a file, let it be restserver.js for example, and start our server with node restserver.js.

On the front end then we can have now a list of devices that are registered:


And selecting the device we can change the human readable name, and add configuration properties:


The front end, based on Angular Js isn’t completly ready, since it’s main purpose is to graph data from Sparkfuns Phant server.

Sming ESP8266 Code

To make this complete, we only need to start coding the ESP8266. I’ve abandoned NodeMcu and Lua language, due to several factors. Sming is a great firmware and already has support for a lot of devices and also includes a JSON library.

The sample Sming code is as simple as this:

#include <user_config.h>
#include <SmingCore/SmingCore.h>

#ifndef WIFI_SSID
    #define WIFI_SSID "defineme" // Put you SSID and Password here
    #define WIFI_PWD "passwd"

HttpClient provServer;  // Provisioning server client
String deviceid;

void onProvSent(HttpClient &client, bool successful )
    char cbuf[2048];
    StaticJsonBuffer<2048> jsonBuffer;
    if (successful)
        Serial.println("Registration request sent sucessufully");
        Serial.println("Registration request has failed");
    String response = client.getResponseString();
    Serial.println("Server response: '" + response + "'"); 
        response.toCharArray( cbuf , 2048 );      
        JsonObject& config = jsonBuffer.parseObject( cbuf );
        if ( !config.success() ) {
            Serial.println("Error decoding JSON.");
        } else {
            Serial.println("Received config: " + config.toJsonString() );
            const char* name = config["name"];
            const char* datec= config["datec"];
            Serial.println ("Device Name: " + String(name) );
            Serial.println ("Date: " + String(datec) );
            // Process the JSON Array with the configuration data...


void registerDevice ()
        String devIP;
        StaticJsonBuffer<256> jsonBuffer;
        devIP = WifiStation.getIP().toString();
        Serial.println("Sending device ID " + deviceid + " to provisioning server..." );
        Serial.println("Device IP : " + devIP );      
        JsonObject& config = jsonBuffer.createObject();
        config.addCopy("ipaddr", devIP );
        config.add("ssid"  , WIFI_SSID );
        provServer.downloadString("" + deviceid , onProvSent );

// Will be called when WiFi station was connected to AP
void connectOk()
    Serial.println("I'm CONNECTED");
        // Let's get the Mac address, that should be unique...
        deviceid = WifiStation.getMAC();
        Serial.println("Device ID: " + deviceid );
        // Register Device on the provisioning Server

    // Start send data loop
    // procTimer.initializeMs(25 * 1000, sendData).start(); // every 25 seconds

// Will be called when WiFi station timeout was reached
void connectFail()
    Serial.println("I'm NOT CONNECTED. Need help :(");

    // Start soft access point
    WifiAccessPoint.config("CONFIG ME PLEEEEASE...", "", AUTH_OPEN);

    // .. some you code for configuration ..

void init()
    Serial.begin(SERIAL_BAUD_RATE); // 115200 by default
    Serial.systemDebugOutput(false); // Disable debug output to serial

    WifiStation.config(WIFI_SSID, WIFI_PWD);

    // Run our method when station was connected to AP (or not connected)
    WifiStation.waitConnection(connectOk, 20, connectFail); // We recommend 20+ seconds for connection timeout at start

And that’s it. After startup, the device after connecting to WiFi, makes a request to the provisioning server. If the request is from a new device it registers, if not it receives it’s configuration data. From this point we can do anything, namely pooling periodically the provisioning server to check for new configuration (by comparing the cfgsn hold by the device with the received one).