SparkFun Phant server data stream graphing

This application Phant Graphs is a simple AngularJs and ChartJs single page web application that allows to configure and create a dashboard of graphs that display stream data available on Sparkfun Phant servers, either localy, or on the internet ( for example).

The datasources ( servers and streams) can be configured rapidly, the graphs types can be chosen and also the fields that make up the graphs can also be chosen, so we can have some nice graphs without coding a single line. The application also pools periodically the data streams for the latest data and updates the graphs if needed.
For example, on the following image, for the same data stream there are several graphs, where each one show a different set of fields on different graph type. This is useful, and needed, for example for graphing temperature and pressure. Due to the scale differences between unit values a single graph won’t show properly the temperature for a graph that also has the pressure field.

PhantGraphs Dashboard
PhantGraphs Dashboard

How to deploy
Deploy the supporting rest server:, and start the server. The provided database already has some configuration done.
Deploy PhantGraphs on a web server, and change the REST server endpoint on the restapi.js file to point to the correct location of the Rest Server.

To configure, goto Config->Phant Servers where we can add, edit and delete servers. We should provide a name and the base URL for the Phant Server.
Please notice, that at least at this stage, the REST server doesn’t provide referential integrity on the data, so it’s a bad idea to delete a server that has streams and graphs associated…

Phant Servers
Phant Servers

After a server is configured, we can associate data streams to it. Just goto Config->Phant Streams and create a stream. The stream public key is needed, so we have to get it on the Phant server web interface. The PhantGraphs application will try to fetch from the stream the available fields.

Phant Streams
Phant Streams

If it fails, an error will appear at the top of the page.

After all the streams are configured, we can now create the graphs for them, one or more, with different graph types and for different stream fields.
Some things are not quite ready here, namely the graph display order, that must be defined, but it’s not used for now.

Phant Graphs configuration
Phant Graphs configuration

We can play around with the Graph type, like Line, Bar, Pie, and so on. The option Single Bars only applies to Bar Graphs, that for these type of graphs instead of using a time series for the X axis, it uses only bars for the fields defined.

After this, we just go to Dashboard and enjoy our graphs.

On Config->Dashboard we can change the refresh period for getting the latest data. This value shouldn’t be to low because it will overload the browser and the servers. We can also define the number of datapoints shown on the graphs, the number of errors that the application will endure before stopping the automatic refresh. To resume, just refresh the page or click on any graph. The label interval is used in the Line graph charts so that the X axis doesn’t become too overcrowded. Anyway, just play with the values.

The MQTT is just a simple MQTT over web sockets monitor, that can be configured in Config->MQTT Server. The MQTT broker must support MQTT over web sockets, for example Mosquitto 1.4 or Mosca.

The code is based on the flm-ui application available on Github.

IoT Device Provisioning
The application also allows to list IoT devices, namely the ESP8266, that registered on the RestServer, check their status, and make a simple JSON based properties configuration that is requested and loaded by the devices:

IoT Device Provisioning
IoT Device Provisioning

Configuring properties for the device:


Check out for more information: IoT device provisioning and tracking

And that’s it. Enjoy your new graphs.


Mosquitto Broker with Websockets enabled on the Synology NAS

Synology NAS are great devices, mainly due to the bundled software (CloudStation, Photo Station, to name a few), but also because it can run other software that is provided on non official package sources.
One of such providers is the SynoCommunity that provides a Mosquitto MQTT broker packaged for the Synology NAS. While I was able to successfully cross compile the Mosquitto broker for my DS212+ NAS with websockets enabled (I do have a draft post with the steps that I’ve never published), the SynoCommunity package allows a simpler way to install and use a Mosquitto broker that has the WebSockets protocol support compiled in.

So, what is needed to be done?

Setting up:

  • 1st: Add to your Synology server the SynoCommunity package source. See the detailed instructions on the community web page.
  • 2nd: Install the Mosquitto package from the Synology package manager.
  • 3rd: Edit the file located at /usr/local/mosquitto/var/mosquitto.conf. You need to connect through ssh as root to do this.

At the end of this file add the following lines:

listener 1883
listener 9001
protocol websockets

Note that the 1883 is the standard MQTT broker port, and 9001 is the port that I’ve choose have the websockets server listening/answering.

After the change, stop and start the Mosquitto broker on the package manager action dropdown box for the Mosquitto package.

Still using ssh, and logged in as root, check now that the port 9001 is active:

root@DiskStation:/volume1/homes/root# netstat -na | grep 9001 
tcp        0      0  *               LISTEN       
tcp        0      0      ESTABLISHED

In the above case we can see that there is a websocket client connected to the broker from my workstation.


For testing we can use the HiveMQ websocket client: that allows to communicate using the websockets transport.

Despite the fact that the page is available on an external, internet based server, the websocket client will be running on YOUR web browser, on your machine in your own (internal) network, so at the connection settings for the broker, for Host I use my internal IP address:, and for the Port: I use the 9001 value.

I can now subscribe and publish to topics using the browser, and also using MQTT-SPY we can see and check the MQTT Websockets communication.

And that’s it!

Setting up an IoT framework/dashboard with NodeRed, Mosca/Mosquitto and dashboard

With the low cost of the esp8266 chip and boards, it is now easy and affordable to add connectivity  to almost anything that we can think of. Still if we build an Internet of Things device by enabling it with the esp8266 chip to connect to the internet, or other networks, what we can do with the data that we gather? Basically we do what we do with any data that we get, store it, process it and show it, and that’s what we are going to see on this post.

The software (running on Linux platform) that I’ll use is the following:

IBM NodeRed: ( This great software will allow us to get data, process it, store and publish it, with a workflow based designer. It runs under Node-Js Javascript based server engine.

MQTT Broker Mosca: ( There are several MQTT brokers available, but this one runs on Node-Js and so is platform agnostic, and runs where Node-Js runs. Also it has websocket support out of the box, which allows to use the dashboard in a straight foward way. Dashboard: ( Is a great dashboard and freely available at the project GitHub page ( It allows to design and store several dashboards with some cool widgets.

Step 1: Setting up NodeRed:

First of all make sure that you have node and npm (the node package manager) installed on your distribution. You can see that with the commands node -v and npm -v for checking out their versions. In my case, and for now, it’s version v0.10.33 for nodejs and 1.4.28 for npm.

First I’ve created a directory named IoTServer where I’ll keep all software and configurations.

Download the zip file from nodered GitHub page or just clone the repository. I clone the repository, because it’s simpler in the future to update it with just git pull. First just make sure that you have the icu/libicu libraries installed.

cd ~
mkdir IoTServer
cd IoTServer
git clone
cd node-red
npm install

We can now run the nodered server with the command node red.js -v. I use -v at the end to see if any error crops up and also to see if any node modules are missing, and need to be installed. For example:

pcortex@cloudsrv:~/IoTServer/node-red$ node red.js -v  
   Welcome to Node-RED 
24 Feb 11:35:16 - [info] Node-RED version: v0.10.3.git 
24 Feb 11:35:16 - [info] Node.js  version: v0.10.33 
24 Feb 11:35:16 - [info] Loading palette nodes 
24 Feb 11:35:59 - [warn] ------------------------------------------ 
24 Feb 11:35:59 - [warn] [arduino] Error: Cannot find module 'arduino-firmata' 
24 Feb 11:35:59 - [warn] [rpi-gpio] Info : Ignoring Raspberry Pi specific node. 
24 Feb 11:35:59 - [warn] [redisout] Error: Cannot find module 'redis' 
24 Feb 11:35:59 - [warn] [mongodb] Error: Cannot find module 'mongodb' 
24 Feb 11:35:59 - [warn] ------------------------------------------ 
24 Feb 11:35:59 - [info] Server now running at 
24 Feb 11:35:59 - [info] Flows file not found : flows_clouds.json 
24 Feb 11:35:59 - [info] Starting flows

As we can see the node modules arduino, rpi-gpio, redisout and mongodb are missing. If we are going to use them, we need to install them with the npm tool. Also the following message: Flows file not found : flows_clouds.json informs us that no saved nodered workflows where found. We can now access the nodered at adress http://localhost:1880/. If we want to change the bind address or add authentication, we should change the settings.js file. Take a look at for more information, and start nodered with: node red.js -v -s settings.js

We can now start designing our nodered workflows that receive data from the esp8266 via MQTT protocol or even by simple REST based HTTP protocol, process it and store the data payload, if any.

For storing data into a database, we can use MongoDB, or mysql, for example, installing the needed modules.

For mysql we can do:

cd ~/IoTServer/node-red
npm install bignumber.js require-all readable-stream
npm install mysql
npm install node-red-node-mysql

And we need to restart nodered. The mysql node should be available now in storage pallet.

Step 2: Installing Mosca MQTT broker (Alternative 1 for the MQTT Broker)

We can use several brokers, but since we already are running nodered on node-js, we will use Mosca. Both nodered and Mosca can provide a websocked based interface. This is of interest because we can use browser based websocket applications, like, to consume data from our IoT devices.

cd ~
sudo -s
sudo npm install mosca bunyan -g

And to run it just do: mosca -v | bunyan

For running it with websockets enabled, on TCP port 3000, just do:

mosca -v --http-port 3000 --http-bundle --http-static ./ | bunyan

Check out the documentation for more information at:

Step 2: Mosquitto MQTT broker with Websockets support (Alternative 2 for the MQTT Broker)

If having trouble installing or using Mosca, we can use the latest Mosquitto version 1.4 that has websockets support.

For that we need to make sure that cmake and uuid/uuid-dev are installed (For example: sudo apt-get install cmake uuid uuid-dev).

Also we need to download the libwesockets library, compile it and install it:

cd ~/IoTServer
git clone git://
cd libwesockets
mkdir build
cd build
cmake ..
sudo make install
We then need to download and compile Mosquitto 1.4:
cd ~/IoTServer
tar xvzf mosquitto-1.4.tar.gz
cd mosquitto-1.4
We need to edit the file and change the option WITH_WEBSOCKETS:=no  to WITH_WEBSOCKETS:=yes
And finally:
sudo make install
To activate websocket support we need to edit the mosquitto.conf file located at /etc/mosquitto or some other directory and add the following lines, for example at the end of the file:
listener 1883

listener 9001
protocol websockets

Then we run it with:

pcortex@cloudsrv:~/IoTServer$ mosquitto -c /etc/mosquitto/mosquitto.conf
1424790588: mosquitto version 1.4 (build date 2015-02-24 14:38:47+0000) starting 
1424790588: Config loaded from /etc/mosquitto/mosquitto.conf. 
1424790588: Opening ipv4 listen socket on port 1883. 
1424790588: Opening ipv6 listen socket on port 1883. 
1424790588: Opening websockets listen socket on port 9001.

Step 3: Installing dashboard

We are almost at the end of this long post. For our infrastructure we need now to install the dashboard that will allow us to see the data in the browser.
cd ~/IoTServer
git clone

For now I’ll just use apache to serve the freeboard dashboards that is installed in my server. Because freeboard is not installed on the root web directory we need to make the freeboard available to Apache. The easiest way to do this, as long as Apache has the FollowSymLinks option enabled for the document root, is to create a link to the freeboard directory on the web document root:

sudo -s
cd /var/www
ln -s /home/pcortex/IoTServer/freeboard iot

And now freeboard is available at the url http://myserveraddres/iot.

We need now, and finally to add MQTT over Websockets support to freeboard… We are almost there. This post   shows how it’s done but I’ll repeat it here again:

1st: Download the development version of mqtt.js here: ( and save it:

cd ~/IoTServer/freeboard/plugins
mkdir mqtt
cd mqtt
wget --output-document mqttws31.js

We will now download the freeboard MQTT plugin:

cd ~/IotServer
git clone
cd freeboard/plugins/mqtt
cp ~/IoTServer/freeboard-mqtt/paho.mqtt.plugin.js .

We need to edit the paho.mqtt.plugin.js file and change the line:

                "external_scripts" : [
                        "<full address of the paho mqtt javascript client>" 


               "external_scripts" : [

and finally we need to change the index.html file from:

                // *** Load more plugins here *** 


                // *** Load more plugins here *** 

That’s it. We are finally ready. All infrastructure is done. We need now just to configure it.

Final step: Testing it out with mosquitto websockets!:

Let’s start Mosquitto: nohup mosquitto -c /etc/mosquitto/mosquitto.conf &

And check if the WS protocol is active:

netstat -nap | grep 9001

tcp        0      0  *               LISTEN      18973/mosquitto

Setting up  It’s very important to notice that the Dashboard will run on our browser, making in fact our browser the client. This means that when we configure freeboard, we must take notice of this when filling up the MQTT broker address. It will be localhost if the browser AND the broker runs on the same machine, but for the dashboard to work anywhere we should use the public ip of the machine running the broker:

Access the dashboard, and select Datasources -> Add. From the dropdown box, we select the Paho MQTT provider.


We need to configure the data source as follows:


Change the MQTT server address to something that makes sense in your config.

For Mosquitto with websockets enabled, the port is 9001, and for Mosca, the port is 3000, but of course this can change, depending of the configuration used.

We can now add a Panel and a gauge to see data from the topic /data that we defined on the above datasource.

So, Add Pane, and on the new Pane select the + (plus) to add a widget. In our case we will select Gauge. We fill out the required info and on the Value field we press Datasource, and select the newly previous created datasource named Data and because we are not using JSON for data, we just append .msg at the end. So something like datasource[“DS_data”].msg should be writen.


We press Save, and that’s it.

We can now publish into the topic, and the dashboard should update:


For example, in this case: mosquitto_pub -t /data -m 23

That’s it. It works fine with Mosquitto 1.4 with websockets enabled and with the Mosca MQTT broker.

Final setup:

We should make our dashboard permanent and for doing so we need to save the dashboard locally as a JSON file, and move it to the server running the dashboard to the root directory of freeboard.

After that we call directly the saved dashboard with the following URL:


Edit: For Websocket support on the Dashboard, please checkout Freeboard and WebSockets

Warning:The current project at Github (as of 12/2015) has problems saving dashboards when using Firefox. For saving dashboards use Chrome/Chromium.

ESP8266, NodeMcu and MQTT: Event publishing queueing

The MQTT code that I’ve copied and changed that is on the following post: works fine for subscribing topics as long that you don’t overload it:

The code subscribes an topic array for receiving and this works, more or less fine. But it also creates a timer to call periodically a publish_data functions, for testing purposes. In reality imagine that we want to publish something when an Input pin changes value. We can do that with the trig function. For example:

gpio.trig(5, "down",publish_data1)
gpio.trig(4, "down",publish_data2)

Where publish_data# are functions that publish something to a MQTT topic. The problem is that it might happen that sometimes both triggers (two buttons pressed at the same time, for example) are activated and try to publish to a MQTT topic at the same time.
Let’s simulate this ( please note that this code is a section of the code in the MQTT Getting Started post.

-- Sample publish functions:
function publish_data(topic, message)
if pub_sem == 0 then
pub_sem = 1
m:publish( topic, message ,0 ,0 , function(conn)
print("Sending to " .. topic ..": " .. message)
pub_sem = 0
id1 = id1 +1

function publish_data1()
publish_data("Out1","Mensagem para out1" )
publish_data("Out2","Mensagem para out2" )

In the above code sample, the Out2 topic message is never published because it takes time to the message for the previous topic to go out (Out1), and so the message gets lost. And this happens because we can’t call the m:publish for MQTT publishing before it has ended the previous call, hence the semaphore to avoid this.

So we are going to do some queueing for publishing:

I’m using an unbound Lua array, with two pointers, one for the Head, and another for the Tail.
Please note that this code is not perfect. The last message is only sent if adding a new message.
Also note that this simple code takes huge amounts of memory, even on “compile/interpretation” phase.
But it works for a while before crashing down with lack of memory.

-- Configuration to connect to the MQTT broker.
BROKER = "" -- Ip/hostname of MQTT broker
BRPORT = 1883 -- MQTT broker port
BRUSER = "user" -- If MQTT authenitcation is used then define the user
BRPWD = "pwd" -- The above user password
CLIENTID = "ESP8266-" .. node.chipid() -- The MQTT ID. Change to something you like

-- MQTT topics to subscribe
topics = {"topic1","topic2","topic3","topic4"} -- Add/remove topics to the array

-- Control variables.
pub_sem = 0 -- MQTT Publish semaphore. Stops the publishing whne the previous hasn't ended
current_topic = 1 -- variable for one currently being subscribed to
topicsub_delay = 50 -- microseconds between subscription attempts, worked for me (local network) down to 5...YMMV

-- Publishing structures
pub_topic = {}
pub_message = {}
pub_head = 1
pub_tail = 1

-- connect to the broker
print("heap: " .. node.heap() )
m = mqtt.Client( CLIENTID, 120, BRUSER, BRPWD)

print "Connecting to MQTT broker. Please wait..."
print("heap: " .. node.heap() )
m:connect( BROKER , BRPORT, 0, function(conn)
print("Connected to MQTT:" .. BROKER .. ":" .. BRPORT .." as " .. CLIENTID )
mqtt_sub() --run the subscription function

function mqtt_sub()
if table.getn(topics) < current_topic then
m:subscribe(topics[current_topic] , 0, function(conn)
current_topic = current_topic + 1
tmr.alarm(5, topicsub_delay, 0, mqtt_sub )

function _publish_data()
if pub_head ~= pub_tail then
if pub_sem == 0 then
pub_sem = 1
m:publish( pub_topic[pub_head], pub_message[pub_head] ,0 ,0 , function(conn)
print("Sending to " .. pub_topic[pub_head] ..": " .. pub_message[pub_head])
pub_sem = 0
pub_topic[pub_head] = nil
pub_message[pub_head] = nil
pub_head = pub_head + 1

function publish_data(topic , message )
pub_topic[pub_tail] = topic
pub_message[pub_tail] = message
pub_tail = pub_tail + 1

function publish_data1()
publish_data("Out1","Mensagem para out1" )
publish_data("Out2","Mensagem para out2" )

function run_main_prog()
print("Main program")

tmr.alarm(2, 5000, 1, publish_data1 )

-- Callback to receive the subscribed topic messages.
m:on("message", function(conn, topic, data)
print(topic .. ":" )
if (data ~= nil ) then
print ( data )
end )

ESP8266, NodeMcu and MQTT: Event subscription load testing

I’ve being doing some testing with the latest NodeMcu regarding MQTT support. My experience is, more or less, documented here: .

I wanted to check if concurrent subscription seems to work fine, since I know that publishing is not straightforward. In the code from the above link, the esp8266 subscribes to four topics, named topic1, topic2 and so on.

This simple bash script, calls repeatedly the mosquitto broker publishing command:

for i in {1..60}
/home/pcortex/01.Develop/mosquitto/mosquitto-1.3.5/client/mosquitto_pub -h -m "Mensagem $i" -t topic1
/home/pcortex/01.Develop/mosquitto/mosquitto-1.3.5/client/mosquitto_pub -h -m "Mensagem $i" -t topic2

But with low values on the loop count, the MQTT code on the esp8266 seems to work fine, but with higher values, as above (60), the esp8266 hangs and reboots. One interesting thing was that after some testing not even doing a reset or completely powering it off, I was able to bring the esp8266 back online. It just put out garbage on the serial output and that was it. Strange indeed.

So I’ve reflash it again with the latest NodeMcu version (lots of reflashing hey 🙂 ), but still the esp8266 did not recover! Only garbage out when connecting to the serial port (And no, it wasn’t the serial port rate issue). So I’ve reflashed again the original AT Firmware, and it came back to life again, and then reflash it again with the latest Nodemcu version.

Adding a delay on the bash script, before the done line, like sleep 1 seems to keep the esp8266 running without issue.

So indeed this shows, that bombarding the esp8266 with MQTT messages in burst mode is not a good idea…

Synology DS: Cross compiling Eclipse/IBM RSMB MQTT broker

RSMB: Really Small Message Broker is an MQTT broker/server, simpler than the alternatives like Mosquitto, RabiitMQ, and so on. While it has its origins in IBM labs (as the MQTT protocol), it is now a project/sub project on Eclipse Paho MQTT related software. While downloading the RSBM does provide some binaries for some common platforms, it doesn’t offer any binaries for, in my case, the DS212+ Marvell 88F628x ARM processor.

So let’s see how to cross compile the RSMB for Synology DS and in this process also learn how to cross compile in your desktop computer native software for the Diskstation.


Setting up the cross compiling environment:

First, a read of the following document The 3rd party developer guide from Synology located here  is recommended. Based on this document (page 6 and 7)and on this page, we can know what version of the Synology tool chain we need to download from here:

Download the required tool chain for your Synology version. In my case I have the Synology DS212+ that has the Marvel 88F628x ARM processor, so download this file:

Uncompress the file into the /usr/local directory. DO USE this directory. The tool chain is configured to get all files, libraries and so on from the /usr/local/… directory:

sudo tar xvzf gcc464_glibc215_88f6281-GPL.tgz -C /usr/local/

(Note: It’s a CAPITAL C. Check Synology documentation).

We can now get the RSMB sources.

Cross compiling RSMB:

Open an shell terminal (preferably bash but other shells might work) and create and change to a working directory. Clone the RSMB repository located in with git tools:

mkdir work_dir
cd workdir
git clone
cd org.eclipse.mosquitto.rsmb/rsmb/src/

While, for this case, not all the below settings are needed, for documentation purposes I document them here:

export INSTALLDIR=/usr/local/arm-marvell-linux-gnueabi
export TARGETMACH=arm-marvell-linux-gnueabi
export BUILDMACH=i686-pc-linux-gnu
export CROSS=arm-marvell-linux-gnueabi
export CC=${CROSS}-gcc
export LD=${CROSS}-ld
export AS=${CROSS}-as
export AR=${CROSS}-ar

Just make sure that the INSTALLDIR variable and TARGETMACH and CROSS variables point to the correct settings.

Also, in this case, for compiling RSMB, we need also to add the following variable:

export GCC=${CROSS}-gcc

Otherwise we need to change the Makefile and change the line GCC=gcc to point to the correct compiler. We can compile now:


And we should have the broker executable among others.

Let’s make sure that it is ok:

pcortex@dune:~/01.Develop/org.eclipse.mosquitto.rsmb/rsmb/src$ file broker
broker: ELF 32-bit LSB  executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, stripped

If the output is this:

broker: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=815abb3a1aad7f430c6e825670601c4991b45bd5, stripped

The wrong compiler was called.

Synology installation:

Copy the following files to your synology station: broker and Messages. From your workstation:

scp broker root@diskstation:/usr/local/bin
scp Messages. root@diskstation:/usr/local/bin

Access through ssh the Synology terminal, and make sure that broker is executable and do a test run:

cd /usr/local/bin
chmod +x broker
20150104 190523.162 CWNAN9999I Really Small Message Broker
20150104 190523.162 CWNAN9998I Part of Project Mosquitto in Eclipse
20150104 190523.163 CWNAN0053I Version, Jan  2 2015 20:13:39
20150104 190523.163 CWNAN0054I Features included: bridge
20150104 190523.163 CWNAN9993I Authors: Ian Craggs (, Nicholas O'Leary
20150104 190523.163 CWNAN0014I MQTT protocol starting, listening on port 1883

And success! We can now test with MQTT-Spy (, Android Client, or Eclipse Paho tools.

Configuration and start and stopping:

For configuring the RSMB, we really should really read the documentation… 🙂 that is provided…

A simple configuration file should be located at /usr/local/etc and named rsmb.conf with the following basic contents:

# sample configuration on port 1883 with retained persistence in /tmp
port 1883
max_inflight_messages 50
max_queued_messages 200
persistence_location /tmp/
retained_persistence true

And at the /usr/local/etc/rc.d create a file named with the following content:


case $1 in
    nohup /usr/local/bin/broker /usr/local/etc/rsmb.conf >> /var/log/rsmb.log&
    /usr/bin/logger -p1 -t "rsmb: INFO  " " Service started."
    /usr/bin/killall broker
    /usr/bin/logger -p1 -t "rsmb: INFO  " " Service stopped."

Save and chmod +x

Now the broker should start and stop automatically.

Final notes:

To use the broker from the Internet the port 1883/TCP should be open/forward at your router/firewall, and you should add authentication to the MQTT broker.