Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 2: Database configuration and data collection

On the previous post we’ve installed and the base software for our Grafana based dash board.

We need now to configure our InfluxDB database and Node Red to start collecting data.

Configuring InfluxDB:
Detailed instructions for configuring an InfluxDB database are on this InfluxDB documentation link..

The main concepts that we need to be aware when using the InfluxDB is that record of data has a time stamp, a set of tags and a measured value. This allows, for example to create a value named Temperature and tag it depending on the source sensor:

Temperature: Value=22.1 , Sensor=Kitchen
Temperature: Value=21.9 , Sensor=Room1

This allows to process all the data or only process data based on a certain tag or tags. Values and tags can be created on the fly without previously define them, which is a bit different from standard RDBMS engines.

Creating an InfluxDB database:
To create the database, we need to access the machine hosting the InfluxDB server and execute the command influx:

odroid@odroid:~$ influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> create database SensorData
> show databases
name: databases
name
----
_internal
SensorData

> 

Now we have our database created and I’ve named SensorData. To make an example with the above temperature data we can do the following:

> insert Temperature,Sensor=kitchen value=22.1
ERR: {"error":"database is required"}

Note: error may be due to not setting a database or retention policy.
Please set a database with the command "use " or
INSERT INTO . 
> use SensorData
Using database SensorData
> 

As we can see we need first to select the database where we are going to insert data with the command use SensorData:

> use SensorData
Using database SensorData
> insert Temperature, Sensor=kitchen value=22.1
ERR: {"error":"unable to parse 'Temperature, Sensor=kitchen value=22.1': missing tag key"}

> insert Temperature,Sensor=kitchen value=22.1
> insert Temperature,Sensor=Room1 value=21.9
> select * from Temperature
name: Temperature
time                Sensor  value
----                ------  -----
1487939008959909164 kitchen 22.1
1487939056354678353 Room1   21.9

Note that we can’t use spaces between the Measure name and the tags. The correct syntax is as follows:

 insert MeasureName,tag1=t1,tag2=t2,...   value1=val1,value2=val2,value3=val3,....

Also note that no DDL (data definition language) was used to create the tags or the measured value, we’ve just inserted data for our measurement with the our tags and value(s) without the need of previously define the schema.

Configuring Node-Red
Since we now have a database we can configure the InfluxDB Node Red nodes to store data onto the database:

There are two types of InfluxDB nodes, one that has an Input and Output and other that only has Input. The former is for making queries to the database where we provide on the input node the query, and on the output the results are returned. The later is for storing data only onto the database.
For both nodes we need to configure an InfluxDB server:

InfluxDB Server Configuration

We need to press the Pen icon right next to the server to add or reconfigure a new InfluxDB server:

InfluxDB server

A set of credentials are required, but since I’ve yet configured security, we can just put admin/admin as username and password. In a real deployment we must activate security.

From now on it is rather simple. Referring to InfluxDB node configuration screenshot (Not the InfluxDB server configuration) we have a configuration field named Measurement. This is our measure name that we associate a value. Picking up on the above example with the Insert command it will be Temperature, for example.

Now if the msg.payload provided has input to the node is a single value, let’s say 21, this is equivalent to do:

Insert Temperature value=12

We other formats for msg.payload that allows to associate tags and measures. Just check the Info tab for the node.

Simple example:

The following flow shows a simple example of a value received through MQTT, in this case the free heap from one of my ESP8266 and its storage in InfluxDB:

Sample Flow

[{"id":"20bec5de.8881c2","type":"mqtt in","z":"ced40abb.3c92e","name":"Heap","topic":"/outbox/ESP12DASH/Heap","qos":"2","broker":"2a552b3c.de8d2c","x":83.16668701171875,"y":206.41668701171875,"wires":[["e0d9c912.8c57f8","876fb151.6f2fa"]]},{"id":"876fb151.6f2fa","type":"debug","z":"ced40abb.3c92e","name":"","active":true,"console":"false","complete":"false","x":408.5,"y":177,"wires":[]},{"id":"e0d9c912.8c57f8","type":"influxdb out","z":"ced40abb.3c92e","influxdb":"bbd62a93.1a7108","name":"","measurement":"heap","x":446.1666717529297,"y":224.58335876464844,"wires":[]},{"id":"2a552b3c.de8d2c","type":"mqtt-broker","broker":"192.168.1.17","port":"1883","clientid":"node-red","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":15,"cleansession":true,"willQos":"0","birthQos":"0"},{"id":"bbd62a93.1a7108","type":"influxdb","z":"","hostname":"127.0.0.1","port":"8086","protocol":"http","database":"SensorData","name":"ODroid InfluxDB"}]

We can see with this flow the data stored in InfluxDB:

> select * from heap;
name: heap
time                value
----                -----
1487946319638000000 41600
1487946440913000000 41600
1487946562206000000 41600
1487946683474000000 41600
1487946804751000000 41600
1487946926061000000 41600
1487947047309000000 41616
1487947168594000000 41600

Now we have data that we can graph with Grafana, subject of my next posts.

Node Red Dashboard and UPS Monitoring

Just a quick hack to use the Node Red dashboard to monitor some of the UPS values that is attached to My Synology NAS.

Gathering the data and feeding it to Node-Red
First I thought to do some sort of Python or NodeJS program to run the upsc command, process the output and feed it, through MQTT, to Node Red.
But since it seemed to me a bit of overkill to just process a text output, transform it to JSON and push it through MQTT by using a program, I decided that I’ll use some shell scripting, bash to be more explicit.

I’m running on my Odroid C1+ “server” all the necessary components, namely Node Red with the Dashboard UI module.

So on Odroid I also have the ups monitoring tools, and upsc outputs a text with the ups status:

odroid@odroid:~$ upsc ups@192.168.1.16
Init SSL without certificate database
battery.charge: 100
battery.charge.low: 10
...
input.transfer.high: 300
input.transfer.low: 140
input.voltage: 230.0
...
ups.load: 7
ups.mfr: American Power Conversion
...
ups.model: Back-UPS XS 700U  
...

So all we need now is to transform the above output from that text format to JSON and feed it to MQTT.
This means that we need to put between ” the parameter names and values, replace the : by , and also we need to replace the . on parameter names to _ so that in Node Red javascript we don’t have problems working with the parameter names.

Since I’m processing each line of the output, I’m using gawk/awk that allows some text processing. The awk program is as follow:

BEGIN {print "{"}
 {
   print lline  "\42" $1 "\42:\42"$2"\42"
 }
 {lline =", "}
END {print "}"}

This will at the beginning print the opening JSON bracket, then line by line the parameter name and value between ” and separated by : .
The lline variable at the first line is empty, so it prints nothing, but at the following lines it prints , which separates the JSON values.
We just need awk now to recognize parameters and values, and that is easy since they are separated by :

So if the above code is saved as procupsc.awk file, then the following command:

 upsc ups@192.168.1.16 2>/dev/null | awk -F: -f ~/upsmon/procupsc.awk |  sed 's/[.]/_/g'

Transforms the upsc output into a JSON output, including the replacement of . on variable names into _

{
"battery_charge":" 100"
, "battery_charge_low":" 10"
, "battery_charge_warning":" 50"
...
, "ups_load":" 7"
..
, "ups_vendorid":" xxxx"
}

Now all we need is to feed the output to the MQTT broker, and for this I’ll use the mosquitto_pub command, that has a switch that accepts the message from the standard input:

upsc ups@192.168.1.16 2>/dev/null | awk -F: -f /home/odroid/upsmon/procupsc.awk |  sed 's/[.]/_/g' | mosquitto_pub -h 192.168.1.17 -t upsmon -s

So we define the host and the topic: upsmon and the message is the output of the previous command (the -s switch).

All we need now is on Node Red to subscribe to the upsmon topic and process the received JSON object.

Since I’m running this periodically on crontab, I also add the PATH variable so that all files and commands are found.
The complete script is as follows:

upsmon.sh

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
upsc ups@192.168.1.16 2>/dev/null | awk -F: -f /home/odroid/upsmon/procupsc.awk |  sed 's/[.]/_/g' | mosquitto_pub -h 192.168.1.17 -t upsmon -s

and on Crontab:

# m h  dom mon dow   command
*/5 * * * * /home/odroid/upsmon/upsmon.sh

Node Red processing and visualization
On node red side, now is easy. We receive the above upsc JSON object as a string on msg.payload, and we use the JSON node to separate into different msg.# variables.
From here we just feed the data to charts and gauges. The code is:

[{"id":"603732e7.bb8464","type":"mqtt in","z":"a8b82890.09ca7","name":"","topic":"upsmon","qos":"2","broker":"2a552b3c.de8d2c","x":114.5,"y":91,"wires":[["193a550.2b0ea2b"]]},{"id":"f057bae4.8e4678","type":"debug","z":"a8b82890.09ca7","name":"","active":true,"console":"false","complete":"payload","x":630.5,"y":88,"wires":[]},{"id":"563ec2f5.6475e4","type":"ui_gauge","z":"a8b82890.09ca7","name":"UPS Load","group":"ba196e43.b35398","order":0,"width":0,"height":0,"gtype":"donut","title":"Load","label":"%","format":"{{value}}","min":0,"max":"100","colors":["#00b500","#e6e600","#ca3838"],"x":630.5,"y":173,"wires":[]},{"id":"dbdb6731.531e8","type":"function","z":"a8b82890.09ca7","name":"UPS_Load","func":"msg.payload = Number(msg.payload.ups_load);\nreturn msg;","outputs":1,"noerr":0,"x":372.5,"y":175,"wires":[["563ec2f5.6475e4","f057bae4.8e4678","57bc92ab.ce4234"]]},{"id":"193a550.2b0ea2b","type":"json","z":"a8b82890.09ca7","name":"To Json","x":129.5,"y":178,"wires":[["dbdb6731.531e8","5433b116.a93b4","52604271.71f43c","11ad64b4.f70ad3","589635b3.9bacc4"]]},{"id":"57bc92ab.ce4234","type":"ui_chart","z":"a8b82890.09ca7","name":"Ups Load/Time","group":"ba196e43.b35398","order":0,"width":0,"height":0,"label":"Load/Time","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","ymin":"0","ymax":"100","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"x":639.5,"y":226,"wires":[[],[]]},{"id":"5433b116.a93b4","type":"function","z":"a8b82890.09ca7","name":"Battery Status","func":"var Vbats = msg.payload.battery_voltage;\n\nvar Vbat = Vbats.replace(\"_\",\".\");\n\nmsg.payload = Number(Vbat);\n\nreturn msg;","outputs":1,"noerr":0,"x":400.5,"y":336,"wires":[["8c6f269f.87b618"]]},{"id":"52604271.71f43c","type":"function","z":"a8b82890.09ca7","name":"V IN","func":"var Vins = msg.payload.input_voltage;\n\nvar Vin = Vins.replace(\"_\",\".\");\n\nmsg.payload = Number(Vin);\nreturn msg;","outputs":1,"noerr":0,"x":369.5,"y":570,"wires":[["240f6e31.7aa5da","b37dffd2.1cb0d"]]},{"id":"8c6f269f.87b618","type":"ui_gauge","z":"a8b82890.09ca7","name":"","group":"421e19.192041e8","order":0,"width":0,"height":0,"gtype":"gage","title":"Curr. Bat. Voltage","label":"V. Bat","format":"{{value}}","min":"11","max":"15","colors":["#b50000","#e6e600","#00b500"],"x":616.5,"y":332,"wires":[]},{"id":"240f6e31.7aa5da","type":"ui_gauge","z":"a8b82890.09ca7","name":"Input AC Voltage","group":"4e74439e.ee7e74","order":0,"width":0,"height":0,"gtype":"gage","title":"VIN AC","label":"V AC","format":"{{value}}","min":"190","max":"240","colors":["#00b500","#e6e600","#ca3838"],"x":660.5,"y":569,"wires":[]},{"id":"11ad64b4.f70ad3","type":"function","z":"a8b82890.09ca7","name":"Bat Runtime","func":"msg.payload = Number(msg.payload.battery_runtime);\nreturn msg;","outputs":1,"noerr":0,"x":397.5,"y":397,"wires":[["98292081.f4f718","62bb1456.0f45ec"]]},{"id":"98292081.f4f718","type":"ui_gauge","z":"a8b82890.09ca7","name":"UPS Level","group":"fccb6f27.7691d8","order":0,"width":0,"height":0,"gtype":"wave","title":"UPS Runtime","label":"UPS Level","format":"{{value}}","min":0,"max":"1500","colors":["#00b500","#e6e600","#ca3838"],"x":642.5,"y":391,"wires":[]},{"id":"589635b3.9bacc4","type":"function","z":"a8b82890.09ca7","name":"Bat Charge","func":"var Vcharges = msg.payload.battery_charge;\n\nvar Vcharge = Vcharges.replace(\"_\",\".\");\n\nmsg.payload = Number(Vcharge);\n\n\nreturn msg;","outputs":1,"noerr":0,"x":405,"y":495,"wires":[["d2ab2543.d53a68"]]},{"id":"d2ab2543.d53a68","type":"ui_gauge","z":"a8b82890.09ca7","name":"Battery Charge","group":"421e19.192041e8","order":0,"width":0,"height":0,"gtype":"gage","title":"Battery Charge","label":"%","format":"{{value}}","min":0,"max":"100","colors":["#ff0000","#e6e600","#00ff01"],"x":661.5,"y":494,"wires":[]},{"id":"62bb1456.0f45ec","type":"ui_chart","z":"a8b82890.09ca7","name":"","group":"fccb6f27.7691d8","order":0,"width":0,"height":0,"label":"Runtime (sec)","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"x":625.5,"y":441,"wires":[[],[]]},{"id":"b37dffd2.1cb0d","type":"ui_chart","z":"a8b82890.09ca7","name":"Vin/Time","group":"4e74439e.ee7e74","order":0,"width":0,"height":0,"label":"VAC In/Time","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","ymin":"190","ymax":"240","removeOlder":"12","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"x":635.5,"y":641,"wires":[[],[]]},{"id":"2a552b3c.de8d2c","type":"mqtt-broker","broker":"192.168.1.17","port":"1883","clientid":"node-red","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":15,"cleansession":true,"willQos":"0","birthQos":"0"},{"id":"ba196e43.b35398","type":"ui_group","z":"","name":"UPS Load","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"421e19.192041e8","type":"ui_group","z":"","name":"UPS Battery","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"4e74439e.ee7e74","type":"ui_group","z":"","name":"Input Voltage","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"fccb6f27.7691d8","type":"ui_group","z":"","name":"UPS Runtime","tab":"61ec3881.53526","disp":true,"width":"6"},{"id":"61ec3881.53526","type":"ui_tab","z":"","name":"UPS","icon":"dashboard","order":2}]

The final output is as follow:

Node Red UPS Monitoring

Node Red UPS Monitoring

MQTT Mosquitto broker – Client Authentication and Client Certificates

After seeing how to set up transport layer security for the Mosquitto MQTT broker by using the Transport layer security on this post, we need to see how to setup client authentication (only authorized clients can connect to the broker) either by using the common user/password based authentication method or using client certificates.

Authentication by user and password:
First let’s enable authentication to the broker by setting up user and password authentication. For enabling this kind of authentication we need to modify the broker configuration (/etc/mosquitto/mosquitto.conf in my case) file and change the following entries:

allow_anonymous false
password_file /etc/mosquitto/passwd_mqtt

Before restarting the MQTT broker we need to add some users to the passwd_mqtt file with the command mosquitto_passwd:

mosquitto_passwd -c passwd_mqtt user1

The -c parameter is for creating the initial password file, if doesn’t exist, otherwise it will overwrite it!. For adding new users or update the passwords just run the command without the -c parameter.

After restarting the broker will should not be able to logon anonymously:

# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d
Client mosqpub/25688-pcortex sending CONNECT
Client mosqpub/25688-pcortex received CONNACK
Connection Refused: not authorised.
Error: The connection was refused.

But providing the user name and password:

# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d -u user1 -P secret
Client mosqpub/25709-pcortex sending CONNECT
Client mosqpub/25709-pcortex received CONNACK
Client mosqpub/25709-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (7 bytes))
Client mosqpub/25709-pcortex sending DISCONNECT 

For subscription, the same has to be done, again by providing an user and password:

# mosquitto_sub -t \$SYS/broker/bytes/\# -v --cafile /etc/mosquitto/certs/ca.crt -p 8883 -u user1 -P secret
$SYS/broker/bytes/received 217
$SYS/broker/bytes/sent 20

Authentication by using client certificates
Using client certificates, signed by a certificate authority, assures the client identity. The certificate authority used must be the same used by the server certificates and is only supported over TLS/SSL.
For using client certificates for authentication, we need to change the listener configuration for TLS/SSL by adding the following directives:

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/hostname.crt
keyfile /etc/mosquitto/certs/hostname.key
require_certificate true
use_identity_as_username true

The require_certificate directive with the value true means that clients must now provide a client certificate to connect.
The use_identity_as_username means that the user name of the connecting user is taken from the CN (Common Name) property of the certificate, otherwise we still need to provide an user and password.

With the above new configuration, we now can’t access the broker with user/password token:

mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d -u user1 -P secret
Client mosqpub/26549-pcortex sending CONNECT
Error: A TLS error occurred.

the result on the log file is:

1478537500: New connection from 127.0.0.1 on port 8883.
1478537500: OpenSSL Error: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
1478537500: Socket error on client , disconnecting.

For accessing now the broker we must provide a client certificate and a private key. The client certificate must be generated from the same CA (Certificate authority) that created the server certificate, otherwise, the client certificate can’t be validated, and the connection fails:

1478537629: New connection from 127.0.0.1 on port 8883.
1478537629: OpenSSL Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
1478537629: Socket error on client , disconnecting.

So on the same directory where the ca.crt and ca.key are residing execute the script to create the client certificate. For example for creating a client certificate for user1, we need to execute the following command:

# ./gen_CA.sh client user1

And we should have three files, two of them the user1.crt, the user certificate, and the user1.key, the user1 private key.

We can now logon to the broker:

# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d  --cert user1.crt --key user1.key 
Client mosqpub/30264-pcortex sending CONNECT
Client mosqpub/30264-pcortex received CONNACK
Client mosqpub/30264-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (7 bytes))
Client mosqpub/30264-pcortex sending DISCONNECT

If we check the log, we have the following:

1478539830: New client connected from 127.0.0.1 as mosqpub/27159-pcortex (c1, k60, u'user1').
1478539830: Client mosqpub/27159-pcortex disconnected.
1478601507: New connection from 127.0.0.1 on port 8883.

We can see that the broker extracted the username from the certificate property CN. If we don’t use the use_identity_as_username as true we need to provide the username and password and the client certificate. In this case, the certificate is validated, and the user used to logon might not be the same as the one defined on the CN certificate property. So without this directive we either need to allow anonymous logon again or define user and passwords.

Additional thoughts:
With user and password authentication we can revoke access to an user, by deleting it from the password file or changing the password.
But what about the authentication based on client certificates? As long the certificate is valid, the user can logon at will, since the broker will always accept it until the end of the certificate validation date.
The solution for this is to revocate the client certificate so when it is used, the broker rejects it. For this functionality, most Certificate Authorities provide the revocation list by providing either by CRL (Certificate Revocation list) file, or by OSCP (Online Status Certificate Protocol), and the server checks the client certificate on this list before allowing access. Mosquitto broker only works with CRL files.

We need to modify the listener configuration to verify the client certificates if it was revocated or not:

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/hostname.crt
keyfile /etc/mosquitto/certs/hostname.key
require_certificate true
use_identity_as_username true
crlfile /etc/mosquitto/certs/ca.crl

The issue now is that the script that we are using for generating certificates doesn’t generate CRLs, and so we can’t revocate certificates by using this script.

We can do it by hand, but that wouldn’t make our life much easier, since revocating certificates means that the CA server must now track all certificates and their status.
One possible solution is to use scripts that allow to create (a not very secure) CA: easy-ca. These scripts will replace the genCA.sh script and allow the revocation of certificates.

Getting the easy-CA scripts:

git clone https://github.com/fcgdam/easy-ca.git

I’m omitting a lot of output that these scripts do, namely asking for the Organization name, CA passwords and so on.

Somewhere where we find suitable we create our CA:

Generating the CA:

# cd ~ 
# ~/easy-ca/create-root-ca -d CA
# cd CA

Then at the CA directory we created, we can now create our broker certificate and user certificate. Make sure that the names make sense.

Generating the broker certificate:

# bin/create-ssl -s brokername
# bin/create-client -c user1

At the end we have the following files:

ca/ca.crt

We now have all the needed files for setting up our broker with support for CRL.

# sudo -s
# cp ~/CA/ca/ca.crt /etc/mosquitto/certs
# cp ~/CA/crl/ca.crl /etc/mosquitto/certs
# cp ~/CA/certs/brokername.server.crt /etc/mosquitto/certs/brokername.crt
# cp ~/CA/private/brokername.server.key /etc/mosquitto/certs/brokername.key

And restarting the broker we have the new certificates in place.
We can test now by using our CA user certificate:

# cd ~/CA
# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d  --cert certs/user1.client.crt --key private/user1.client.key 
Client mosqpub/7424-pcortex sending CONNECT
Client mosqpub/7424-pcortex received CONNACK
Client mosqpub/7424-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (7 bytes))
Client mosqpub/7424-pcortex sending DISCONNECT

It works. Now let’s revoke the user certificate:

# bin/revoke-cert -c certs/user1.client.crt 

Revoking certificate 'certs/user1.client.crt'

Reason for revocation: 

1. unspecified
2. keyCompromise
3. CACompromise
4. affiliationChanged
5. superseded
6. cessationOfOperation
7. certificateHold

Enter 1-7 [1]: 2
You are about to revoke this certificate with reason 'keyCompromise'.
Are you SURE you wish to continue? [y/N]: y
Using configuration from conf/ca.conf
Revoking Certificate 02.
Data Base Updated
Using configuration from conf/ca.conf

Server certificate revoked.

We need to copy the new CRL to the directory where mosquitto expects the CRL:

# sudo cp crl/ca.crl /etc/mosquitto/certs

And after restarting the broker we have:

mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d  --cert certs/user1.client.crt --key private/user1.client.key  
Client mosqpub/7640-pcortex sending CONNECT
Error: A TLS error occurred.

And on the logs:

1478625909: OpenSSL Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
1478625909: Socket error on client , disconnecting.
1478625910: New connection from 127.0.0.1 on port 8883.

MQTT Mosquitto broker with SSL/TLS transport security

Just a quick note in setting up transport layer security on the MQTT Mosquitto broker for both supported protocols: MQTT and WebSockets.

There are several posts on the web regarding this, namely:

SSL Client certs to secure mqtt and Mosquitto websocket support

Those posts explain more or less what is needed to be done to have TLS/SSL transport security. These are just my notes:

Generating the server certificates:
This can be quite easily accomplished by using the following script: https://github.com/owntracks/tools/blob/master/TLS/generate-CA.sh.
This script will generate a self signed certificate to be used by Mosquito for providing TLS for the MQTT and WebSocket protocol. All that is needed to run the script is to have openssl installed on your Linux machine.

If the script is called without parameters, it will generate a self signed certificate for the hostname where the script is running. Otherwise a we can pass a hostname as the first parameter to the script.

After running the script, the following files are generated:

  1. ca.crt – The CA (Certificate Authority, who published the host certificate) public certificate.
  2. hostname.crt – The hostname, that will run the mosquitto broker, public certificate.
  3. hostname.key – The hostname private key.

After having these files, we need to configure the Mosquitto Broker to use them.

Mosquitto configuration:
To configure the Mosquito broker we need first to copy the certificates and key files to a known directory. We will create a certs directory under /etc/mosquitto:

sudo -s
mkdir -p /etc/mosquitto/certs
cp ca.crt /etc/mosquitto/certs
cp hostname.* /etc/mosquitto/certs

After this we can modify the mosquitto configuration file. One important thing to keep in mind is that lines must be following each other without blank lines after the listener directive.

So:

# Plain MQTT protocol
listener 1883

# End of plain MQTT configuration

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/hostname.crt
keyfile /etc/mosquitto/certs/hostname.key

# End of MQTT over TLS/SLL configuration

# Plain WebSockets configuration
listener 9001
protocol websockets

# End of plain Websockets configuration

# WebSockets over TLS/SSL
listener 9883
protocol websockets
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/hostname.crt
keyfile /etc/mosquitto/certs/hostname.key

We will make one more change, but restart mosquitto broker now and do some testing.

Testing MQTT TLS/SSL configuration
We can use Mqtt-Spy to subscribe to our defined test topic: test. We can use plain MQTT or use MQTT over TLS/SSL:

MQTT Spy simple TLS configuration

MQTT Spy simple TLS configuration

We can use then the MQTT spy tool to publish or subscribe MQTT topics.

By command line, the mosquitto_sub and mosquitto_pub only worked if the port number for MQTTS is provided, otherwise it gives a TLS error:

mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883

mosquitto_sub -t \$SYS/broker/bytes/\# -v --cafile /etc/mosquitto/certs/ca.crt -p 8883

This should work without any issues.

Testing MQTT websockets over TLS/SSL configuration
The issue with this testing is that we are using a self signed certificate, so only useful for local, restricted, testing.
Before we can use the MQTT websockets with TLS/SSL enabled we need to use the browser and visit the following URL:

  https://MQTT_BROKER_IP_OR_HOSTNAME:9883/mqtt

Note that we are using HTTPS. When connecting to the above URL, the browser should complain about the insecure connection, due to the self signed certificate, and we need to add an exception and always accept that certificate. After that the error should be something like connection reset or failed to load page. This is normal, since the browser won’t upgrade the connection to a web socket.
We can now use the Hive MQTT Websockets Client to test our connection, and it should work fine (Note the connected green icon and SSL is selected):
Hive MQTT WebSocket client

Forcing TLSv1.2
All this work of enabling TLS/SSL on the Mosquitto Broker is needed, since most IoT clouds that have MQTT interface need that the connection is over TLS/SSL. More specifically AWS IoT cloud needs the connection to be protected by TLS/SSL, but that connection must be only on version 1.2 of the TLS protocol. AWS IoT cloud also requires client authentication through client certificates, but we are not dealing with this part on this post.

So we are now configuring our Mosquitto broker to only accept TLSv1.2 connections. To do that we modify the mosquitto.conf file and add the following line:

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/hostname.crt
keyfile /etc/mosquitto/certs/hostname.key
tls_version tlsv1.2

# WebSockets over TLS/SSL
listener 9883
protocol websockets
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/hostname.crt
keyfile /etc/mosquitto/certs/hostname.key
tls_version tlsv1.2

and restart the broker.

Testing TLS V1.2
We can specify the TLS version on the mosquitto command line utils:

[pcortex@pcortex:~]$ mosquitto_pub --cafile ./ca.crt --tls-version tlsv1.2 -h localhost -t "test" -m "mes" -p 8883 -d
Client mosqpub/26994-pcortex sending CONNECT
Client mosqpub/26994-pcortex received CONNACK
Client mosqpub/26994-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (3 bytes))
Client mosqpub/26994-pcortex sending DISCONNECT
[pcortex@pcortex:~]$ mosquitto_pub --cafile ./ca.crt --tls-version tlsv1.1 -h localhost -t "test" -m "m3224" -p 8883 
Error: A TLS error occurred.

As we can see the lower versions of the TLS protocol are now not accepted.
The Websockets client should work without any issues.

Conclusion:
This configuration only solves the transport security, not the authentication security. The later can be accomplished by using the username/password process or using client certificates, which is the process that Amazon AWS IoT cloud uses. But those are topics for other posts. Edit: Follow-up at: Client authentication

MQTT Dashboards

The freeboard.io dashboard is one of the most popular dashboards to build an interface for displaying data. Still, for supporting the MQTT protocol some configuration is needed that is not exactly straight forward. Also if don’t want to use MQTT but web sockets, a similar not straight forward configuration is also needed. After that configuration (MQTT support for freeboard.io and WebSocket support for freeboard.io) we can add MQTT and Web Sockets datasources to our dashboards, and we get a nice customable dashboard framework.

Anyway there are some nice alternatives available that are worth some attention for someone that wants a simpler way of doing things:

CroutonCrouton MQTT Dashboard
Node Red UINode-Red-ui

Crouton
This dashboard is really nice since it has something that we might call as dashboard auto-configuration. We only need to configure the MQTT broker (with web sockets enabled, off course) and add device names to the configuration list. The device name is very important because it will be used in MQTT topic subscriptions (see below).

The Crouton dashboard configuration is stored on the browser local storage, so it’s not central. This means, that if you change browser or machine the configuration needs to be done again for that specific instance of browser and/or machine. Also, based on my experience, this is rather fidly, sometime it saves, other times it won’t, and need to reconfigure everything again after closing and opening the browser again.

The Crouton MQTT configuration is simple:

Crouton MQTT Configuration

MQTT Configuration

After the dashboard is connected to the MQTT broker, it searchs for specific topics based on the names configured on the Devices list. For each device it publishes to topic /inbox/devicename and subscribes to /outbox/devicename. At the initialization phase the Dashboard sends a get message to /inbox/devicename/deviceInfo. We can see that the device name is used to “find” a MQTT subscriber that answers to this topic. The subscriber that “owns” the topic /inbox/devicename answers to the get message with a JSON description of the device parameters to be shown at the dashboard on the /outbox/devicename/deviceInfo topic.

For example, on the following configuration:

Device Configuration

Device Configuration

The Crouton Dashboard sends to /inbox/Odroid/deviceInfo (Case Sensitive!!) and to /inbox/DiskStation/deviceInfo the message get and waits for a specific JSON response on /outbox/Odroid/deviceInfo and /outbox/DiskStation/deviceInfo to know how to automatically create the dashboard. With another words, the JSON response defines the dashboard. This means that to add a new metric, for example, we just need to add on the device the new metric and report it, no need to mess again with the dashboard configuration.

Auto Generated MQTT Dashboard

We can now use the mouse to move around the tiles, but the it seems also that this configuration isn’t stored.

This is the neat part of Crouton Dashboard, this auto-configuration.

The configuration is defined on the JSON response to /outbox/deviceName/deviceInfo MQTT topic. The JSON object defines the number of metrics and display format that device publishes and automatically associated to the metric name there is the MQTT topic /outbox/deviceName/Metric that the dashboard listens to.

On the Crouton GitHub there are two examples of this, one in Python and other in Lua for the ESP8266 NodeMCU based firmware.

I have a simpler one in Python, derived from the original Python example, for retrieving and send metrics from my Odroid SBC. The code is available here.

On the example note two things:

At the all.py file line 9, the clientName variable is set to Odroid. This is the device name to set on the Crouton dashboard.

Then for example, I publish my root disk space with the following JSON object segment:

        ....
        ....
        "endPoints": {
            "RootDisk": {
                "values": {
                    "labels": [],
                    "series": [0]
                },
                "total": 100,
                "centerSum": true,
                "units": "%",
                "card-type": "crouton-chart-donut",
                "title": "Root Disk"
            },....
            .....

The above configuration means that the end point /outbox/Odroid/RootDisk will have some value published that should be shown as a doughnut chart with % units.

A bit below on the code, there is an infinite loop, that each 30 seconds publishes to the topic the available disk space in percentage:

   client.publish("/outbox/"+clientName+"/RootDisk" , '{"series":['+str(disk_usage(rootPath))+']}')

As a conclusion, this Dashboard is rather interesting and worth a look, even with it’s, for now, issues with saving the configuration.

Node-Red-UI
While the Crouton Dashboard is an standalone Node.JS based application, the Node-Red-UI is an extension the Node-Red itself. It adds a new range of UI nodes to the Node-Red available nodes that allows to also build a Dashboard, but this time, not limited to MQTT. In fact any node that can inject data into a node-red-ui works.

The UI is available on the same address as Node-Red, but at the path /ui: Example:http://localhost:1880/ui/

For example, the following flow:

Node-Red-UI Flow

Node-Red-UI Flow

Generates the following UI:

Node-Red-UI

Node-Red-UI

The UI is based on the same MQTT topics and data published by my Crouton Python Client. The only thing needed is to extract the data from the MQTT message:

var space = msg.payload.series[0];
var msg ={"value":space , "payload":space};
return msg;

And the data feed from the “agents” work on both dashboards.

Anyway, regarding Node-Red-UI, it is an all other world, and I recommend to join and/or read the discussion group for this project.

ODroid – Mosquitto MQTT Broker install

Just for a quick reference, the following instructions detail how to install the latest Mosquitto MQTT broker with Websockets enabled on the ODroid C1+ running (L)Ubuntu release. The instructions are probably also valid for other platforms, but not tested.

1. Install the prerequisites
As the root user, install the following libraries:

apt-get update
apt-get install uuid-dev libwebsockets-dev

Probably the SSL libraries, and a few others are also needed, but in my case they where already installed.

2. Mosquitto install
Download and compile the Mosquitto broker:

mkdir ~/mosq
cd ~/mosq
wget http://mosquitto.org/files/source/mosquitto-1.4.5.tar.gz
tar xvzf mosquitto-1.4.5.tar.gz
cd mosquitto-1.4.5/

Edit the config.mk file to enable the websockets support:

# Build with websockets support on the broker.
WITH_WEBSOCKETS:=yes

and compile and install:

make
make install

3. Configuration
Copy the file mosquitto.conf to /usr/local/etc, and edit the file:

cp mosquitto.conf /usr/local/etc
cd /usr/local/etc

Add at least the following lines to mosquitto.conf file to enable websockets support.

# Port to use for the default listener.
#port 1883
listener 1883
listener 9001
protocol websockets

Add an operating system runtime user for the Mosquitto daemon:

useradd -lm mosquitto
cd /usr/local/etc
chown mosquitto mosquitto.conf

If needed, or wanted, change also on the configuration file the logging level and destination.
For example:

# Note that if the broker is running as a Windows service it will default to
# "log_dest none" and neither stdout nor stderr logging is available.
# Use "log_dest none" if you wish to disable logging.
log_dest file /var/log/mosquitto.log

# If using syslog logging (not on Windows), messages will be logged to the
# "daemon" facility by default. Use the log_facility option to choose which of
# local0 to local7 to log to instead. The option value should be an integer
# value, e.g. "log_facility 5" to use local5.
#log_facility

# Types of messages to log. Use multiple log_type lines for logging
# multiple types of messages.
# Possible types are: debug, error, warning, notice, information, 
# none, subscribe, unsubscribe, websockets, all.
# Note that debug type messages are for decoding the incoming/outgoing
# network packets. They are not logged in "topics".
log_type error
log_type warning
log_type notice
log_type information

# Change the websockets logging level. This is a global option, it is not
# possible to set per listener. This is an integer that is interpreted by
# libwebsockets as a bit mask for its lws_log_levels enum. See the
# libwebsockets documentation for more details. "log_type websockets" must also
# be enabled.
websockets_log_level 0

# If set to true, client connection and disconnection messages will be included
# in the log.
connection_messages true

# If set to true, add a timestamp value to each log message.
log_timestamp true

4. Automatic start
The easiest way is to add the following file to the init.d directory and link it to the current runlevel:

Create the file named mosquitto under /etc/init.d

#!/bin/bash

case "$1" in
    start)
        echo "Starting Mosquitto MQTT Broker"
        touch /var/log/mosquitto.log
        chown mosquitto /var/log/mosquitto.log
        su - mosquitto -c "/usr/local/sbin/mosquitto -c /usr/local/etc/mosquitto.conf" & > /dev/null 2>/dev/null
        ;;
    stop)
        echo "Stopping Mosquitto MQTT Broker"
        killall mosquitto
        ;;
    *)
        echo "Usage: /etc/init.d/mosquitto start|stop"
        exit 1
        ;;
esac

exit 0

Find out the current run level. On Odroid it seems that is level 2:

root@odroid:/etc/init.d# runlevel
N 2
root@odroid:/etc/init.d# 

And link the automatic start for Mosquitto broker at run level 2:

cd /etc/rc2.d
ln -s  /etc/init.d/mosquitto S97mosquitto

And that’s it.

We can start manually the broker with the command /etc/init.d/mosquitto start, and stop it with the mosquitto stop command.

Sming – MQTT (part 2)

While doing some test with MQTT using the Sming Framework for the ESP8266 I had some issues that are worthy to be documented.
The Sming Framework uses the libemqtt library for MQTT support, and it seems it works fine. Connection, subscribing and publishing work out of the box, and one important thing, the keep alive mechanism also works. We can see how MQTT can be used with the Sming framework in here.

But what is this keep alive mechanism on the MQTT protocol? The MQTT protocol specification defines this:

It is the responsibility of the Client to ensure that the interval between Control Packets being sent does not exceed the Keep Alive value. In the absence of sending any other Control Packets, the Client MUST send a PINGREQ Packet.

This means that before a keep alive time is reached the client must do a request to the MQTT broker, and in case it doesn’t, it should send a PINGREQ.

For example, on a Mosquitto broker with logs enabled I can have this:

1450189246: New client connected from 192.168.1.200 as ESP8266 (c1, k10)

It shows that a client connected from 192.168.1.200 with the client identification ESP8266 and it defined as the keep alive interval 10s.

This means that at least before a ten second period is reached, the esp8266 MQTT client should send a PINGREQ or do something, like publishing to a topic.

The above requirements are probably the root of my problems and the fact that the ESP8266 chip/firmware can’t do too many things at once….

The test that I was doing was that I didn’t published nothing from the ESP8266 to the MQTT broker. All I was doing was to subscribe some MQTT topic and print the messages on the I2C 16×02 LCD that I have connected to the ESP8266.

Without any activity the esp8266 stayed connected to broker without any problems.

If I published something to the topic subscribed by the esp8266, it worked fine.

But if I published something massively to the topic subscribed, then the broker would drop the connection:

1450190071: Socket error on client ESP8266, disconnecting.

And that was a bit odd…

So at a first approach, I added a check timer to see if the MQTT connection was valid, and to reconnect if it wasn’t:

// Connect to the MQTT broker.
// If BRUSER is not empty, connect with user and password.
void startMqttClient()
{
    Serial.println("Mqtt server connect & subscribe..");   
    String mqttId = "esp8266-" + String(system_get_chip_id());

    if ( strlen( BRUSER ) == 0 )   
        mqtt.connect( mqttId );
    else
        mqtt.connect( mqttId, BRUSER , BRPWD );
    
    // Let's subscribe topics    
    mqtt.subscribe("/data");
    
}

void checkMQTT() {
        if (mqtt.getConnectionState() != eTCS_Connected) {
            Serial.println("MQTT Server: Lost connection... ");
            Serial.println("MQTT Server: Reconnect");
            startMqttClient(); // Auto reconnect
        }
}

And the timer that is defined to check the MQTT connection:

Timer checkTimer;

void startMain() {
  //...
  //...
  
  startMqttClient();  // Connect to the MQTT broker.
  checkTimer.initializeMs( 60 * 1000 , checkMQTT).start();
}

So every minute the connection to the broker is checked, and the reconnection, if needed, happens automatically.

But this solution doesn’t explain why the connection is lost when the burst of messages arrive at the topic.

Anyway to keep the long story short, the burst duration of messages where superior to the keep alive interval. Since that during that period I didn’t publish nothing, AND, the Mqtt library was busy to receive the incoming data burst, the broker assumed that the esp8266 was dead, and drop the connection.

Since the default keep alive was 10s, that value was too short to keep the connection stable under load.

The real solution is to increase the MQTT keep alive interval, to something that is bigger that any duration of probable data bursts. So what is needed is to specify a higher keep alive interval:

void startMqttClient()
{
    Serial.println("Mqtt server connect & subscribe..");   
    String mqttId = "esp8266-" + String(system_get_chip_id());

    mqtt.setKeepAlive(180);  // Keep alive of 180 seconds...

    if ( strlen( BRUSER ) == 0 )   
        mqtt.connect( mqttId );
    else
        mqtt.connect( mqttId, BRUSER , BRPWD );
    
    // Let's subscribe topics    
    mqtt.subscribe("/data");
    
} 

And with this change we got at the broker:

1450190936: New client connected from 192.168.1.200 as ESP8266 (c1, k180)

I can now burst messages to the esp8266 MQTT client without the connection being dropped.

So as long that incoming burst of messages are under 180 seconds ( which is easy) or some publishing from the ESP8266 happens, the connection stays up. Anyway, if it fails, the check timer will reconnect again to the broker.