Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 1: Installing

A more or less standard software stack used for control, processing and displaying data, has emerged that is almost used by everyone when hacking around on Arduinos, ESP8266, Raspeberry Pi’s and other plethora of devices. This “standard” software stack basically always includes the MQTT protocol, some sort of Web based services, Node-Red and several different cloud based services like Thingspeak, PubNub and so on. For displaying data locally, solutions like Freeboard and Node-Red UI are a great resources, but they only shows current data/status, and has no easy way to see historical data.

So on this post I’ll document a software stack based on Node-Red, InfluxDB and Graphana that I use to store and display data from sensors that I have around while keeping and be able to display historical memory of data. The key asset here is the specialized time-series database InfluxDB that keeps data stored and allows fast retrieval based on time-stamps: 5 minutes ago, the last 7 days, and so on. InfluxDB is not the only Time-Series database that is available, but it integrates directly with Grafana the software that allows the building of dashboards based on stored data.

I’m running an older version of InfluxDB on my ARM based Odroid server, since a long time ago, ARM based builds of InfluxDB and Grafana where not available. This is now not the case, but InfluxDB and Grafana have ARM based builds so we can use them on Raspberry PI and Odroid ARM based boards.

So let’s start:

Setting up Node-Red with InfluxDB
I’ll not detail the Node-Red installation itself since it is already documented thoroughly everywhere. To install the supporting nodes for InfluxDB we need to install the package node-red-contrib-influxdb

cd ~/.node-red
npm install  node-red-contrib-influxdb

We should now restart Node-red to assume/detect the new nodes.

Node Red InfluxDB nodes

Installing InfluxDB
We can go to the InfluxDB downloads page and follow the installation instructions for our platform. In my case I need the ARM build to be used on Odroid.

cd ~
wget https://dl.influxdata.com/influxdb/releases/influxdb-1.2.0_linux_armhf.tar.gz
tar xvzf influxdb-1.2.0_linux_armhf.tar.gz

The InfluxDB engine is now decompressed in the newly created directory influxdb-1.2.0-1. Inside this directory there are the directories that should be copied to the system directories /etc, /usr and /var:

sudo -s
cd /home/odroid/influxdb-1.2.0-1

Copy the files to the right location. I’ve added the -i switch just to make sure that I don’t overwrite nothing.

root@odroid:~/influxdb-1.2.0-1# cp -ir etc/ /etc
root@odroid:~/influxdb-1.2.0-1# cp -ir usr/* /usr
root@odroid:~/influxdb-1.2.0-1# cp -ir var/* /var

We need now to create the influxdb user and group:

root@odroid:~/influxdb-1.2.0-1# groupadd influxdb
root@odroid:~/influxdb-1.2.0-1# useradd -M -s /bin/false -d /var/lib/influxdb -G influxdb influxdb

We need now to change permissions on /var/lib/influxdb:

cd /var/lib
chown influxdb:influxdb influxdb

We can now set up the automatic start up script. On the directory /usr/lib/influxdb/scripts there are scripts for the systemctl based Linux versions and init.d based versions that is my case. So all I have to do is to copy the init.sh script from that directory to the /etc/init.d and link it to my run level:

root@odroid:~# cd /etc/init.d
root@odroid:/etc/init.d# cp /usr/lib/influxdb/scripts/init.sh influxdb
root@odroid:/etc/init.d# runlevel
 N 2
root@odroid:/etc/init.d# cd /etc/rc2.d
root@odroid:/etc/init.d# ln -s /etc/init.d/influxdb S90influxdb

And that’s it. We can now start the database with the command /etc/init.d/influxdb start

root@odroid:~# /etc/init.d/influxdb start
Starting influxdb...
influxdb process was started [ OK ]

We can see the influxdb logs at /var/log/influxdb and start using it through the command line client influx:

root@odroid:~# influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> show databases
name: databases
name
----
_internal

> 

Installing Grafana
We need now to download Grafana. In my case for Odroid since it is an ARMv7 based processor, no release/binary is available.
But a ARM builds are available on this GitHub Repository: https://github.com/fg2it/grafana-on-raspberry for both the Raspberry Pi and other ARM based computer boards, but only for Debian/Ubuntu based OS’s. Just click on download button on the description for the ARMv7 based build and at the end of the next page a download link should be available:

odroid@odroid:~$ wget https://bintray.com/fg2it/deb/download_file?file_path=main%2Fg%2Fgrafana_4.1.2-1487023783_armhf.deb -O grafana.deb

And install:

root@odroid:~# dpkg -i grafana.deb
Selecting previously unselected package grafana.
(Reading database ... 164576 files and directories currently installed.)
Preparing to unpack grafana.deb ...
Unpacking grafana (4.1.2-1487023783) ...
Setting up grafana (4.1.2-1487023783) ...
Installing new version of config file /etc/default/grafana-server ...
Installing new version of config file /etc/grafana/grafana.ini ...
Installing new version of config file /etc/grafana/ldap.toml ...
Installing new version of config file /etc/init.d/grafana-server ...
Installing new version of config file /usr/lib/systemd/system/grafana-server.service ...

Set the automatic startup at boot:

root@odroid:~# ln -s /etc/init.d/grafana-server /etc/rc2.d/S91grafana-server

And we can now start the server:

root@odroid:~# /etc/init.d/grafana-server start
 * Starting Grafana Server    [ OK ] 
root@odroid:~# 

We can now access the server at the address: http://server:3000/ where server is the IP or DNS name of our ODroid or RPi.

Conclusion:
This ends the installation part for the base software.

The following steps are:

  • Create the Influx databases
  • Receive data from sensors/devices and store it on the previously created database
  • Configure and create Grafana data sources and dashboards
  • Add some plugins to Grafana

CubicSDR and SoapyRemote on Orange PI PC

So I’m using remotely my RTL-SDR dongle connected to an Orange PI PC using GQrx on my desktop computer while on the Orange Pi PC I’m running the rtl_tcp server.

This combination works fine, but still some times I have some lag, not common, but it happens.

Anyway Gqrx is great, but I wanted to try another SDR programs, and one of those programs is Cubic SDR. Cubic SDR uses an abstration layer for accessing the SDR hardware either locally connected or over the network. So I wanted to see how the behaviour of CubicSdr, comparing to Gqrx when accessing the RTLSDR over the network.

Installing CubicSDR on the Desktop
I’ve not used the available binaries, but I’ve used the code from the Git repository: https://github.com/cjcliffe/CubicSDR. So far the git repository works fine.

I’m not posting here the instructions for building the CubicSDR because the full instructions are at the Cubic SDR wiki.

Just make sure, when compiling, to give the correct path to wxWidgets when building Cubic SDR.

So at the desktop, we need to obtain, build and install the following components:

  • SoapySDR – Abstraction layer
  • Liquid-dsp – The digital signal processing libs
  • wxWidgets – The display widgets
  • CubicSDR – The Sdr program itself
  • SoapyRTLSDR – The Soapy abstraction layer driver for the RTLSDR USB type dongles. To use the dongles locally

Installing the above software we can use the attached locally RTLSDR dongle.

Installing SoapySDR Remote on the remote server
The remote server where my RTLSDR dongle is connected is an Orange Pi PC running Armbian. To let my desktop running CubicSDR program to access remotely the dongle, we need to install SoapySDR Remote that allows remote access to the SDR.

So at the remote server we need to obtain, build and install the following components:

  • SoapySDR – The abstraction layer
  • SoapyRTLSDR – The driver for our RTL USB dongle
  • SoapySDR Remote – The server for remotely access the RTL dongle

So basically the instructions are something like this:

  mkdir ~/SDR 
  cd ~/SDR
  git clone https://github.com/pothosware/SoapySDR.git
  cd SoapySDR/
  mkdir build
  cd build
  cmake ../ -DCMAKE_BUILD_TYPE=Release
  make -j4
  sudo make install
  sudo ldconfig
  SoapySDRUtil --info

Building the SoapySDR remote:

 cd ~/SDR
 git clone https://github.com/pothosware/SoapyRemote.git
 cd SoapyRemote/
 cd build
 cmake ..
 make
 sudo make install

and build the RTLSDR driver:

  cd ~/SDR
  sudo apt-get install librtlsdr-dev
  git clone https://github.com/pothosware/SoapyRTLSDR.git
  cd SoapyRTLSDR/
  mkdir build
  cd build
  cmake .. -DCMAKE_BUILD_TYPE=Release
  make
  sudo make install
  sudo ldconfig

So at the end we should have the following outputs:

opi@opi:~# SoapySDRUtil --probe
######################################################
## Soapy SDR -- the SDR abstraction library
######################################################

Probe device 
Found Rafael Micro R820T tuner
Found Rafael Micro R820T tuner

----------------------------------------------------
-- Device identification
----------------------------------------------------
  driver=RTLSDR
  hardware=RTLSDR
  origin=https://github.com/pothosware/SoapyRTLSDR
  rtl=0

----------------------------------------------------
-- Peripheral summary
----------------------------------------------------
  Channels: 1 Rx, 0 Tx
  Timestamps: NO
  Other Settings:
     * Direct Sampling - RTL-SDR Direct Sampling Mode
       [key=direct_samp, default=0, type=string, options=(0, 1, 2)]
     * Offset Tune - RTL-SDR Offset Tuning Mode
       [key=offset_tune, default=false, type=bool]
     * I/Q Swap - RTL-SDR I/Q Swap Mode
       [key=iq_swap, default=false, type=bool]

----------------------------------------------------
-- RX Channel 0
----------------------------------------------------
  Full-duplex: YES
  Supports AGC: YES
  Stream formats: CS8, CS16, CF32
  Native format: CS8 [full-scale=128]
  Stream args:
     * Buffer Size - Number of bytes per buffer, multiples of 512 only.
       [key=bufflen, units=bytes, default=16384, type=int]
     * Buffer Count - Number of buffers per read.
       [key=buffers, units=buffers, default=15, type=int]
  Antennas: RX
  Full gain range: [0, 49.6] dB
    TUNER gain range: [0, 49.6] dB
  Full freq range: [23.999, 1764] MHz
    RF freq range: [24, 1764] MHz
    CORR freq range: [-0.001, 0.001] MHz
  Sample rates: [0.25, 3.2] MHz

And SoapySDR should have the following configuration:

opi@opi:~# SoapySDRUtil --info
######################################################
## Soapy SDR -- the SDR abstraction library
######################################################

API Version: v0.5.0-gfec33c63
ABI Version: v0.5-2
Install root: /usr/local
Module found: /usr/local/lib/SoapySDR/modules/libremoteSupport.so
Module found: /usr/local/lib/SoapySDR/modules/librtlsdrSupport.so
Loading modules... done
Available factories...null, remote, rtlsdr, 

So all we need is to start our server:

opi@opi:~# SoapySDRServer --bind
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

ba73bc08-3d0f-1458-8337-93d5a4502801
Launching the server... tcp://[::]:55132
Server bound to [::]:55132
Launching discovery server... 
Press Ctrl+C to stop the server

Using CubicSDR and SoapySDRRemote
So all we need is now on the startup SDR device screen selection add (by pressing the Add button) the remote IP of the Orange Pi PC server to access remotely the RTLSDR dongle.

My Orange PI PC IP address is 192.168.1.19:

CubicSDR device selection

CubicSDR device selection

And here it is the CubicSDR in action.

CubicSDR in action

CubicSDR in action

Conclusion and final notes
The CPU usage and temperature on the Orange PI PC is not a problem when using the server. CPU usage floats around 40%, and no meaningful or worrying changes on the CPU temperature. So the Orange PI PC is up to the task without any issues when serving data with SoapySDRRemote.

Also with CubicSDR and SoapySDRRemote, I’ve experienced no lag when changing frequencies, namely when dragging the frequency selector.. It seems that all changes are instantaneous and note that my desktop connects to the remote server through a 200Mbps PLC and only then it is cable network to the Orange Pi. According to my desktop PC network widget, when receiving data, I have around 6.5Mbps of data comming in when using the maximum sample rate of 3.2MHz.

Also it took me a while to get used to the CubicSDR user interface, but overall for things like fine tuning, since it has a dedicaded codec screen, is much better than Gqrx.

CubicSDR  fine tuning

CubicSDR fine tuning

Also one great feature is if we keep dragging the spectrogram window, the central frequency changes so it keeps up with the SDR bandwidth, shile in Gqrx we need to dial in.

Still I’m using Gqrx and rtl_tcp since CubicSDR has no data output, other then piping audio. Gqrx can pipe to UDP, that allows the decoding of digital modes locally or on other servers without messing around with PulseAudio and Jackd.
Also bookmarking isn’t as direct/easy as with Gqrx. Not sure if I can give labels/names to bookmarks and search them, like I can in Gqrx, but then the problem might be between the chair and computer…

Anyway CubicSDR is a great SDR application and the future looks bright.

I do recommend to give it a test drive.

Dropbox doesn’t start (Linux)

So a quick note regarding Dropbox running on ArchLinux with Nvidia drivers. Since I use KDE Plasma 5 that doesn’t show system tray icons of some applications that have not migrated to the new system tray protocol, I didn’t notice that after upgrading Dropbox it stopped working.

A quick investigation, I’ve found out that starting Dropbox from the command line did nothing. It started and stopped. Something is going on…

So I’ve used the strace command to see if I could catch something meaningful:

[pcortex@pcortex:~]$ cd .dropbox-dist/dropbox-lnx.x86_64-3.10.8/
[pcortex@pcortex:dropbox-lnx.x86_64-3.10.8]$ strace ./dropbox 2> stracedump.txt

And sure enough in the stracedump.txt I had the following error:

open("/usr/lib/dri/tls/swrast_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/dri/swrast_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
rt_sigaction(SIGSEGV, {0x7f09e1dea010, [SEGV], SA_RESTORER|SA_RESTART, 0x7f09e1a88d60}, NULL, 8) = 0
write(4, "Fatal Python error: ", 20)    = 20
write(4, "Segmentation fault", 18)      = 18
write(4, "\n\n", 2)                     = 2
write(4, "Traceback (most recent call firs"..., 36) = 36
write(4, "  File ", 7)                  = 7
write(4, "\"", 1)                       = 1
write(4, "d", 1)                        = 1

So the error is related to the Software Raster library, that I don’t have because I’m running Nvidia proprietary drivers.

The confirmation came with the following command:

[pcortex@pcortex:dropbox-lnx.x86_64-3.10.8]$ LIBGL_DEBUG=verbose ./dropbox
libGL: OpenDriver: trying /usr/lib/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /usr/lib/dri/swrast_dri.so
libGL error: dlopen /usr/lib/dri/swrast_dri.so failed (/usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: swrast_dri.so
libGL error: reverting to indirect rendering

The solution?
Quite simple, just remove (backup) the libGL.so.1 library that exists on the Dropbox directory.
After the removal, Dropbox starts just fine.

Odroid-C1+ – The low power super server

On the ARM based small computers front, running Linux or Android, there are several contenders, being the most famous one the Raspberry PI. Due to the huge community and documentation available the Raspberry Pi is the most chosen for a whole range of projects including low power servers, controllers, media players and so on.
Anyway, there are several alternatives, even cheaper, than the RPi. For example the 15$ Orange Pi looks promising on the price front, but not on the software side due to the Allwinner chip, it doesn’t look very promising without closed binary blobs. Other alternatives are the Banana Pi, Beaglebone board, Cubie board, and so on.

Anyway, since I needed something low power for running Node-Red, Sparkfun’s Phant, for logging data from the ESP8266, the RPi, was the way to go. But for the same price and with better specs, the Odroid-C1+ boosts also a very good community, supports Android, Ubuntu and even Arch Linux for ARM V7.

The advantages from the Odroid-C1+ over the RPi are:
– Same price (at least for me, 45€ for the RPi vs 44€ for the Odroid)
– Powerful 1.5 GHz quad core cpu
– Dedicated Ethernet chip not sharing the USB ports with 1GBps port.
– An emmc slot for storage allowing faster I/O than the SD cards. Sizes available between 8GB and 64Gb
– IR receiver
– Supports Ubuntu, Android, Arch Linux…

And so on.

As same as the RPi, the board by itself can’t do much, so I’ve also bought the power supply, acrylic box and a 32Gb emmc solid disk.
I was torn between the cheaper typical sd card and this emmc disk. Since the odroid supports it, why not?

As we can see, while it’s not faster than a typical SSD, it has the same performance than a spinning regular harddisk:

root@odroid:/etc/NetworkManager# hdparm -Tt /dev/mmcblk0p2

/dev/mmcblk0p2:
 Timing cached reads:   1440 MB in  2.00 seconds = 720.23 MB/sec
 Timing buffered disk reads: 240 MB in  3.02 seconds =  79.56 MB/sec
root@odroid:/etc/NetworkManager# 

Anyway, while the emmc is expensive, more than the price of the Odroid for the 32GB, it feels snappy and boots quite fast.

The emmc comes with a standard 4GB partition, and we need to expand it to the full emmc capacity using the provided odroid utility, or also, the gparted utility that is shipped on the ubuntu release.

So after powering up:

– Went to my router web interface and under the DHCP server tried to find which IP the Odroid got.
– Note, if connected to a HDMI monitor and if a keyboard and mouse is also connected, we can use the Odroid utility on the LXE window manager to expand the partition size from the default 4GB.
– The odroid utility can also be used from the command line: odroid-utility.sh
– With the IP I’ve got, I’ve ssh to the odroid and logon as root with odroid as password.
– Started up Gparted and increased the partition from 4GB to the 32GB emmc capacity.

root@odroid:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mmcblk0p2 29G 4.6G 23G 17% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 420M 4.0K 420M 1% /dev
tmpfs 425M 8.0K 425M 1% /tmp
tmpfs 85M 2.6M 83M 4% /run
none 5.0M 0 5.0M 0% /run/lock
none 425M 8.0K 425M 1% /run/shm
none 100M 24K 100M 1% /run/user
/dev/mmcblk0p1 129M 7.2M 122M 6% /media/boot
root@odroid:~#

– Updated the system: apt-get update and apt-get upgrade and it took a while to upgrade a lot of packages.

– After the upgrade I added a user: pcortex and made it to belong to the sudo group. This will be my working user, to avoid mishaps with the root user….

adduser –home /home/pcortex pcortex
passwd pcortex
adduser pcortex sudo

After the initial updating, and setting up, the remaining tasks where to install nodejs, node-red, phant and sqlite3.
Still as root:

apt-get install nodejs npm sqlite3
npm install -g node-red
npm install -g phant
npm install -g node-red-node-sqlite

For Android Notifications using the Google Cloud Messaging infrastructure:
npm install -g node-gcm (Check out: Node-Red GCM notifications

And finally making a link to let node be an alias of nodejs: ln -s /usr/bin/nodejs /usr/bin/node

Now with the user pcortex, we can start Node-Red and phant and starting doing interesting things with a low power super server 🙂

Edit: I’ve also changed the IP from a dynamic IP to a fixed IP. I’ve kept the hostname, odroid is cool enough 🙂

So I’ve edit the NetworkManager.conf file located at /etc/NetworManager, and changed the managed=false to managed=true under the section [ifupdown]. The final file is this one:

[main]
plugins=ifupdown,keyfile,ofono
dns=dnsmasq

[ifupdown]
managed=true

 
And then changed the network interface configuration to the fixed IP by editing the file interfaces located in /etc/network

# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
   address 192.168.1.45
   netmask 255.255.255.0
   broadcast 192.168.1.255
   gateway 192.168.1.254
   dns-nameservers 8.8.8.8 208.67.222.123

Note: I’ve also installed nginx instead of Apache just to keep resource usage low, but that’s another story…

Arch Linux e Cartão do Cidadão

Só um quick post para lembrar como se pode por o Midleware/software do cartão do cidadão a funcionar em Arch Linux.

– Primeiro determinar que o cartão é detectado pelo sistema:

lsusb
Bus 007 Device 002: ID 058f:9520 Alcor Micro Corp. EMV Certified Smart Card Reader

– Segundo instalar o software opensc:

[root@pcortex ~]# pacman -S opensc

Isto só vai servir para verificar que conseguimos aceder correctamente ao leitor do cartão.

– Testar o acesso ao leitor:

[root@pcortex ~]# opensc-tool -l
No smart card readers found.

Se for este o caso, arrancar com o daemon de suporte de acesso aos leitores:

[minime@pcortex ~]# sudo -s
[root@pcortex ~]# pcscd

Testar novamente:

[minime@pcortex ~]# opensc-tool -l
# Detected readers (pcsc)
Nr. Card Features Name
0 No Alcor Micro AU9520 00 00

– Podemos agora instalar o software do cartão do cidadao. No meu caso usei a versão disponível no repositório AUR:

yaourt -S cartao-cidadao

E no fim, executa-se:

[minime@pcortex ~]# pteigui

Pode-se ignorar as mensagens de erros, e o icon deverá aparecer no SysTray.

Para ter acesso aos certificados do cartão a partir do FireFox, por exemplo, bastará em princípio seguir estas instrucções: https://faqs.up.pt/faq/content/29/557/pt/como-configurar-o-mozilla-firefox-para-usar-o-cart%C3%A3o-de-cidad%C3%A3o.html

Setting up an IoT framework/dashboard with NodeRed, Mosca/Mosquitto and Freeboard.io dashboard

With the low cost of the esp8266 chip and boards, it is now easy and affordable to add connectivity  to almost anything that we can think of. Still if we build an Internet of Things device by enabling it with the esp8266 chip to connect to the internet, or other networks, what we can do with the data that we gather? Basically we do what we do with any data that we get, store it, process it and show it, and that’s what we are going to see on this post.

The software (running on Linux platform) that I’ll use is the following:

IBM NodeRed: (http://nodered.org/) This great software will allow us to get data, process it, store and publish it, with a workflow based designer. It runs under Node-Js Javascript based server engine.

MQTT Broker Mosca: (http://www.mosca.io/) There are several MQTT brokers available, but this one runs on Node-Js and so is platform agnostic, and runs where Node-Js runs. Also it has websocket support out of the box, which allows to use the Freboard.io dashboard in a straight foward way.

Freeboard.io Dashboard: (http://freeboard.io/) Is a great dashboard and freely available at the project GitHub page (https://github.com/Freeboard/freeboard). It allows to design and store several dashboards with some cool widgets.

Step 1: Setting up NodeRed:

First of all make sure that you have node and npm (the node package manager) installed on your distribution. You can see that with the commands node -v and npm -v for checking out their versions. In my case, and for now, it’s version v0.10.33 for nodejs and 1.4.28 for npm.

First I’ve created a directory named IoTServer where I’ll keep all software and configurations.

Download the zip file from nodered GitHub page https://github.com/node-red/node-red or just clone the repository. I clone the repository, because it’s simpler in the future to update it with just git pull. First just make sure that you have the icu/libicu libraries installed.

cd ~
mkdir IoTServer
cd IoTServer
git clone https://github.com/node-red/node-red
cd node-red
npm install

We can now run the nodered server with the command node red.js -v. I use -v at the end to see if any error crops up and also to see if any node modules are missing, and need to be installed. For example:

pcortex@cloudsrv:~/IoTServer/node-red$ node red.js -v  
   Welcome to Node-RED 
   =================== 
 
24 Feb 11:35:16 - [info] Node-RED version: v0.10.3.git 
24 Feb 11:35:16 - [info] Node.js  version: v0.10.33 
24 Feb 11:35:16 - [info] Loading palette nodes 
24 Feb 11:35:59 - [warn] ------------------------------------------ 
24 Feb 11:35:59 - [warn] [arduino] Error: Cannot find module 'arduino-firmata' 
24 Feb 11:35:59 - [warn] [rpi-gpio] Info : Ignoring Raspberry Pi specific node. 
24 Feb 11:35:59 - [warn] [redisout] Error: Cannot find module 'redis' 
24 Feb 11:35:59 - [warn] [mongodb] Error: Cannot find module 'mongodb' 
24 Feb 11:35:59 - [warn] ------------------------------------------ 
24 Feb 11:35:59 - [info] Server now running at http://127.0.0.1:1880/ 
24 Feb 11:35:59 - [info] Flows file not found : flows_clouds.json 
24 Feb 11:35:59 - [info] Starting flows

As we can see the node modules arduino, rpi-gpio, redisout and mongodb are missing. If we are going to use them, we need to install them with the npm tool. Also the following message: Flows file not found : flows_clouds.json informs us that no saved nodered workflows where found. We can now access the nodered at adress http://localhost:1880/. If we want to change the bind address or add authentication, we should change the settings.js file. Take a look at http://nodered.org/docs/configuration.html for more information, and start nodered with: node red.js -v -s settings.js

We can now start designing our nodered workflows that receive data from the esp8266 via MQTT protocol or even by simple REST based HTTP protocol, process it and store the data payload, if any.

For storing data into a database, we can use MongoDB, or mysql, for example, installing the needed modules.

For mysql we can do:

cd ~/IoTServer/node-red
npm install bignumber.js require-all readable-stream
npm install mysql
npm install node-red-node-mysql

And we need to restart nodered. The mysql node should be available now in storage pallet.

Step 2: Installing Mosca MQTT broker (Alternative 1 for the MQTT Broker)

We can use several brokers, but since we already are running nodered on node-js, we will use Mosca. Both nodered and Mosca can provide a websocked based interface. This is of interest because we can use browser based websocket applications, like freeboard.io, to consume data from our IoT devices.

cd ~
sudo -s
sudo npm install mosca bunyan -g

And to run it just do: mosca -v | bunyan

For running it with websockets enabled, on TCP port 3000, just do:

mosca -v --http-port 3000 --http-bundle --http-static ./ | bunyan

Check out the documentation for more information at: https://github.com/mcollina/mosca/wiki/MQTT-over-Websockets

Step 2: Mosquitto MQTT broker with Websockets support (Alternative 2 for the MQTT Broker)

If having trouble installing or using Mosca, we can use the latest Mosquitto version 1.4 that has websockets support.

For that we need to make sure that cmake and uuid/uuid-dev are installed (For example: sudo apt-get install cmake uuid uuid-dev).

Also we need to download the libwesockets library, compile it and install it:

cd ~/IoTServer
git clone git://git.libwebsockets.org/libwebsockets
cd libwesockets
mkdir build
cd build
cmake ..
make
sudo make install
We then need to download and compile Mosquitto 1.4:
cd ~/IoTServer
wget http://mosquitto.org/files/source/mosquitto-1.4.tar.gz
tar xvzf mosquitto-1.4.tar.gz
cd mosquitto-1.4
We need to edit the config.mk file and change the option WITH_WEBSOCKETS:=no  to WITH_WEBSOCKETS:=yes
And finally:
make
sudo make install
To activate websocket support we need to edit the mosquitto.conf file located at /etc/mosquitto or some other directory and add the following lines, for example at the end of the file:
listener 1883

listener 9001 127.0.0.1
protocol websockets

Then we run it with:

pcortex@cloudsrv:~/IoTServer$ mosquitto -c /etc/mosquitto/mosquitto.conf
1424790588: mosquitto version 1.4 (build date 2015-02-24 14:38:47+0000) starting 
1424790588: Config loaded from /etc/mosquitto/mosquitto.conf. 
1424790588: Opening ipv4 listen socket on port 1883. 
1424790588: Opening ipv6 listen socket on port 1883. 
1424790588: Opening websockets listen socket on port 9001.

Step 3: Installing freeboard.io dashboard

We are almost at the end of this long post. For our infrastructure we need now to install the dashboard that will allow us to see the data in the browser.
cd ~/IoTServer
git clone https://github.com/Freeboard/freeboard.git

For now I’ll just use apache to serve the freeboard dashboards that is installed in my server. Because freeboard is not installed on the root web directory we need to make the freeboard available to Apache. The easiest way to do this, as long as Apache has the FollowSymLinks option enabled for the document root, is to create a link to the freeboard directory on the web document root:

sudo -s
cd /var/www
ln -s /home/pcortex/IoTServer/freeboard iot

And now freeboard is available at the url http://myserveraddres/iot.

We need now, and finally to add MQTT over Websockets support to freeboard… We are almost there. This post http://jpmens.net/2014/11/12/freeboard-a-versatile-dashboard/   shows how it’s done but I’ll repeat it here again:

1st: Download the development version of mqtt.js here: (http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.javascript.git/plain/src/mqttws31.js?h=develop) and save it:

cd ~/IoTServer/freeboard/plugins
mkdir mqtt
cd mqtt
wget --output-document mqttws31.js http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.javascript.git/plain/src/mqttws31.js?h=develop

We will now download the freeboard MQTT plugin:

cd ~/IotServer
git clone https://github.com/alsm/freeboard-mqtt
cd freeboard/plugins/mqtt
cp ~/IoTServer/freeboard-mqtt/paho.mqtt.plugin.js .

We need to edit the paho.mqtt.plugin.js file and change the line:

                "external_scripts" : [
                        "<full address of the paho mqtt javascript client>" 
                ],

to

               "external_scripts" : [
                        "plugins/mqtt/mqttws31.js" 
                ],

and finally we need to change the index.html file from:

   <scripttype="text/javascript">
        head.js("js/freeboard+plugins.min.js", 
                // *** Load more plugins here *** 
                function(){

to

   <scripttype="text/javascript">
        head.js("js/freeboard+plugins.min.js", 
                "plugins/mqtt/paho.mqtt.plugin.js", 
                // *** Load more plugins here *** 
                function(){

That’s it. We are finally ready. All infrastructure is done. We need now just to configure it.

Final step: Testing it out with mosquitto websockets!:

Let’s start Mosquitto: nohup mosquitto -c /etc/mosquitto/mosquitto.conf &

And check if the WS protocol is active:

netstat -nap | grep 9001

tcp        0      0 0.0.0.0:9001            0.0.0.0:*               LISTEN      18973/mosquitto

Setting up Freeboard.io:  It’s very important to notice that the Dashboard will run on our browser, making in fact our browser the client. This means that when we configure freeboard, we must take notice of this when filling up the MQTT broker address. It will be localhost if the browser AND the broker runs on the same machine, but for the dashboard to work anywhere we should use the public ip of the machine running the broker:

Access the freeboard.io dashboard, and select Datasources -> Add. From the dropdown box, we select the Paho MQTT provider.

Selection_055

We need to configure the data source as follows:

Selection_056

Change the MQTT server address to something that makes sense in your config.

For Mosquitto with websockets enabled, the port is 9001, and for Mosca, the port is 3000, but of course this can change, depending of the configuration used.

We can now add a Panel and a gauge to see data from the topic /data that we defined on the above datasource.

So, Add Pane, and on the new Pane select the + (plus) to add a widget. In our case we will select Gauge. We fill out the required info and on the Value field we press Datasource, and select the newly previous created datasource named Data and because we are not using JSON for data, we just append .msg at the end. So something like datasource[“DS_data”].msg should be writen.

Selection_057

We press Save, and that’s it.

We can now publish into the topic, and the dashboard should update:

Selection_058

For example, in this case: mosquitto_pub -t /data -m 23

That’s it. It works fine with Mosquitto 1.4 with websockets enabled and with the Mosca MQTT broker.

Final setup:

We should make our dashboard permanent and for doing so we need to save the dashboard locally as a JSON file, and move it to the server running the dashboard to the root directory of freeboard.

After that we call directly the saved dashboard with the following URL:

http://myserveraddress/iot/index.html?load=dashboard.json

Edit: For Websocket support on the Freeboard.io Dashboard, please checkout Freeboard and WebSockets

Warning:The current freeboard.io project at Github (as of 12/2015) has problems saving dashboards when using Firefox. For saving dashboards use Chrome/Chromium.

Upgrading to an SSD, moving from Kubuntu to Arch Linux and setting up KDE

This is a single post just to make notice of some points:

My home desktop PC, was running since 2009 the Kubuntu distribution. I’ve never reinstalled it and did always the upgrades from 8.10 to 9.04, 9.10 and so on until 14.04. Almost six (!) years with no virus, trojan horses, and a single initial installation. I’ve upgraded through the years the KDE desktop since the ill famed initial 4.0 release, to the latest 4.6 release.

But things where starting to go awry… At the beginning I had a ATI 4870 graphic board, but ATI dropped support for it on Linux (and Windows) and since I had so many troubles with the proprietary driver, and to add it up I had no support for the board, I sold it and bought an Nvidia GTX660.

But with both graphic boards, I had suffered from a long time problem: I could only do a sleep/resume a number of times before the system hanged. The symptoms starting from loosing network connectivity and then restoring right back, and some cycling of these symptoms to a complete lockup. At first I thought that was due to the network driver used by my motherboard, a Asus P6T with  a Realtek RTL8111/8168/8411 board, but with the correct driver for it, the issue continued.  Booting up was a painful two/three minute wait from BIOS to desktop.

But after a while I could pin point my issues not to the network board but to the Xorg and video driver combination. Some EQ buffer overflow, hanged X, and finally the machine.

Ubuntu and Kubuntu based distributions have PPA’s repositories where some alternative software and versions of mainstream software are available. I had trouble with some of these alternative PPA’s when upgrading, and had to remove them (with ppa-purge) to be able to upgrade the Kubuntu distribution. Anyway, there is an PPA, xorg-edgers,  where alternative and more recent versions of Xorg Software and Nvidia drivers are provided and that almost solved my sleep/resume problem. The problem was that after adding this PPA the Ubuntu/Kubuntu nvidia driver and the Xorg-edgers driver, after updates the X server was unable to find the nvidia driver, despite being installed, and had to reinstall the driver again to get my desktop working.

That was it. I had enough with Ubuntu/Kubuntu distributions. My bad and my issues, sure…

I’ve being testing Arch Linux on my work laptop for almost an year with great success. I’ve chosen Arch Linux because I didn’t had to loose an unknown number of hours per year upgrading to the next distribution version, and an rolling release version makes more sense. And Arch has the greatest Linux Wiki out there even solving problems that happen on the Ubuntu/Kubuntu line, and also mainly there are only one (well two) repositories. No PPA’s mess.

So I completely moved all my systems to Arch Linux, and my sleep/resume issue is 100% solved.

SSD on a SATA II based motherboard

My desktop computer has an Asus P6T motherboard. This motherboard only has SATA II (SATA 2) ports. Is it worth to use an SSD on such a board without upgrading to a SATA III board?

The answer is YES. If you have a SATA II based motherboard AND not buying an SSD because of that, do not wait any more. Buy it now. It is totally worth it.

Arch Linux is of course way lighter than Kubuntu, and with an SSD ( by the way it’s a Crucial MX100 256GB) my two/three minute boot to desktop time came down to 15s…

hdparm -Tt /dev/sdc
/dev/sdc:
Timing cached reads:   16710 MB in  2.00 seconds = 8361.40 MB/sec
Timing buffered disk reads: 728 MB in  3.00 seconds = 242.59 MB/sec

Note the Samsung EVO 120GB that I have on my work computer, that has the flawed firmware that slows down with time, on a SATA III based motherboard:

/dev/sda:
Timing cached reads: 23864 MB in 2.00 seconds = 11943.41 MB/sec
Timing buffered disk reads: 164 MB in 3.01 seconds = 54.47 MB/sec

Compare this with my desktop old boot disk, an WD 640GB blue:
/dev/sdb:
Timing cached reads: 16580 MB in 2.00 seconds = 8296.87 MB/sec
Timing buffered disk reads: 356 MB in 3.00 seconds = 118.59 MB/sec

I hadn’t yet had the time to upgrade the firmware on the EVO…

Making KDE looking good on Arch Linux

To end this rather long and tedious post, and also to my future reference, to make KDE looking good:

    • Make sure that the infinality modifications are installed: https://wiki.archlinux.org/index.php/Infinality. Make sure that you update.
    • Make sure that the DPI detected by your X server are correct for your monitor. Install xdpyinfo with pacman -S xorg-xdpyinfo. Figure out what might be your monitor dpi using this site: http://dpi.lv/ . Execute xdpyinfo | grep -B2 resolution and see if are similar. If not you need to set the correct dpi either in xorg.conf or in the Kde settings.
      In my case I have a dual monitor setup, and hence the weird dpi 95×96.screen

      #0: dimensions: 3200×1080 pixels (856×286 millimeters)

      resolution: 95×96 dots per inch

      So my font settings are:

Selection_002

  • While the above settings make my fonts look good, the taskbar fonts where awful and took me a while to figure out that was not the dpi settings but the Desktop Theme.
  • To solve this first install Krita with pacman -S calligra-krita . This will install the Krita color themes that are, in my opinion very good. And on KDE System Settings -> Workspace Appearance -> Desktop Themes select Get New Themes and get the AndroidBit theme.
  • That’s it:Selection_004
  • I’ve selected the Produkte Desktop Theme
  • And the colors:Selection_005