Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 4: Installing Grafana and configuring

Our software stack installation ends with configuring and installing Grafana. Ubuntu installation instructions are at this Grafana documentation link..

Installation is as easy as to download and install the latest Grafana dashboard version:

root@server:~# wget https://grafanarel.s3.amazonaws.com/builds/grafana_4.1.2-1486989747_amd64.deb
root@server:~# sudo dpkg -i grafana_4.1.2-1486989747_amd64.deb

In my case, the above instructions where enough to install successfully the software.

After installing if we will access the Dashboard through a reverse proxy, check out my previous post to configure correctly the Grafana server: Grafana Reverse Proxy

We can now start the Grafana server but if the server is exposed to the public internet the first thing to do is to change the default admin user password to something safe, or better yet, create a new admin user with a strong password and delete the admin default account.

We should press the top left icon, choose admin and then Global users. We can now select the Admin user and change the default password:

Here we also can create a new admin account, which is my recommendation.

Data source configuration:

Dashboard data provides from data sources that we configure on  Grafana. We can configure several data sources that we will used to fetch data to feed to our dashboards and graphs.

In our case we will configure a single InfluxDB datasource. Note that we didn’t defined any authentication for the InfluxDB server, so no access credentials are needed yet, but we should set ones as soon as possible.

Since Grafana and InfluxDB are running on the same server we use the base URL http://localhost:8086. Make sure that the connection type is set as proxy.

After setting the datasource name and datasource database, press Save & Test and it should report success.

Initial Dashboard configuration

We are now able to create a basic first dashboard by building some charts/graphs based on the previous created data source.

Press the New Dashboard button to create a new dashboard. In case that you don’t have a screen as the above screen shot, just press the top left Grafana Icon, select Dashboards and then New.

The following screen should appear where we can add graphical panels to show our data. We should the way that we want to show our data, by selecting the Graph type. The selected graph type should reflect immediately on the panel below. To edit the panel source data we should press the panel title and a pop-up window will appear, and then we press Edit.

We are now able to edit the associated queries that will fill the panel with data. Associated with the panel we can have multiple queries for the source data. In the default case a default query named A is already set up. We just need to change our measurement field to the correct one, which in our case could be Temperature or heap, based on the previous post (InfluxDB data configuration)

.

We need now to press the Select Measurement and a drop down box with the available measurement should appear. We select the one that we want and data should now appear on the above graph.

From now on is just a bit of perfecting things out like giving a sound name to the panel in the General tab and at the end pressing the Save icon on the top of the screen.

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 3: Single point of access – Reverse proxy the services with nginx

Since we will be running a lot of services, each running on its own port, the following configuration, is optional, but allows to access all services through the same entry point by using Nginx server as a reverse proxy to Node-Red, Node-Red UI/Dashboard, Node-Red Worldmap and Grafana.

With this configuration the base URL is always the same without any appended ports, and the only thing that changes are the URL path:

http://server/nodered
http://server/nodered/worldmap
http://server/grafana

To allow this we install and configure Nginx:

apt-get install nginx

The configuration files will reside in /etc/nginx directory. Under that directory there are two directories: sites-available and sites-enable where the later normally contains a link to configuration files located at sites-available.
At that directory there is a file named default that defines the default web site configuration used by Nginx. This is the file where we will add the reverse proxy directives.

Reverse proxy for Node-Red and Node-Red Contrib Worldmap
For setting up the reverse proxy for Node-Red we must first change the base URL for Node Red from / (root) to something else that we can map the reverse proxy.

For this we will need to edit the settings.js file located on the .node-red directory on the home path of the user running Node-Red.

We need to uncomment and change the entry httpRoot to point to our new base URL.

   httpRoot: '/nodered',

Don’t forget the trailing comma.

We need to restart now Node-Red and it should be accessible at the URL http://server:1880/nodered instead of http://server:1880/.

To configure Nginx, we edit the file default at /etc/nginx/sites-available and add the following section:


        location  /nodered {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_pass "http://127.0.0.1:1880";
        }

        location /socket.io {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_pass "http://127.0.0.1:1880";

        }

Note the following: The first location defines the reverse proxy URL /nodered to be served by the backend server http://127.0.0.1:1880. The incoming path, /nodered, will be passed to the backend server URL /nodered, since paths are passed directly. No need to add the /nodered path to the backend server definition.
Also I’m using the 127.0.0.1 address instead of localhost to avoid the IPv6 mapping to the localhost. In this way I’m sure that IPv4 will be used.

The location mapping for /nodered will make all the functionality of node red to work as it should at the base url /nodered. But some nodes, like node-red-contrib-worldmap will request to the proxy server ignoring the node-red base root map. Hence the /socket.io mapping. It will allow the worldmap nodes to work, but will stop this mapping to be used for something else.

Reverse proxy for Grafana

Setting up the reverse proxy for Grafana we can, and should use the following documentation: Grafana Reverse Poxy. For me the following configuration worked:

First edit the [server] section on the Grafana configuration file grafana.ini located at /etc/grafana.

Uncomment and edit the following lines:

[server]
# Protocol (http or https)
protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = server.domain.com

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = http://server.domain.com/grafana/

Note the ending slash at the root_url. The same applies to the Nginx configuration

The files for the Nginx configuration are the same as the above configuration for reverse proxy.

We just need to add the following section after the previous location directives:

        location /grafana/ {
                proxy_pass http://localhost:3000/;
        }

We should now restart nginx to refresh the configuration, and all should be working as it should by accessing the Grafana dashboard at http://server.domain.com/grafana

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 2: Database configuration and data collection

On the previous post we’ve installed and the base software for our Grafana based dash board.

We need now to configure our InfluxDB database and Node Red to start collecting data.

Configuring InfluxDB:
Detailed instructions for configuring an InfluxDB database are on this InfluxDB documentation link..

The main concepts that we need to be aware when using the InfluxDB is that record of data has a time stamp, a set of tags and a measured value. This allows, for example to create a value named Temperature and tag it depending on the source sensor:

Temperature: Value=22.1 , Sensor=Kitchen
Temperature: Value=21.9 , Sensor=Room1

This allows to process all the data or only process data based on a certain tag or tags. Values and tags can be created on the fly without previously define them, which is a bit different from standard RDBMS engines.

Creating an InfluxDB database:
To create the database, we need to access the machine hosting the InfluxDB server and execute the command influx:

odroid@odroid:~$ influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> create database SensorData
> show databases
name: databases
name
----
_internal
SensorData

> 

Now we have our database created and I’ve named SensorData. To make an example with the above temperature data we can do the following:

> insert Temperature,Sensor=kitchen value=22.1
ERR: {"error":"database is required"}

Note: error may be due to not setting a database or retention policy.
Please set a database with the command "use " or
INSERT INTO . 
> use SensorData
Using database SensorData
> 

As we can see we need first to select the database where we are going to insert data with the command use SensorData:

> use SensorData
Using database SensorData
> insert Temperature, Sensor=kitchen value=22.1
ERR: {"error":"unable to parse 'Temperature, Sensor=kitchen value=22.1': missing tag key"}

> insert Temperature,Sensor=kitchen value=22.1
> insert Temperature,Sensor=Room1 value=21.9
> select * from Temperature
name: Temperature
time                Sensor  value
----                ------  -----
1487939008959909164 kitchen 22.1
1487939056354678353 Room1   21.9

Note that we can’t use spaces between the Measure name and the tags. The correct syntax is as follows:

 insert MeasureName,tag1=t1,tag2=t2,...   value1=val1,value2=val2,value3=val3,....

Also note that no DDL (data definition language) was used to create the tags or the measured value, we’ve just inserted data for our measurement with the our tags and value(s) without the need of previously define the schema.

Configuring Node-Red
Since we now have a database we can configure the InfluxDB Node Red nodes to store data onto the database:

There are two types of InfluxDB nodes, one that has an Input and Output and other that only has Input. The former is for making queries to the database where we provide on the input node the query, and on the output the results are returned. The later is for storing data only onto the database.
For both nodes we need to configure an InfluxDB server:

InfluxDB Server Configuration

We need to press the Pen icon right next to the server to add or reconfigure a new InfluxDB server:

InfluxDB server

A set of credentials are required, but since I’ve yet configured security, we can just put admin/admin as username and password. In a real deployment we must activate security.

From now on it is rather simple. Referring to InfluxDB node configuration screenshot (Not the InfluxDB server configuration) we have a configuration field named Measurement. This is our measure name that we associate a value. Picking up on the above example with the Insert command it will be Temperature, for example.

Now if the msg.payload provided has input to the node is a single value, let’s say 21, this is equivalent to do:

Insert Temperature value=12

We other formats for msg.payload that allows to associate tags and measures. Just check the Info tab for the node.

Simple example:

The following flow shows a simple example of a value received through MQTT, in this case the free heap from one of my ESP8266 and its storage in InfluxDB:

Sample Flow

[{"id":"20bec5de.8881c2","type":"mqtt in","z":"ced40abb.3c92e","name":"Heap","topic":"/outbox/ESP12DASH/Heap","qos":"2","broker":"2a552b3c.de8d2c","x":83.16668701171875,"y":206.41668701171875,"wires":[["e0d9c912.8c57f8","876fb151.6f2fa"]]},{"id":"876fb151.6f2fa","type":"debug","z":"ced40abb.3c92e","name":"","active":true,"console":"false","complete":"false","x":408.5,"y":177,"wires":[]},{"id":"e0d9c912.8c57f8","type":"influxdb out","z":"ced40abb.3c92e","influxdb":"bbd62a93.1a7108","name":"","measurement":"heap","x":446.1666717529297,"y":224.58335876464844,"wires":[]},{"id":"2a552b3c.de8d2c","type":"mqtt-broker","broker":"192.168.1.17","port":"1883","clientid":"node-red","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":15,"cleansession":true,"willQos":"0","birthQos":"0"},{"id":"bbd62a93.1a7108","type":"influxdb","z":"","hostname":"127.0.0.1","port":"8086","protocol":"http","database":"SensorData","name":"ODroid InfluxDB"}]

We can see with this flow the data stored in InfluxDB:

> select * from heap;
name: heap
time                value
----                -----
1487946319638000000 41600
1487946440913000000 41600
1487946562206000000 41600
1487946683474000000 41600
1487946804751000000 41600
1487946926061000000 41600
1487947047309000000 41616
1487947168594000000 41600

Now we have data that we can graph with Grafana, subject of my next posts.

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 1: Installing

A more or less standard software stack used for control, processing and displaying data, has emerged that is almost used by everyone when hacking around on Arduinos, ESP8266, Raspeberry Pi’s and other plethora of devices. This “standard” software stack basically always includes the MQTT protocol, some sort of Web based services, Node-Red and several different cloud based services like Thingspeak, PubNub and so on. For displaying data locally, solutions like Freeboard and Node-Red UI are a great resources, but they only shows current data/status, and has no easy way to see historical data.

So on this post I’ll document a software stack based on Node-Red, InfluxDB and Graphana that I use to store and display data from sensors that I have around while keeping and be able to display historical memory of data. The key asset here is the specialized time-series database InfluxDB that keeps data stored and allows fast retrieval based on time-stamps: 5 minutes ago, the last 7 days, and so on. InfluxDB is not the only Time-Series database that is available, but it integrates directly with Grafana the software that allows the building of dashboards based on stored data.

I’m running an older version of InfluxDB on my ARM based Odroid server, since a long time ago, ARM based builds of InfluxDB and Grafana where not available. This is now not the case, but InfluxDB and Grafana have ARM based builds so we can use them on Raspberry PI and Odroid ARM based boards.

So let’s start:

Setting up Node-Red with InfluxDB
I’ll not detail the Node-Red installation itself since it is already documented thoroughly everywhere. To install the supporting nodes for InfluxDB we need to install the package node-red-contrib-influxdb

cd ~/.node-red
npm install  node-red-contrib-influxdb

We should now restart Node-red to assume/detect the new nodes.

Node Red InfluxDB nodes

Installing InfluxDB
We can go to the InfluxDB downloads page and follow the installation instructions for our platform. In my case I need the ARM build to be used on Odroid.

cd ~
wget https://dl.influxdata.com/influxdb/releases/influxdb-1.2.0_linux_armhf.tar.gz
tar xvzf influxdb-1.2.0_linux_armhf.tar.gz

The InfluxDB engine is now decompressed in the newly created directory influxdb-1.2.0-1. Inside this directory there are the directories that should be copied to the system directories /etc, /usr and /var:

sudo -s
cd /home/odroid/influxdb-1.2.0-1

Copy the files to the right location. I’ve added the -i switch just to make sure that I don’t overwrite nothing.

root@odroid:~/influxdb-1.2.0-1# cp -ir etc/ /etc
root@odroid:~/influxdb-1.2.0-1# cp -ir usr/* /usr
root@odroid:~/influxdb-1.2.0-1# cp -ir var/* /var

We need now to create the influxdb user and group:

root@odroid:~/influxdb-1.2.0-1# groupadd influxdb
root@odroid:~/influxdb-1.2.0-1# useradd -M -s /bin/false -d /var/lib/influxdb -G influxdb influxdb

We need now to change permissions on /var/lib/influxdb:

cd /var/lib
chown influxdb:influxdb influxdb

We can now set up the automatic start up script. On the directory /usr/lib/influxdb/scripts there are scripts for the systemctl based Linux versions and init.d based versions that is my case. So all I have to do is to copy the init.sh script from that directory to the /etc/init.d and link it to my run level:

root@odroid:~# cd /etc/init.d
root@odroid:/etc/init.d# cp /usr/lib/influxdb/scripts/init.sh influxdb
root@odroid:/etc/init.d# runlevel
 N 2
root@odroid:/etc/init.d# cd /etc/rc2.d
root@odroid:/etc/init.d# ln -s /etc/init.d/influxdb S90influxdb

And that’s it. We can now start the database with the command /etc/init.d/influxdb start

root@odroid:~# /etc/init.d/influxdb start
Starting influxdb...
influxdb process was started [ OK ]

We can see the influxdb logs at /var/log/influxdb and start using it through the command line client influx:

root@odroid:~# influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> show databases
name: databases
name
----
_internal

> 

Installing Grafana
We need now to download Grafana. In my case for Odroid since it is an ARMv7 based processor, no release/binary is available.
But a ARM builds are available on this GitHub Repository: https://github.com/fg2it/grafana-on-raspberry for both the Raspberry Pi and other ARM based computer boards, but only for Debian/Ubuntu based OS’s. Just click on download button on the description for the ARMv7 based build and at the end of the next page a download link should be available:

odroid@odroid:~$ wget https://bintray.com/fg2it/deb/download_file?file_path=main%2Fg%2Fgrafana_4.1.2-1487023783_armhf.deb -O grafana.deb

And install:

root@odroid:~# dpkg -i grafana.deb
Selecting previously unselected package grafana.
(Reading database ... 164576 files and directories currently installed.)
Preparing to unpack grafana.deb ...
Unpacking grafana (4.1.2-1487023783) ...
Setting up grafana (4.1.2-1487023783) ...
Installing new version of config file /etc/default/grafana-server ...
Installing new version of config file /etc/grafana/grafana.ini ...
Installing new version of config file /etc/grafana/ldap.toml ...
Installing new version of config file /etc/init.d/grafana-server ...
Installing new version of config file /usr/lib/systemd/system/grafana-server.service ...

Set the automatic startup at boot:

root@odroid:~# ln -s /etc/init.d/grafana-server /etc/rc2.d/S91grafana-server

And we can now start the server:

root@odroid:~# /etc/init.d/grafana-server start
 * Starting Grafana Server    [ OK ] 
root@odroid:~# 

We can now access the server at the address: http://server:3000/ where server is the IP or DNS name of our ODroid or RPi.

Conclusion:
This ends the installation part for the base software.

The following steps are:

  • Create the Influx databases –
  • Receive data from sensors/devices and store it on the previously created database
  • Configure and create Grafana data sources and dashboards
  • Add some plugins to Grafana

Upgrading NodeJs and Node Red on Odroid

I run many services on my Odroid C1+ including Node-Red. But since NodeJs on Odroid C1+ is version v0.10 is starting to be seriously old for running Node-Red or other NodeJS dependent software.

So my quick instructions for upgrading NodeJS and Node-Red on the Odroid C1+

Upgrading NodeJS

First verify what version is available/installed on the Odroid:

odroid@odroid:~$ node -v
v0.12.14
odroid@odroid:~$ nodejs -v
v0.10.25

Since I’ve already had previously installed a more recent version of NodeJS (the node command), the version used by Node-Red is v0.12.14 while the default NodeJS version is v0.10.25.

We can also, and should, check the npm version:

odroid@odroid:~$ npm -v
2.15.1

We also need to find what architecture we are using, just for completeness since ODroid C1+ is an ARM7 based architecture:

odroid@odroid:~$ uname -a
Linux odroid 3.10.96-151 #1 SMP PREEMPT Wed Jun 15 18:47:37 BRT 2016 armv7l armv7l armv7l GNU/Linux

This will allow us to download the correct version of the NodeJS binaries from the NodeJS site: NodeJS downloads.
In our case we choose the ARM7 architecture binaries, which at the current time is file: node-v6.9.2-linux-armv7l.tar.xz
So I’ve just copied the link from the NodeJS site and did a wget on the Odroid:

wget https://nodejs.org/dist/v6.9.2/node-v6.9.2-linux-armv7l.tar.xz

I then created a working directory and “untared” the file:

odroid@odroid:~$ mkdir nodework
odroid@odroid:~$ cd nodework
odroid@odroid:~/nodework$ tar xvf ../node-v6.9.2-linux-armv7l.tar.xz
odroid@odroid:~/nodework$ cd node-v6.9.2-linux-armv7l/
odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l$ 

Since there isn’t an install script we need to move the new NodeJS files to the correct locations:

  1. Binaries to /usr/bin
  2. Include files to /usr/include
  3. Libs files to /usr/lib

Copy the binaries, replacing, if existing the older versions:

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/bin
$ sudo cp -i node /usr/bin
cp: overwrite ‘/usr/bin/node’? y
odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/bin$ 

Copy the include files:

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/include
$ sudo cp -R node  /usr/include/

and copy the libraries

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/lib
$ sudo cp -R node_modules /usr/lib

and finally:

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/share
$ sudo cp -R . /usr/share

We need now to make npm to point to the correct nodejs script so, we need to delete the npm link at the /usr/bin and /usr/local/bin directories:

odroid@odroid:~$ sudo rm /usr/bin/npm
odroid@odroid:~$ sudo rm /usr/local/bin/npm

and re-create the correct links:

odroid@odroid:~$ sudo ln -s /usr/lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm
odroid@odroid:~$ sudo ln -s /usr/lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm

Running now the node and npm commands should report the latest versions:

odroid@odroid:~$ node -v
v6.9.2
odroid@odroid:~$ npm -v
3.10.9
odroid@odroid:~$ 

Success!

Upgrading Node-Red

From the Node-Red startup log, we can see the previous versions of node-red and nodejs used:

Welcome to Node-RED
===================

28 Dec 17:55:40 - [info] Node-RED version: v0.15.2
28 Dec 17:55:40 - [info] Node.js  version: v0.12.14
28 Dec 17:55:40 - [info] Linux 3.10.96-151 arm LE
28 Dec 17:55:42 - [info] Loading palette nodes
28 Dec 17:55:50 - [info] Dashboard version 2.1.0 started at /ui
28 Dec 17:55:54 - [warn] ------------------------------------------------------
28 Dec 17:55:54 - [warn] [rpi-gpio] Info : Ignoring Raspberry Pi specific node
28 Dec 17:55:54 - [warn] ------------------------------------------------------

we can upgrade now Node-Red according to the Node Red upgrading instructions:

odroid@odroid:~$ sudo npm cache clean
odroid@odroid:~$ sudo npm install -g --unsafe-perm node-red

and after a while the upgrade should be done.

Before starting up node-red I went to the node-red module directories, and did an update:

odroid@odroid:~/.node-red$ npm update
/home/odroid/.node-red
└── crypto-js@3.1.8 

Starting up Node-Red should show now the new software versions:

Welcome to Node-RED
===================

1 Jan 20:35:46 - [info] Node-RED version: v0.15.2
1 Jan 20:35:46 - [info] Node.js  version: v6.9.2
1 Jan 20:35:46 - [info] Linux 3.10.96-151 arm LE
1 Jan 20:35:47 - [info] Loading palette nodes
1 Jan 20:35:54 - [info] Dashboard version 2.2.1 started at /ui

Done!

Some thoughts on Lorawan

It cropped up on my news feed here on WordPress an article regarding three major flaws on the Lorawan protocol. Since I’m following the Lorawan topic I was interested to see what those “Major” flaws where, and found it rather interesting, that the text, also derived from a commercial vendor, looked like the conundrum story like if a glass is half full or half empty.

First there is an huge misconception regarding low power communications protocols (LPWAN), being either Lorawan, Sigfox or anything else. The key is power, and lower power usage, the less the better. Asking for LPWAN protocols to do the same that other high energy, higher bandwidth protocols can do, is mixing apples with oranges.

Anyway the “flaws”, if we can call then that, are as follow:

– All uplink messages are unacknowledged.
This not true. Lorawan supports three classes of devices. Class A, the less power hungry of them, opens two receive windows for downlink messages from the server, that can be used for acknowledging the uplink message. If it makes sense to have acknowledge, it depends on the business case… If it is required probably LoraWan neither Sigfox or other LPWAN protocols are adequate to be used…

Also any protocols that use the ISM radio bands must obey the defined rules by the government bodies that control the radio spectrum. These rules apply to any protocol on those bands, so it’s not a flaw specific to any protocol. The 1% duty cycle applies to Lorawan and SigFox. In fact Sigfox will enforce the 1% duty cycle by refusing messages that exceed that percentage, and the LoraWan backend provider The Things Network will do the same.

Also there is some confusion regarding the medium access protocol, in this case the radio spectrum. The medium is shared by anyone, so collisions and interference will happen. Sigfox adds some resilience to this this by transmitting each message three times in three different ISM band frequencies, for example. As Lorawan it also doesn’t check for the medium before transmitting but only transmits once, since the physical modulation CSS (Chirp Spread Spectrum) has more resilience to interference. Also due to the availability of what is called Spread Factor, several transmissions can happen at the same time at the same frequency, and successfully be decoded at the other end.

On radio protocols checking for medium occupancy before transmitting only makes sense for non constrained devices, since the process of checking the medium before transmitting will consume power (a lot by having the radio on) and without any sort of guarantee that interference will not start right away or the interferance is happening not at the node side, but on the gateway side. So since one of the engineering requirements for LPWAN is low power, then the exchange between power and medium access control is made, which means ALOHA and let’s transmit!. So now we can have devices that have batteries that last years.

– All gateways in range see all uplink traffic which is not safe
I find this one rather funny since, turning on my radio can catch any available radio broadcasters, or any radio scanner can receive anything. Just check out Web SDR for hours of amusement.

The fact that all gateways see all traffic is a direct consequence of the radio medium, not an issue with the protocol. Any protocol that uses radio has the same “flaw” and it applies to Lorawan, SigFox, UNB, Weightless, 3G, LTE, you name it.

To solve this, encryption is used and at least in Lorawan there are several encryption keys and ways of providing them.
Lorawan can use fixed provided keys (ABP – Activation by personalisation) or variable keys through OTAA (over the air activation).

Anyway the gateways can receive any Lorawan packets, but without at least the 128 bit Network key and 128 bit application key, can’t do anything with the data. Gateways only forward data to network backend servers, and there, if they have the correct keys, decryption can be done and data forward to the correct application servers.

Check out this for more information.

– LoRaWAN requires an enormous amount of bandwidth
Well, yes, it is a SPREAD spectrum technology and it makes part of the holy war between Narrow band supporters vs Wide band supporters. Ones say that UNB is better, others don’t, and so on. Spread spectrum technology exists since a long time ago. Lorawan bandwidth can be 150Khz, 250Khz and 500Khz vs the 200Hz of Sigfox.

While SST can be used and detected below the noise floor level and accepts variations on the frequencies (reflections, Doppler effects), UNB is, on the receiver side way more complicated since it requires very precise crystals for frequency reference and higher power levels on the spectrum.

So in the article that I read it seems that Lorawan SST is just bad, without any consideration of the advantages vs UNB, which by itself is a discussion on different technologies which have advantages and disadvantages each.

Conclusion:
Nothing is perfect in engineering. Trade offs need to be made to achieve the requested requirements, and then based on the initial implementations, improve on it. So Lorawan, as Sigfox, solve the same issues by different means with the associated advantages and disadvantages.

So the above flaws, are just engineering trade offs that can be applied to any protocol.

LPWAN – Starting up with LoraWAN and The Things Network

LPWAN Networks – A simple introduction
Low Power Wide Area Communications (LPWAN) classifies a group of communication protocols featuring low power usage and a high communication range. For Internet of Things communications, where battery powered devices and constrained devices (weak CPU, low RAM/ROM) are the norm, LPWAN use as the communication protocol for IoT makes sense, since it makes possible to have standalone devices with batteries that last years, instead of hours or days, and might be hundreds of meters to Kms away from a base station/gateway.

But LPWAN protocols, in contrast have low communication bandwidth, since from power, range and bandwidth, we can only have two of those, and while this might be a problem for certain applications, IoT devices don’t require high bandwidth and since most have sparse communication requirements, and when they do communicate they don’t need to transmit a lot of data.

Starting up with LoraWan and The Things Network
One of the possible ways of starting to use LPWAN networks in our IoT devices, is to use a LPWAN protocol named LoraWan, which is based on the Lora physical protocol, and if available at our area, the crowd sourced LPWAN network The Things Network as the backend network for receiving data.

Lora uses the 868Mhz ISM radio band and it uses a form of signal transmission called CSS. The ISM radio band can be used freely by everybody, but under certain rules that depends of the country where the devices are located. So to use that band, rules like maximum emission power and duty cycle (usage of the available spectrum) are defined, and in Europe, the maximum power is 20mW and 1% duty cycle per day. Additionally the back end operator can add other restrictions.

Over Lora, the Lorawan protocol is an open standard and source code for implementing it is available and already ported to several devices, including the Arduino, Raspberry PI and the ESP8266, among others.

The software for implementing the LoraWan protocol at the end devices (nodes) is IBM’s LMIC library, available at Github and on Platformio libs:

[ ID  ] Name             Compatibility         "Authors": Description
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[ 852 ] IBM LMIC framework arduino, atmelavr, atmelsam, espressif8266, intel_arc32, microchippic32, nordicnrf51, teensy, timsp430 "IBM, Matthijs Kooijman": Arduino port of the LMIC (LoraWAN-in-C, formerly LoraMAC-in-C) framework provided by IBM. | @1.5.0+arduino-1

Based on this library we can build code and nodes that are able to transmit data through LoraWan to a network backend. Specifically for the Arduino, the Atmega328 is the bare minimum to begin with, since the library, due to use of AES encryption functions occupies a lot of memory.

The backend can be provided by really anyone, including Telco operators, or private and crowd source operators like The Things Network (TTN). TTN provides the backend and management services, but depends on crowd sourced gateways, and might not be available at our area. Still it is possible, for testing, to build our own gateways, our buy them, and connect them to the Things Network. TTN doesn’t require any access fees (yet).

So with LoraWan an the Things Network, we can build our own nodes and gateways since code and hardware can be easily obtained, connect them and use it to build applications.

Regarding LoraWan we can also read this great introduction from Design Spark.

Lora hardware:

Anyway the easiest way for starting up using Lora and Lorawan, is to buy an Dragino Lora Shield and connect it to an Arduino Uno or Mega.

Dragino Lora Shield

Dragino Lora Shield

This is a great shield to startup since doesn’t need any soldering or complex configuration, just plug it on an Arduino shield and use the LMIC library and some code samples. Also the Dragino Shield can be used to build a simple gateway by connecting it to a Raspberry PI or ESP8266 to build what is called a single channel gateway, that allows to test gateway functionality, but it isn’t quite compatible with the LoraWan specifications. Anyway it gets the job done if there is no gateway nearby. Just make sure that you by version 1.3 or 1.4 of the shield. Mine also came with an SMA antenna.

Other alternatives to start using Lorawan are available at eBay/Aliexpress and other renowned name shops, namely Lora radio transceivers at around 8/16€, for example the HopeRF RFM95. Those we can also use them with Arduino or ESP8266 to build our nodes or single channel gateways.

Just make sure that the frequency for these modules and shields must match the allowed radio transmission frequency in your area. In my case, in Europe is 868Mhz, but for example at USA is 900Mhz.

Dragino Lora Shield Jumpers
The Shield has some jumpers and only looking at the schematic and cross referenced them with the RFM95 module (used by the shield as the Lora transceiver) I could see what they where for:

– There are two set of jumpers:

  • One defines the pins for SPI communication: SV# Jumpers;
  • The other set defines which data pins of the RFM95 module are exposed to the shield allowing them to be connected to the Arduino: JP# Jumpers.

The JP# jumpers connect the following if closed:

  • JP1 – RFM95 DI1 – Arduino pin D6
  • JP2 – RFM95 DI2 – Arduino pin D7
  • JP3 – RFM95 DI5 – Arduino pin D8

The RFM95 pin DI0 is permanently connected to Arduino pin D2.
The RFM95 pin RST (Reset) is permanently connected to Arduino pin D9.

Their function for Lora might be the following (The function depends how the RFM95 module is used)

  • RFM95 DI0 – Indicates if data was received and is ready to be read through the SPI bus. Indicates end of transmission for data that was previously sent: RXReady/TXReady
  • RFM95 DI1 – Receive timeout for Lora mode.
  • RFM95 DI2 – Receive timeout for FSK mode.
  • RFM95 DI5 – Used by Semtech Library. LMIC library doesn’t use it.

The SV# jumpers connect:

  • SV2 – SPI Clock line. Default on pin D13, otherwise on Arduino SPI CLK on the ICSP header.
  • SV3 – SPI Data line In (MOSI). Default on pin D11, otherwise on Arduino MOSI pin on the ICSP header.
  • SV4 – SPI Data line Out (MISO). Default on pin D12, otherwise on Arduino MISO pin on the ICSP header.

The SPI Chip Select line is always at pin D10.

So now we know that D10, D9 and D2 are used permanently connected and used by the shield, and the others can be connected or disconnected if needed or not.

LMIC software with Dragino Lora Shield:

To start using the Dragino Lora Shield so it connects to a LPWAN network, we can start using the following example: TTN node by ABP. ABP means Activation by Personalization, which means that all data for joining the network will be explicitly defined on the code. Other alternative would be OTAA: Over the air activation, where the gateway sends the configuration data to the node. Since we don’t know if we have a gateway in range, let’s start first with ABP mode.

The above code uses the LMIC library for implementing the LoraWan stack.
Since LMIC library doesn’t use DI05, we can remove the JP3 jumper, and free this IO line for other things, like another shield.

To use the LMIC library we must define first the pins that are used according to the shield configuration:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, 7}             //DIO pin mapping for DIO0, DIO1 and DIO2  
};

But DIO2 pin is only used for FSK modulation, and so if only using LoraWan we can also open up the JP2 jumper, and our LMIC pin configuration can be as follows:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, LMIC_UNUSED_PIN} //DIO pin mapping for DIO0, DIO1. DIO2 is not used  
};

Connecting to TTN
Connecting to The Things Network (TTN) depends of course of an available TTN gateway at the nodes range. Still we need to configure some parameters to allow the node to connect.

On this example the connection is done through Activation by Personalization. This means that we should put on our code the Network Session key and Application Session key on the code. To obtain these values we need to register on the TTN site, TTN Dashboard and add an ABP device..

From this site, then we can get the three configuration parameters that we need:

  • Device ID
  • Network Session key
  • Application Session key

Note by pressing the icon we can get the values in a form ready to paste it on the code:

/*************************************
 * TODO: Change the following keys
 * NwkSKey: network session key, AppSKey: application session key, and DevAddr: end-device address
 *************************************/
static u1_t NWKSKEY[16] = { 0xAA, 0x0F, 0x29, 0xD3, 0x9D, 0x7A, 0xAE, 0x3B, 0x54, 0xCF, 0xDF, 0x2F, 0x2A, 0x23, 0x55, 0xB5 };
static u1_t APPSKEY[16] = { 0x2F, 0x85, 0x43, 0x5B, 0x34, 0x9C, 0x80, 0xC6, 0xA8, 0xFA, 0x27, 0x49, 0x5A, 0x36, 0x82, 0x21 };
static u4_t DEVADDR = 0x95337738;

And that’s it, when running the sketch, and if in range of a gateway, the sent message should appear at the Dashboard:

Messages received for the device

Messages received for the device

Final thoughts:
In my area there are at least, supposedly, two active TTN gateways and I’m around 2Km’s away from them in a dense urban area.
But when running the sketch the first times, I had no results what so ever.

One of the configurations options for LoraWan is what is called a Spread Factor that, in a simplest way, exchanges range with on-air time for transmission, being SF7 the fastest speed and shorter range and SF12 the slowest speed and higher range. The sketch default value for the sprea factor was SF7, and changing it to SF12:

 ...
    // TTN uses SF9 for its RX2 window.
    LMIC.dn2Dr = DR_SF9;

    // Set data rate and transmit power for uplink (note: txpow seems to be ignored by the library)
    LMIC_setDrTxpow( DR_SF12 ,14);

    // Start job
    do_send(&sendjob);

With SF12 the message started be received and the sequence numbers of the messages received where continuous, so no lost messages.

Dropping to SF11, also worked fine, and the message sequence received and shown on the TTN Dashboard where still continuous.

At SF10, some of the messages where lost, almost 75% of them. In this case, arranging the antenna position (to exactly vertical) and placement did alter the reception successes.

Processing received data
After the data is at the TTN backend there are several ways of getting it. For reference on the TTN site there are instructions in how to access the data.

Crash course on Lora
For further information the next one hour video is a great starting point:

The slides for the presentation are available here:

https://www.dropbox.com/s/87htha4dq5b32wa/ttn-lora-crash-course.pdf?dl=0