Setting up Openocd for programming the Nordic nRF52832 chip

The following post organizes a set of information gathered from around the web to allow the use openocd utility to also program/access the nRF52 based chips.

Openocd must be patched to be used against the nRF52832 chip. Most of the required information is on this pcbreflux post.

Retriving the sources and patchs
At least, at the current date, on Arch Linux, Openocd version is 0.10. We will keep this version and install side by side the patched version of Openocd that supports the NRF52 chip.

First let’s retrieve the code and patchs:

[pcortex@pcortex:opt]$ git clone git://git.code.sf.net/p/openocd/code openocd-code
Cloning into 'openocd-code'...
remote: Counting objects: 55979, done.
remote: Compressing objects: 100% (20335/20335), done.
remote: Total 55979 (delta 46033), reused 43114 (delta 35495)
Receiving objects: 100% (55979/55979), 12.54 MiB | 2.88 MiB/s, done.
Resolving deltas: 100% (46033/46033), done.
[pcortex@pcortex:opt]$ cd openocd-code
[pcortex@pcortex:openocd-code|master]$ git pull http://openocd.zylin.com/openocd refs/changes/15/3215/2
remote: Counting objects: 1482, done
remote: Finding sources: 100% (11/11)
remote: Total 11 (delta 0), reused 2 (delta 0)
Unpacking objects: 100% (11/11), done.
From http://openocd.zylin.com/openocd
 * branch              refs/changes/15/3215/2 -> FETCH_HEAD
Auto-merging src/flash/nor/drivers.c
CONFLICT (content): Merge conflict in src/flash/nor/drivers.c
Auto-merging src/flash/nor/Makefile.am
CONFLICT (content): Merge conflict in src/flash/nor/Makefile.am
Automatic merge failed; fix conflicts and then commit the result.
[pcortex@pcortex:openocd-code|master *+|MERGING]$ 

After this we have two conflicting files that have changes that we must deal manually.

The src/flash/nor/drivers.c merge conflict is easy to solve. The conflict error is due to clash between empty sections and some added code lines. The end result file is here.

The Makefile.am takes more work to merge, but it is available -> here <- with the merges done.

Then, for specifically support the nRF52832 chip which has 512Kb of flash we need to change the openocd-code/src/flash/nor/nrf52.c currently on line 133, by adding the chip information:

static const struct nrf52_device_spec nrf52_known_devices_table[] = {
	{
		.hwid   = 0x0053,
		.variant  = "QFAA",
		.build_code = "AA",
		.flash_size_kb  = 512,
	},
{
                .hwid        = 0x00C7,
                .variant    = "QFN48",
                .build_code    = "B00",
                .flash_size_kb    = 512,
    	},
};

Done!

We just need now to compile:

    cd /opt/openocd-code/
    ./bootstrap
    ./configure
    make

In my case I just didn’t executed the last sudo make install since I want to have two versions of Openocd side by side.

We should have now at /opt/openocd-code/src the openocd binary with the patchs:

[pcortex@pcortex:src|master *+|MERGING]$ ./openocd -v
Open On-Chip Debugger 0.10.0+dev-00146-g1025be36-dirty (2017-06-04-10:06)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html

Connecting to the nRF52832 using ST-Link

We can now use this version to connect to the nRF52 chip.
A sample openocd config file can be as follows:

nRF52832.cfg

#nRF52832 Target
source [find interface/stlink-v2.cfg]

transport select hla_swd

source [find target/nrf52.cfg]

But since I did not run the sudo make install command the tcl files from the Arch Linux package are different from the patched version, so it is important that the patched version also uses the patched/new files:

./openocd -s /opt/openocd-code/tcl -f nRF52832.cfg

The expected output:

Open On-Chip Debugger 0.10.0+dev-00146-g1025be36-dirty (2017-06-04-10:06)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 10000 kHz
Info : Unable to match requested speed 10000 kHz, using 4000 kHz
Info : Unable to match requested speed 10000 kHz, using 4000 kHz
Info : clock speed 4000 kHz
Info : STLINK v2 JTAG v21 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.261252
Info : nrf52.cpu: hardware has 6 breakpoints, 4 watchpoints

We should be able now to run the nrf52 command:

telnet localhost 4444
Trying ::1...
Connection failed: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
> nrf52
nrf52
  nrf52 mass_erase
nrf52.cpu
  nrf52.cpu arm
    nrf52.cpu arm core_state ['arm'|'thumb']
...

And programming:

> program /opt/nRF5_SDK_12.3.0_d7731ad/examples/ble_peripheral/ble_app_uart/hex/ble_app_uart_pca10028_s130.hex verify
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x000006d0 msp: 0x000007c0
** Programming Started **
auto erase enabled
nRF51822-QFN48(build code: B00) 512kB Flash
Padding image section 0 with 2112 bytes
Padding image section 1 with 32 bytes
using fast async flash loader. This is currently supported
only with ST-Link and CMSIS-DAP. If you have issues, add
"set WORKAREASIZE 0" before sourcing nrf52.cfg to disable it
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000001e msp: 0x000007c0
wrote 126976 bytes from file /opt/nRF5_SDK_12.3.0_d7731ad/examples/ble_peripheral/ble_app_uart/hex/ble_app_uart_pca10028_s130.hex in 3.972911s (31.211 KiB/s)
** Programming Finished **
** Verify Started **
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x000007c0
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x000007c0
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x000007c0
verified 121752 bytes in 1.346008s (88.334 KiB/s)
** Verified OK **
> reset run

Remember, if programming fails, with the program command, check if the correct path is provided for openocd through the -s switch.

Done!

Advertisements

NodeJS BLE Applications using BLENO on Arch Linux

BLENO is a greate NodeJS based library for building applications that communicat with other devices (Smartphones, tables, sensor tags) using Bluetooth Low Energy (BLE).

This post is just to quickly document some requirements for successfully use the BLENO library, in my case, on Arch Linux running the latest Plasma (KDE) desktop.

The tools:

Most the information available on the internet for using and controlling the bluetooth adapter uses the now deprecated tools hcitool, hciconfig and so on. Check here the deprecated list of commands.

So we need to use the new tools from the latest Bluez (Bluetooth Linux implementation): btmgm, btinfo, …

Making Bleno examples work:

The simplest example to try out the BLENO library is the battery example located at: […]/bleno/examples/battery-service

First let’s check if our computer/laptop bluetooth adapter is available: Note that all commands must be ran as the root user:

root@pcortex:/opt/bleno/examples/battery-service# btinfo local
Bluetooth information utility ver 5.45
Failed to open HCI user channel
root@pcortex:/opt/bleno/examples/battery-service# 

This issue can be circumvented by stopping the higher level bluetooth stack:

root@pcortex:/opt/bleno/examples/battery-service# systemctl stop bluetooth
root@pcortex:/opt/bleno/examples/battery-service# btinfo local
Bluetooth information utility ver 5.45
HCI version: 6
HCI revision: 7869
LMP version: 6
LMP subversion: 64512
Manufacturer: 2
root@pcortex:/opt/bleno/examples/battery-service#

In case of previously disabling the Bluetooth through the graphical interface:

Disabling the Bluetooth here will have this behaviour (in this case the bluetooth service is still running):

root@pcortex:/opt/bleno/examples/battery-service# systemctl start bluetooth   (<- After this disable bluetooth on the graphical interface)
root@pcortex:/opt/bleno/examples/battery-service# btinfo local
Bluetooth information utility ver 5.45
Failed to open HCI user channel
root@pcortex:/opt/bleno/examples/battery-service# btmgmt power on
Set Powered for hci0 failed with status 0x12 (Blocked through rfkill)
root@pcortex:/opt/bleno/examples/battery-service# 

Even stopping the Bluetooth service keeps the BT adapter disabled:

root@pcortex:/opt/bleno/examples/battery-service# systemctl stop bluetooth
root@pcortex:/opt/bleno/examples/battery-service# btmgmt power on
Set Powered for hci0 failed with status 0x12 (Blocked through rfkill)
root@pcortex:/opt/bleno/examples/battery-service#

We can check this with the rfkill command:

root@pcortex:/opt/bleno/examples/battery-service# rfkill list
0: phy0: Wireless LAN
        Soft blocked: no
        Hard blocked: no
2: hci0: Bluetooth
        Soft blocked: yes
        Hard blocked: no
root@pcortex:/opt/bleno/examples/battery-service# 

We can unblock now the adapter:

root@pcortex:/opt/bleno/examples/battery-service# rfkill unblock 2
root@pcortex:/opt/bleno/examples/battery-service# rfkill list
0: phy0: Wireless LAN
        Soft blocked: no
        Hard blocked: no
2: hci0: Bluetooth
        Soft blocked: no
        Hard blocked: no
root@pcortex:/opt/bleno/examples/battery-service# btinfo local
Bluetooth information utility ver 5.45
HCI version: 6
HCI revision: 7869
LMP version: 6
LMP subversion: 64512
Manufacturer: 2
root@pcortex:/opt/bleno/examples/battery-service# btmgmt power on
hci0 class of device changed: 0x00010c
hci0 Set Powered complete, settings: powered bondable ssp br/edr le secure-conn 
root@pcortex:/opt/bleno/examples/battery-service# 

So why we are having all this work for making sure that the BT adapter is powered on AND the bluetooth stack is stopped (systemctl stop bluetooth).

The answer is quite simple. If we don’t do this the BLENO examples will seem to work (they start) but the BLE advertised services are the bluetooth Bluez services and not our code.

To explain, check the following behaviour where we start the BLENO Battery Service with the Bluetooth stack started:

root@halcyon:/opt/bleno/examples/battery-service# systemctl start bluetooth
root@halcyon:/opt/bleno/examples/battery-service# node main.js 
on -> stateChange: poweredOn
on -> advertisingStart: success
setServices: success

Using the Nordic nRF Connect Android App we can see the non working behaviour vs what we should expect from the Bleno Battery example:

BLE Scan Results

Pressing Connect we can see on Client that no service are provided. This is due to the fact that the desktop bluetooth is enabled):

Now let’s disable the bluetooth stack (which powers the BT adapter) and start again the Bleno Battery example:

root@pcortex:/opt/bleno/examples/battery-service# systemctl stop bluetooth
root@pcortex:/opt/bleno/examples/battery-service# node main.js 

Example hangs in here, because BT adapter is disabled/off

^Croot@pcortex:/opt/bleno/examples/battery-service# btmgmt power on
hci0 class of device changed: 0x00010c
hci0 Set Powered complete, settings: powered bondable ssp br/edr le secure-conn 
root@pcortex:/opt/bleno/examples/battery-service# node main.js 
on -> stateChange: poweredOn
on -> advertisingStart: success
setServices: success

And now if we scan again and connect to the Battery example with our mobile phone through the Nordic application we have:

It works now!

We can confirm that because on the file battery-service.js the service identifier is defined:

function BatteryService() {
  BatteryService.super_.call(this, {
      //uuid: '180F',
      uuid: 'ff51b30e-d7e2-4d93-8842-a7c4a57dfb07',
      characteristics: [
          new BatteryLevelCharacteristic()
      ]
  });
}

and it is the same detected by the Android application.

CentoOS 7 on Virtual Box

These are some tips and information for configuring CentoOS 7 when running on VirtualBox, namely how to install the Virtual Box Extensions.

Checking the network

After installing we should check if we have connectivity namely if the network interface is up or not.

We can configure the network interface has NAT, which will use an internal DHCP server, but this means that the machine will not be accessible without some kind of port forwarding, or we can configure the interface in BRIDGE mode, which means that the machine will behave as the network interface is directly connected to the physical network. In this case an external DHCP server must be available to provide the interface address or we need to provide a static address.

In any case, I normally use the BRIDGE mode with an external DHCP server.

If by any chance the interface is up but has no address, check the network configuration file located in:

/etc/sysconfig/network-scripts/

If our interface is, as in this example, enp0s3 we should edit the file on the above directory named ifcfg-enp0s3 and change the entry (probably at the end of the file) from ONBOOT=no to ONBOOT=yes and reboot. (Note: Not always I needed to do this)

At the end the command ip a should show an active interface:

[root@localhost ~]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:55:35:7b brd ff:ff:ff:ff:ff:ff
    inet 10.191.0.75/22 brd 10.191.0.255 scope global dynamic enp0s3
       valid_lft 598748sec preferred_lft 598748sec

Updating the system:
Before we are able to install the Virtual Box extensions we should first fully update the system and install some required packages:

yum upgrade
yum install bzip2
yum groupinstall "Development Tools"
yum install "Kernel-devel-uname-r == $(uname -r)"

Installing the guest additions:

After these per-requisites are installed we should go to the Virtual Box menu and Select Devices->Insert Guest Additions CD Image.

Selection_212

Always as the user root, we can do now two things, mount the CDROM on the directory /mnt or on a created subdirectory on /mnt. Let’s do the later:

[root@localhost ~]# cd /mnt
[root@localhost mnt]# mkdir cdrom
[root@localhost mnt]# mount -r /dev/cdrom /mnt/cdrom
[root@localhost mnt]# ls cdrom
32Bit  64Bit  AUTORUN.INF  autorun.sh  cert  OS2  runasroot.sh  VBoxLinuxAdditions.run  VBoxSolarisAdditions.pkg  VBoxWindowsAdditions-amd64.exe  VBoxWindowsAdditions.exe  VBoxWindowsAdditions-x86.exe

We can now install the additions:

[root@localhost mnt]# cd /mnt/cdrom
root@localhost cdrom]# ./VBoxLinuxAdditions.run 
Verifying archive integrity... All good.
Uncompressing VirtualBox 5.1.22 Guest Additions for Linux...........
VirtualBox Guest Additions installer
Removing installed version 5.1.22 of VirtualBox Guest Additions...
Copying additional installer modules ...
Installing additional modules ...
vboxadd.sh: Starting the VirtualBox Guest Additions.
[root@localhost cdrom]# 

It should work without any issues. We can now check if the Virtual Box Guest additions modules are running:

[root@localhost cdrom]# lsmod | grep vbox
vboxsf                 39741  0 
vboxguest             297360  2 vboxsf
[root@localhost cdrom]# 

Success!

Mounting the shared folders
We are now able to mount the shared folders that we define on the virtual box machine configuration.

For example let’s create the following share:
First select Devices->Shared Folders->Shared Folder Settings

Selection_213

And then add a share:
Selection_214

Two important things: The share name is the name that will be used on the mount command. And if the Make Permanent tick box is not selected, the share will be lost on reboot.

So we now can do the following to mount the shared folder:

[root@localhost mnt]# mkdir HostOpt
[root@localhost mnt]# mount -t vboxsf VBOXShare /mnt/HostOpt/
/sbin/mount.vboxsf: mounting failed with the error: Protocol error
[root@localhost mnt]# mount -t vboxsf VBOXShare /mnt/HostOpt
/sbin/mount.vboxsf: mounting failed with the error: Protocol error
[root@localhost mnt]# mount -t vboxsf VBOXShare /mnt/HostOpt

The can happen at least because of two reasons:
First: The share name is invalid. In this case the share name is VBOXShare, but vboxshare can also be used
Second: The configuration is not yet valid or applied (Pressing ok on the create share window).

And that’s it.

Using Node-Red and Grafana WorldMap for geolocalized data visualization

Based on my previous posts we are now able to build a system that can receive, store and visualize data by using Node-Red, InfluxDB and Graphana. Grafana allows us build dashboards, query and visualize the stored data across time efficiently by using, in our case, the InfluxDB database engine. So far we’ve used simple line/bar charts to visualize data but we can use both Node-Red and Grafana to plot data onto a map:

  1. NodeRed Contrib World Map: Openstreet UI based map for plotting data with several options, including icon types, vectors, circles and heatmaps totally controlled through nodered flows.
  2. Grafana WorldMap plugin: Grafana panel with also an OpenStreet map for visualizing data.

Both have pros and cons, but the main differences between the two is that Node-Red Worldmap is suited more to real time display, and the Grafana plugin is better adapted to display data based on some time based query. Other major difference is that Node-Red Worldmap would require some coding, but, at least I consifer it, at an easy level, and the Grafana plugin is much harder to make it work.

Mapping data using Node-Red Worldmap:
One of the easiest ways for mapping data in realtime is using Node Red Worldmap node. The map is plotted and updated in real time.

cd ~
cd .node-red
npm install node-red-contrib-web-worldmap

After restarting and deploying a worldmap node, the map should be available at: http://server:1880/worldmap or other URL depending on the Node-Red base configuration.

One thing to keep in mind is that Node-Red is single user, so all instances of world maps (several different clients/browsers) will always have the same view.
The simplest way to start using the worldmap is just to copy and deploy the demo workflow provided by the node, but the key concept is that each point has a name and a set of coordinates.

msg.payload = {};
msg.payload.name = "CentralLX";
msg.payload.lat = 38.7223;
msg.payload.lon = -9.1393;
msg.payload.layer = "SensorData";
msg.payload.UVLevel = getUVLevel() ;
msg.payload.Temperature = getTemp();

The cool thing is that if we inject repeatedly the above message (keeping the same name) but with different coordinates, the data point will move across the map in real time, and as I said earlier, the move will be reflect onto every client.

So all we need is an Inject node to a function node with the above code and feed it to the world map:

At the end we get this at the URL http://server:1880/worldmap:

Mapping data using Grafana Worldmap plugin:

The Grafana Worldmap plugin can get the location data in several ways. One of them is to use geohash data that is associated to the values/measurements.
There is a Node Red Geohash node that generates the geohash value from the latitude and longitude of data location. As usual we install the node:

cd ~
cd .node-red
npm install node-red-node-geohash

and then the Grafana plugin. We just follow the plugin instructions:

cd ~
grafana-cli plugins install grafana-worldmap-panel
/etc/init.d/grafana-server restart

With this per-requisites installed we can now feed data onto the database, in our case InfluxDB, that will be used by Grafana. We just need make sure that we add the geohash field. The geohash node will calculate from the node-red message properties lat – latitude and lon – longitude the required info:

A simple example:

Using the Influx tool, we can query our database and see that the geohash localization is now set:

> select * from SensorData limit 2
name: DemoValue
time                Temp UVLevel geohash    lat       lon        
----                ---- ------- -------   ---------- ----------  
1490706639630929463 22   8       eyckpjywh 38.7045055 -9.1754669  
1490706651606553077 21   7       eyckpjzjr 38.7044008 -9.1746488 

Anyway for setting up the World map plugin to display the above data was not straight forward, so the following instructions are more for a startup point rather than a solution.

The first thing to know is that the plugin is waiting for two fields: geohash and metric. With this in mind, before wasting too much time with the map plugin, a table panel that is filled with the required query is a precious tool to debug the query:

After we infer from the table that the data is more or less the data we want, we just transfer the query to WorldMap plugin:

Notice two important things: The aliasing for the query field to metric with the alias(metric) instruction, and the Format as: Table.

We can now setup the specific Worldmap settings:

On the Map Visual Options , I’ve centered the map in my location and set the zoom level. Fiddling around here can be seen in real time.

On the Map Data Options for this specific example, the Location Data comes from a table filled with the previous query (hence the format as table on the query output), and we want to see the current values with no aggregation.

When hoovering around a spot plotted on the map we can see a label: value, and the label used is obtained from a table field. In my case I just used geohash (not really useful…). Anyway these changes only work after saving and reloading the panel with F5 in my experience.

At the end we have now graphed data and localized data:

If we drag the selector on the left graphic panel, or select another time interval on top right menu of the grafana dashboard, the visualized information on the map changes.

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 4: Installing Grafana and configuring

Our software stack installation ends with configuring and installing Grafana. Ubuntu installation instructions are at this Grafana documentation link..

Installation is as easy as to download and install the latest Grafana dashboard version:

root@server:~# wget https://grafanarel.s3.amazonaws.com/builds/grafana_4.1.2-1486989747_amd64.deb
root@server:~# sudo dpkg -i grafana_4.1.2-1486989747_amd64.deb

In my case, the above instructions where enough to install successfully the software.

After installing if we will access the Dashboard through a reverse proxy, check out my previous post to configure correctly the Grafana server: Grafana Reverse Proxy

We can now start the Grafana server but if the server is exposed to the public internet the first thing to do is to change the default admin user password to something safe, or better yet, create a new admin user with a strong password and delete the admin default account.

We should press the top left icon, choose admin and then Global users. We can now select the Admin user and change the default password:

Here we also can create a new admin account, which is my recommendation.

Data source configuration:

Dashboard data provides from data sources that we configure on  Grafana. We can configure several data sources that we will used to fetch data to feed to our dashboards and graphs.

In our case we will configure a single InfluxDB datasource. Note that we didn’t defined any authentication for the InfluxDB server, so no access credentials are needed yet, but we should set ones as soon as possible.

Since Grafana and InfluxDB are running on the same server we use the base URL http://localhost:8086. Make sure that the connection type is set as proxy.

After setting the datasource name and datasource database, press Save & Test and it should report success.

Initial Dashboard configuration

We are now able to create a basic first dashboard by building some charts/graphs based on the previous created data source.

Press the New Dashboard button to create a new dashboard. In case that you don’t have a screen as the above screen shot, just press the top left Grafana Icon, select Dashboards and then New.

The following screen should appear where we can add graphical panels to show our data. We should the way that we want to show our data, by selecting the Graph type. The selected graph type should reflect immediately on the panel below. To edit the panel source data we should press the panel title and a pop-up window will appear, and then we press Edit.

We are now able to edit the associated queries that will fill the panel with data. Associated with the panel we can have multiple queries for the source data. In the default case a default query named A is already set up. We just need to change our measurement field to the correct one, which in our case could be Temperature or heap, based on the previous post (InfluxDB data configuration)

.

We need now to press the Select Measurement and a drop down box with the available measurement should appear. We select the one that we want and data should now appear on the above graph.

From now on is just a bit of perfecting things out like giving a sound name to the panel in the General tab and at the end pressing the Save icon on the top of the screen.

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 3: Single point of access – Reverse proxy the services with nginx

Since we will be running a lot of services, each running on its own port, the following configuration, is optional, but allows to access all services through the same entry point by using Nginx server as a reverse proxy to Node-Red, Node-Red UI/Dashboard, Node-Red Worldmap and Grafana.

With this configuration the base URL is always the same without any appended ports, and the only thing that changes are the URL path:

http://server/nodered
http://server/nodered/worldmap
http://server/grafana

To allow this we install and configure Nginx:

apt-get install nginx

The configuration files will reside in /etc/nginx directory. Under that directory there are two directories: sites-available and sites-enable where the later normally contains a link to configuration files located at sites-available.
At that directory there is a file named default that defines the default web site configuration used by Nginx. This is the file where we will add the reverse proxy directives.

Reverse proxy for Node-Red and Node-Red Contrib Worldmap
For setting up the reverse proxy for Node-Red we must first change the base URL for Node Red from / (root) to something else that we can map the reverse proxy.

For this we will need to edit the settings.js file located on the .node-red directory on the home path of the user running Node-Red.

We need to uncomment and change the entry httpRoot to point to our new base URL.

   httpRoot: '/nodered',

Don’t forget the trailing comma.

We need to restart now Node-Red and it should be accessible at the URL http://server:1880/nodered instead of http://server:1880/.

To configure Nginx, we edit the file default at /etc/nginx/sites-available and add the following section:


        location  /nodered {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_pass "http://127.0.0.1:1880";
        }

        location /socket.io {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_pass "http://127.0.0.1:1880";

        }

Note the following: The first location defines the reverse proxy URL /nodered to be served by the backend server http://127.0.0.1:1880. The incoming path, /nodered, will be passed to the backend server URL /nodered, since paths are passed directly. No need to add the /nodered path to the backend server definition.
Also I’m using the 127.0.0.1 address instead of localhost to avoid the IPv6 mapping to the localhost. In this way I’m sure that IPv4 will be used.

The location mapping for /nodered will make all the functionality of node red to work as it should at the base url /nodered. But some nodes, like node-red-contrib-worldmap will request to the proxy server ignoring the node-red base root map. Hence the /socket.io mapping. It will allow the worldmap nodes to work, but will stop this mapping to be used for something else.

Reverse proxy for Grafana

Setting up the reverse proxy for Grafana we can, and should use the following documentation: Grafana Reverse Poxy. For me the following configuration worked:

First edit the [server] section on the Grafana configuration file grafana.ini located at /etc/grafana.

Uncomment and edit the following lines:

[server]
# Protocol (http or https)
protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = server.domain.com

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = http://server.domain.com/grafana/

Note the ending slash at the root_url. The same applies to the Nginx configuration

The files for the Nginx configuration are the same as the above configuration for reverse proxy.

We just need to add the following section after the previous location directives:

        location /grafana/ {
                proxy_pass http://localhost:3000/;
        }

We should now restart nginx to refresh the configuration, and all should be working as it should by accessing the Grafana dashboard at http://server.domain.com/grafana

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 2: Database configuration and data collection

On the previous post we’ve installed and the base software for our Grafana based dash board.

We need now to configure our InfluxDB database and Node Red to start collecting data.

Configuring InfluxDB:
Detailed instructions for configuring an InfluxDB database are on this InfluxDB documentation link..

The main concepts that we need to be aware when using the InfluxDB is that record of data has a time stamp, a set of tags and a measured value. This allows, for example to create a value named Temperature and tag it depending on the source sensor:

Temperature: Value=22.1 , Sensor=Kitchen
Temperature: Value=21.9 , Sensor=Room1

This allows to process all the data or only process data based on a certain tag or tags. Values and tags can be created on the fly without previously define them, which is a bit different from standard RDBMS engines.

Creating an InfluxDB database:
To create the database, we need to access the machine hosting the InfluxDB server and execute the command influx:

odroid@odroid:~$ influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> create database SensorData
> show databases
name: databases
name
----
_internal
SensorData

> 

Now we have our database created and I’ve named SensorData. To make an example with the above temperature data we can do the following:

> insert Temperature,Sensor=kitchen value=22.1
ERR: {"error":"database is required"}

Note: error may be due to not setting a database or retention policy.
Please set a database with the command "use " or
INSERT INTO . 
> use SensorData
Using database SensorData
> 

As we can see we need first to select the database where we are going to insert data with the command use SensorData:

> use SensorData
Using database SensorData
> insert Temperature, Sensor=kitchen value=22.1
ERR: {"error":"unable to parse 'Temperature, Sensor=kitchen value=22.1': missing tag key"}

> insert Temperature,Sensor=kitchen value=22.1
> insert Temperature,Sensor=Room1 value=21.9
> select * from Temperature
name: Temperature
time                Sensor  value
----                ------  -----
1487939008959909164 kitchen 22.1
1487939056354678353 Room1   21.9

Note that we can’t use spaces between the Measure name and the tags. The correct syntax is as follows:

 insert MeasureName,tag1=t1,tag2=t2,...   value1=val1,value2=val2,value3=val3,....

Also note that no DDL (data definition language) was used to create the tags or the measured value, we’ve just inserted data for our measurement with the our tags and value(s) without the need of previously define the schema.

Configuring Node-Red
Since we now have a database we can configure the InfluxDB Node Red nodes to store data onto the database:

There are two types of InfluxDB nodes, one that has an Input and Output and other that only has Input. The former is for making queries to the database where we provide on the input node the query, and on the output the results are returned. The later is for storing data only onto the database.
For both nodes we need to configure an InfluxDB server:

InfluxDB Server Configuration

We need to press the Pen icon right next to the server to add or reconfigure a new InfluxDB server:

InfluxDB server

A set of credentials are required, but since I’ve yet configured security, we can just put admin/admin as username and password. In a real deployment we must activate security.

From now on it is rather simple. Referring to InfluxDB node configuration screenshot (Not the InfluxDB server configuration) we have a configuration field named Measurement. This is our measure name that we associate a value. Picking up on the above example with the Insert command it will be Temperature, for example.

Now if the msg.payload provided has input to the node is a single value, let’s say 21, this is equivalent to do:

Insert Temperature value=12

We other formats for msg.payload that allows to associate tags and measures. Just check the Info tab for the node.

Simple example:

The following flow shows a simple example of a value received through MQTT, in this case the free heap from one of my ESP8266 and its storage in InfluxDB:

Sample Flow

[{"id":"20bec5de.8881c2","type":"mqtt in","z":"ced40abb.3c92e","name":"Heap","topic":"/outbox/ESP12DASH/Heap","qos":"2","broker":"2a552b3c.de8d2c","x":83.16668701171875,"y":206.41668701171875,"wires":[["e0d9c912.8c57f8","876fb151.6f2fa"]]},{"id":"876fb151.6f2fa","type":"debug","z":"ced40abb.3c92e","name":"","active":true,"console":"false","complete":"false","x":408.5,"y":177,"wires":[]},{"id":"e0d9c912.8c57f8","type":"influxdb out","z":"ced40abb.3c92e","influxdb":"bbd62a93.1a7108","name":"","measurement":"heap","x":446.1666717529297,"y":224.58335876464844,"wires":[]},{"id":"2a552b3c.de8d2c","type":"mqtt-broker","broker":"192.168.1.17","port":"1883","clientid":"node-red","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":15,"cleansession":true,"willQos":"0","birthQos":"0"},{"id":"bbd62a93.1a7108","type":"influxdb","z":"","hostname":"127.0.0.1","port":"8086","protocol":"http","database":"SensorData","name":"ODroid InfluxDB"}]

We can see with this flow the data stored in InfluxDB:

> select * from heap;
name: heap
time                value
----                -----
1487946319638000000 41600
1487946440913000000 41600
1487946562206000000 41600
1487946683474000000 41600
1487946804751000000 41600
1487946926061000000 41600
1487947047309000000 41616
1487947168594000000 41600

Now we have data that we can graph with Grafana, subject of my next posts.