Arch Linux with full encrypted disks

So, I’ve bought a new light weight laptop, an HP Envy 13.3 1.2Kg I7, where I wiped out Windows 10 and installed Arch Linux.

Just for security reasons I’ve decided to do a full disk encryption install, including boot.

There are several instructions on the Web, including videos on youtube in how to do it, and so on this gist I have my instructions for the installation, based off course in other gists and instructions.

In this post is the configuration instructions that worked for me and also to allow better find from google/bing/ddg.

# WARNING: WORK IN PROGRESS, USE THESE STEPS WITH CAUTION. IT WILL CLEAR ALL DISK DATA!!
I REALLY recommend first to use a Virtual Box machine with EFI support enabled to test everything before doing it on a real machine.

# Arch installation on a HP ENVY 13 inch laptop (ah0006np part number: 16GB Ram, 512GB SSD)

OBJECTIVE:
Install Arch Linux with encrypted boot, root and swap filesystems and boot from UEFI, completly dumping Windows on the process.
No dual boot.
Windows, if necessary will be run on a Virtual Machine and re-use the Windows key that came with the laptop.

The configuration will be LVM on LUKS. Also a major difference from other tutorials is that the boot partition is also encrypted, and not a standard partition.

# Results so far:

– Disk encryption ok. GRUB boots slow (20s). Otherwise works fine.
– Wireless works OOTB, but errors on dmesg output from time to time when there is high network traffic.
– Sound and microfone works ok.
– Webcam does work but needs configuration: See below at the end.
– Keyboard special keys work fine (brightness, Sound, Mute), including keyboard background lights, but F6 sound Mute Led does not work.
– Some screen corruption with the Intel Driver either SNA or UXA. Nouveau crashes, nvidia driver didn’t work. To be checked -> Issue with QT 5 and Konsole/Kate applications, not a Intel Driver issue.
– KDE SDDM doesn’t recover well if screen DPMS is activated. I’ve disable it so far to solve it.
– Suspend/resume works fine.
– Battery time so far, around 4/5 hours.


# Desired disk layout:

+---------------+----------------+----------------+----------------+
|ESP partition: |Boot partition: |Volume 1:       |Volume 2:       |
|               |                |                |                |
|/boot/efi       |/boot           |root            |swap            |
|               |                |                |                |
|               |                |/dev/vg0/root   |/dev/vg0/swap   |
|/dev/sda1      |/dev/sda2       +----------------+----------------+
|unencrypted    |LUKS encrypted  |/dev/sda3 encrypted LVM on LUKS  |
+---------------+----------------+---------------------------------+

The final result is to have an Arch Linux Installation with full disk encryption and with a basic set of applications such as the KDE Plasma Desktop.

These instructions have several sources, namely:
https://grez911.github.io/cryptoarch.html
and this WordPress post.

The installation process on this guide is for the Arch Linux installation onto an HP Envy 13, 16GB RAM with 512MB ssd laptop. This laptop comes with Windows 10 Home installed, and as far as my model goes, it comes with an Intel WiFi board and a WD Sandisk SN520 512GB NVME SSD.

The official Arch installation guide contains more details that you should refer to during this installation process.
That guide resides at: https://wiki.archlinux.org/index.php/Installation_Guide

## Boot from image

Download the archlinux-\*.iso image from https://www.archlinux.org/download/ and its GnuPG signature.
Use gpg –verify to ensure your archlinux-\*.iso is exactly what the Arch developers intended. For example
at the time of installation:

$ gpg --verify archlinux-2017.10.01-x86_64.iso.sig
gpg: Signature made Sun 01 Oct 2017 07:29:43 AM CEST using RSA key ID 9741E8AC
gpg: Good signature from "Pierre Schmitz "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 4AA4 767B BC9C 4B1D 18AE  28B7 7F2D 434B 9741 E8AC

Currently the Arch ISO is archlinux-2018.11.01-x86_64.iso.

Burn the archlinux-\*.iso to a 1+ Gb USB stick. You can use the dd command, unetbootin or Etcher.

Connect the USB stick to the usb port and power on/cycle the machine to boot.
If your USB stick fails to boot, ensure that Secure Boot is disabled in your UEFI configuration.

Note: To access the BIOS on the Envy Laptop, turn on the laptop and press several times the ESC key or the F10 key to access the BIOS while the screen is black.
First I moved the boot order to have the USB boot at the top.
Then we need to disable the secure boot option and press F10 to save. Confirm saving it.

Attention now: There is a confirmation screen to really commit the secure boot option change. Enter the requested code and save.

After booting up:

Set your keymap only if not you are not using the default English keyboard.

$ loadkeys pt-latin1

We can now, if required backup the HP recovery partition, that I suppose is the Windows Install Media.

# Connect to the Internet.

Execute the wifi-menu command and select a Wifi network. On this HP Envy, the wireless card (Intel) was detected with no issues.

Check with the “ip a” command if there is network connection.

## Prepare your hard disk

In the next steps we will create necessary partitions and encrypt the main partition.

Find the correct block device

$ lsblk

In my case the correct block device (the NVME SSD of my laptop) is ‘nvme0n1’. (Depends on the machine)

Create and size partitions appropriate to your goals using gdisk.

$ gdisk /dev/nvme0n1

Press p to show the partitions.

In my case I have a 260Mb EFI partition, a 16MB Reserved Microsoft Partition, a 460GB partition and a 980MB and another 15GB partition.

From this point on, everything that is to be done, will destroy the disk data.

# Delete all partitions on disk

Use the d command to delete all partitions. Use d, then partition number. Repeat for all partitions

Press o to create the GPT.

Create three partitions: One for the EFI, one for boot and the other will be used to have the Arch Linux installation. To create a partition, press n:

1. Partition 1 = 512 MB EFI partition (Hex code EF00). Initial Sector: ; End: 512M; Type: EF00
2. Partition 2 = 1GB Boot partition (Hex code 8300)
3. Partition 3 = Size it to the last sector of your drive. (default) (Hex code 8E00 – Linux LVM Partition)

Review your partitions with the ‘p’ command.
Write your gdisk changes with ‘w’.

Check again the names with the blkid command to know the partitions name:

1. EFI: /dev/nvme0n1p1
2. BOOT: /dev/nvme0n1p2
3. Arch: /dev/nvme0n1p3

# Create filesystems
The EFI filesystem must be FAT32:

$ mkfs.vfat -F 32 /dev/nvme0n1p1

The other filesystems are to be encrypted.

(optional) Before creating the partitions we can use the command

cryptsetup benchmark 

to see how fast the different encryption algoritms are.

# Encrypted /boot partition

$ cryptsetup -c aes-xts-plain64 -h sha512 -s 512 --use-random luksFormat /dev/nvme0n1p2
$ cryptsetup open /dev/nvme0n1p2 cryptboot
$ mkfs.ext4 /dev/mapper/cryptboot

The first command will ask for the disk passphrase. Do not forget it!.

ATTENTION:
The first crypsetup command will set the LUKS with default iter-time parameters, which may or may not make grub to boot slow (around 20s). If this is not fine add the following parameter: –iter-time=5000 (This will affect security, so use a large key phrase)

The last command will create a /dev/mapper/cryptboot device.
We can check that it was created with the command ls /dev/mapper

# Create encrypted LUKS device for the LVM

cryptsetup -c aes-xts-plain64 -h sha512 -s 512 --use-random luksFormat /dev/nvme0n1p3
cryptsetup open /dev/nvme0n1p3 cryptlvm

## Create encrypted LVM partitions

These steps will create the required root partition and an optional partition for swap.
Modify this structure only if you need additional, separate partitions. The sizes used below are only suggestions.
The VG and LV labels ‘ArchVG, root and swap’ can be changed to anything memorable to you. Use your labels consistently, below!

$ pvcreate /dev/mapper/cryptlvm
$ vgcreate ArchVG /dev/mapper/cryptlvm
$ lvcreate -L +16G ArchVG -n swap
$ lvcreate -l +100%FREE ArchVG -n root

Again, we can see on /dev/mapper if the logical volumes where created.

## Create filesystems on your encrypted partitions

$ mkswap /dev/mapper/ArchVG-swap
$ mkfs.ext4 /dev/mapper/cryptboot
$ mkfs.ext4 /dev/mapper/ArchVG-root

Mount the new system

mount /dev/mapper/ArchVG-root /mnt
swapon /dev/mapper/ArchVG-swap
mkdir /mnt/boot
mount /dev/mapper/cryptboot /mnt/boot
mkdir /mnt/boot/efi
mount /dev/nvme0n1p1 /mnt/boot/efi

# Install the Arch system:

This installation command provides a decent set of basic system programs which will also support WiFi when initially booting into your Arch system.

At this point we need to have a network connection. Since the HP only has Wifi connection, we need to setup the WiFi connection. Other alternative is to use an Ethernet USB dongle that is recognized by the Arch boot ISO.
Also, if you are behind a proxy, you can set the http_proxy and https_proxy variables to access the internet.

(Optional) Use reflector to speedup download (credit goes to u/momasf)

Change COUNTRY to (surprise) your country name.

pacman -Sy reflector
reflector --country 'COUNTRY' --age 12 --protocol https --sort rate --save /etc/pacman.d/mirrorlist

I won’t install base-dev here to save time at the installation.

$ pacstrap /mnt base grub-efi-x86_64 efibootmgr dialog wpa_supplicant vim

# Create and review FSTAB
The -U option pulls in all the correct UUIDs for your mounted filesystems.

 $ genfstab -U /mnt >> /mnt/etc/fstab
 $ nano /mnt/etc/fstab  # Check your fstab carefully, and modify it, if required.
 

Enter the newly installed system

$ arch-chroot /mnt /bin/bash

Set the system clock, you can replace UTC with your actual timezone

$ ln -fs /usr/share/zoneinfo/Europe/Lisbon /etc/localtime
$ hwclock --systohc --utc

Assign your hostname

$ echo mylaptop > /etc/hostname

My requirements for the locale are:
– Metric system
– 24h time format
– dd/mm/yyyy date format
– Portuguese language
– A4 paper size
– But all help, error messages are in English

The *pt_PT.UTF-8* plus *en_US.UTF-8* locale meets those requirements. To set up this locale:

– In /etc/locale.gen

en_US.UTF-8 UTF-8
pt_PT.UTF-8 UTF-8

– In /etc/locale.conf, you should **only** have this line:

LANG=en_US.UTF-8

We will change other settings on Bash profile.

Now run:

$ locale-gen

Create a new file vconsole.conf so that the console keymap is correctly set at boot. Create the file and add the following line:

KEYMAP=pt-latin1

Set your root password

$ passwd

Create a User, assign appropriate Group membership, and set a User password.

$ useradd -m -G audio,games,log,lp,optical,power,scanner,storage,video,wheel -s /bin/bash memyselfandi
$ passwd memyselfandi

Configure mkinitcpio with the correct HOOKS required for your initrd image

$ nano /etc/mkinitcpio.conf

Use this HOOKS statement: (I’ve moved keyboard before keymap, encrypt and so on…)

HOOKS="base udev autodetect modconf block keyboard keymap encrypt lvm2 resume filesystems fsck"

Generate your initrd image

mkinitcpio -p linux

## Install and configure Grub-EFI
Since we have the boot partition INSIDE the encrypted disk, we need to add the following option to the Grub options:

Edit the file /etc/default/grub and uncomment the following line:

GRUB_ENABLE_CRYPTODISK=y

And then we can install Grub, which will create an EFI entry named ArchLinux

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ArchLinux

Edit /etc/default/grub so it includes a statement like this:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/nvme0n1p3:cryptlvm resume=/dev/mapper/ArchVG-swap i915.enable_guc=3"

I’ve also added the i915 configuration line.

Other way of doing it is to use UUID:

blkid /dev/nvme0n1p3 -s UUID -o value

And use the UUID outputed on this command line:

GRUB_CMDLINE_LINUX="cryptdevice=UUID=55994-XXXX-xXXXX-XXXXX:cryptlvm resume=/dev/mapper/ArchVG-swap"

Generate Your Final Grub Configuration:

$ grub-mkconfig -o /boot/grub/grub.cfg

At this point there are some errors regarding failing to connect to lvmetad, which are normal and can be ignored.

# Mounting /boot without password request
Grub will ask for passwords to access the encrypted volumes. We can do this automatically:

dd bs=512 count=8 if=/dev/urandom of=/etc/key
chmod 400 /etc/key
cryptsetup luksAddKey /dev/nvme0n1p2 /etc/key
echo "cryptboot /dev/nvme0n1p2 /etc/key luks" >> /etc/crypttab

# Mounting root LVM without password prompt

dd bs=512 count=8 if=/dev/urandom of=/crypto_keyfile.bin
chmod 000 /crypto_keyfile.bin
cryptsetup luksAddKey /dev/nvme0n1p3 /crypto_keyfile.bin
sed -i 's\^FILES=.*\FILES="/crypto_keyfile.bin"\g' /etc/mkinitcpio.conf
mkinitcpio -p linux
chmod 600 /boot/initramfs-linux*

The mkinitcpio.conf FILES line will look like:

FILES="/crypto_keyfile.bin"

# Enable Intel microcode CPU updates (if you use Intel processor, of course)

pacman -S intel-ucode
grub-mkconfig -o /boot/grub/grub.cfg

# Check EFI Boot Manager
Check that the EFI Boot manager has the ArchLinux entry:

$ efibootmgr

For example if ArchLinux entry is Boot0003, check if on the boot order, 0003 is on the head of the list.
If not change the order with:

$ efibootmg -o 0003,0002,0001,0000

Exit Your New Arch System

$ exit

Unmount all partitions

$ umount -R /mnt
$ swapoff -a

Reboot and Enjoy Your Encrypted Arch Linux System!

reboot

___

# Setup system

We need again to connect to the internet, so run again the *wifi-menu*.

Install bash completion for reduced typing effort and other packages if necessary:

$ pacman -S sudo bash-completion base-devel git

To be able to use sudo from your normal user add wheel to sudoers.

$ EDITOR=nano visudo

Uncomment the line

%wheel      ALL=(ALL) ALL

From this point on, it really depends of what need there is for the machine.

# Making the webcam to work.
The webcam id appears at the lsusb output:

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 8087:0a2a Intel Corp. 
Bus 001 Device 002: ID 04ca:7090 Lite-On Technology Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The webcam is the Bus 001:002 device: ID 04ca:7090.
Add the following rule at /etc/udev/rules.d

KERNEL=="video[0-9]*", SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{idVendor}=="04ca", ATTRS{idProduct}=="7090", SYMLINK+="video-cam"

Load the module to activate the webcam:

modprobe uvcvideo

The /dev/video0 and 1 devices should appear.

Advertisements

Synology Let’s Encrypt Certificate manual renew

My internet access router has port 80, the HTTP port, blocked, which only allows to access to DSM applications and hosted sites only through HTTPS.

Unfortunately this HTTP port blocking has the side effect, of also blocking the automatic certificate renewal for the Let’s encrypt certificates…

This should be clearer, since to create a Let’s Encrypt Certificate on DSM Console I had to open port 80 and add a forward rule on the internet access router, rule which I disabled after the certification creation.

Anyway, through the DSM interface, there is no way to manually renew the Let’s Encript certificate, except through the creation of a Certificate Request for a manual renewal.

Searching the forums, on this post: Synology Forum lays the solution:

– First open up Port 80 on the Router and forward it to the port 80 of the Synology IP address.
– Access to Synology command line prompt by using ssh.
– Execute the following command: /usr/syno/sbin/syno-letsencrypt renew-all
– Wait for around 3 to 4 minutes for the command completion
– Check now on the DSM console that all certificates where renewed.
– Block again port 80 on the internet access router.
– Done.

As a final note, there isn’t, as far as I’m aware, a command line option to renew only a specific certificate, so when renewing, all certificates are renewed.

Skype for Linux – Using corporate proxy

Since, unfortunately, I some times need to use Skype on Linux and because I’m behind a corporate proxy server, Skype doesn’t work or accept, at least on Arch Linux running KDE Plasma desktop, the system proxy settings.

Setting the http_proxy/https_proxy variables have no effect on the Electron (I think…) Skype based app.

Anyway the solution for this is quite simple: Just install ProxyChains.

[pcortex@desktop:~]$ sudo pacman -S proxychains-ng

After installing, edit the file /etc/proxychains.conf and under the [ProxyList] add your proxy:

http  10.1.1.1  3128  my_proxy_username  my_proxy_password

Save the file, and now just run skype with the following command:

[fpcortex@desktop:~]$ proxychains skypeforlinux

Making now the test call works.

Node-Red: Checking network service port status + UI status indicator

This post is about how to do two simple things using Node-Red:

  1. Check if network service on the machine running Node-Red is available by checking the corresponding listening port.
  2. The Node-Red UI doesn’t have a status indicator available, so I’ve built one

The only limitation on the following solution is that it only tests for ports for services that are running on the same server, where Node-Red is also running.

Preparation:

We need to install the Is Port Available NPM Module and make it available into our Node-Red instance.

For doing so, in Linux we must do the following:

root@server:~# cd .node-red/
root@server:~/.node-red# node i --save is-port-available

We need now to make this node module available to Node-Red by editing the settings.js file:

root@server:~/.node-red# vi settings.js

Add the module to the global context on the function named functionGlobalContext:

    functionGlobalContext: {
        // os:require('os'),
        // octalbonescript:require('octalbonescript'),
        // jfive:require("johnny-five"),
        // j5board:require("johnny-five").Board({repl:false})

        portavail:require('is-port-available')
    },

You might have other modules configured, so we need to add the above portavail:require(‘is-port-available’) line to that list preceded by a comma.

We need to restart Node-Red to make the module available to the flows.

The testing flow
In our Function nodes, we can now use the global context object portavail to access the is-port-available module.

For example for testing the InfluxDB server port (1086/TCP) we can write the following function:

    // Instantiate locally on the flow the is-port-available module
    const isPortAvailable = context.global.portavail;

    msg.payload = {};   // Zero out the message. Not really necessary
     
    var port = 1086; // Replace this with your service port number. In this case 1086 is the Influx DB port
    
    isPortAvailable(port).then( status => {
        if(status) {
            //console.log('Port ' + port + ' IS available!');
            msg.payload = {'InfluxDB':false,"title":"InfluxDB","color":"red"};   // The port is available, hence the server is NOT running
            node.send(msg);
        } else {
            //console.log('Port ' + port + ' IS NOT available!');
            //console.log('Reason : ' + isPortAvailable.lastError);
            msg.payload = {'InfluxDB':true,"title":"InfluxDB","color":"green"};    // The port is not available, so the server MIGHT be running
            node.send(msg);
           
        }
    });

    // Note that we DO NOT return a message here since the above code is asynchronous and it will emit the message in the future. 

Since the test is using promises, Node-Red will continue executing without waiting for the test response (the isPortAvailable(port) code ). So we do not send any message further on the normal Node-Red execution flow (hence there is no return msg; object) and the message is only emitted when the promise fulfils. When that happens we just send the message with the node.send(msg) statement.

The message payload can be anything, being the only important properties the title and color that are used for creating the UI status indicator.

The status indicator is a simple Angularjs template that displays the title and a status circle with the chosen colour.

Since pasting CSS and HTML code in WordPress is recipe to disaster, the template code can be accessed on this gist or on the complete test flow below:

[{"id":"1f506795.4be25","type":"inject","z":"53f8b852.885c6","name":"Check todos os 60s","topic":"","payload":"","payloadType":"date","repeat":"60","crontab":"","once":true,"x":260,"y":96,"wires":[["5d180fc7.9ad06","27e67f9b.4f9158"]]},{"id":"5d180fc7.9ad06","type":"function","z":"53f8b852.885c6","name":"Test Influx DB","func":"    const isPortAvailable = context.global.portavail;\n    msg.payload = {};\n     \n    var port = 8086;\n    \n    isPortAvailable(port).then( status =>{\n        if(status) {\n            //console.log('Port ' + port + ' IS available!');\n            msg.payload = {'InfluxDB':false,\"title\":\"InfluxDB\",\"color\":\"red\"};   // The port is available, hence the server is NOT running\n            node.send(msg);\n        } else {\n            //console.log('Port ' + port + ' IS NOT available!');\n            //console.log('Reason : ' + isPortAvailable.lastError);\n            msg.payload = {'InfluxDB':true,\"title\":\"InfluxDB\",\"color\":\"green\"};    // The port is not available, so the server MIGHT be running\n            node.send(msg);\n           \n        }\n    });\n    ","outputs":1,"noerr":0,"x":533.5,"y":97,"wires":[["3f3f8226.c9bfb6"]]},{"id":"3f3f8226.c9bfb6","type":"ui_template","z":"53f8b852.885c6","group":"44e5d7ea.043b2","name":"Status Icon","order":0,"width":0,"height":0,"format":"\n.dot {\n    height: 25px;\n    width: 25px;\n    background-color: #bbb;\n    border-radius: 50%;\n    display: inline-block;\n    float: right;\n}\n\n\n
{{msg.payload.title}}\n \n
","storeOutMessages":true,"fwdInMessages":true,"x":780,"y":96,"wires":[[]]},{"id":"27e67f9b.4f9158","type":"function","z":"53f8b852.885c6","name":"Test MongoDB","func":" const isPortAvailable = context.global.portavail;\n msg.payload = {};\n \n var port = 27017;\n \n isPortAvailable(port).then( status =>{\n if(status) {\n //console.log('Port ' + port + ' IS available!');\n msg.payload = {'MongoDB':false,\"title\":\"MongoDB\",\"color\":\"red\"}; // The port is available, hence the server is NOT running\n node.send(msg);\n } else {\n //console.log('Port ' + port + ' IS NOT available!');\n //console.log('Reason : ' + isPortAvailable.lastError);\n msg.payload = {'MongoDB':true,\"title\":\"MongoDB\",\"color\":\"green\"}; // The port is not available, so the server MIGHT be running\n node.send(msg);\n \n }\n });\n ","outputs":1,"noerr":0,"x":533,"y":158,"wires":[["2e85d9d.cc25126"]]},{"id":"2e85d9d.cc25126","type":"ui_template","z":"53f8b852.885c6","group":"44e5d7ea.043b2","name":"Status Icon","order":0,"width":0,"height":0,"format":"\n.dot {\n height: 25px;\n width: 25px;\n background-color: #bbb;\n border-radius: 50%;\n display: inline-block;\n float: right;\n}\n\n\n
{{msg.payload.title}}\n \n
","storeOutMessages":true,"fwdInMessages":true,"x":781,"y":161,"wires":[[]]},{"id":"44e5d7ea.043b2","type":"ui_group","z":"","name":"System Status","tab":"7011ff77.15cb18","disp":true,"width":"6"},{"id":"7011ff77.15cb18","type":"ui_tab","z":"","name":"Home","icon":"dashboard"}]

The result:

The above flow and Node UI status indicator template should produce the following result:

NR UI Status Indicator
Node-Red UI Status Indicator

Using Node-Red and Grafana WorldMap for geolocalized data visualization

Based on my previous posts we are now able to build a system that can receive, store and visualize data by using Node-Red, InfluxDB and Graphana. Grafana allows us build dashboards, query and visualize the stored data across time efficiently by using, in our case, the InfluxDB database engine. So far we’ve used simple line/bar charts to visualize data but we can use both Node-Red and Grafana to plot data onto a map:

  1. NodeRed Contrib World Map: Openstreet UI based map for plotting data with several options, including icon types, vectors, circles and heatmaps totally controlled through nodered flows.
  2. Grafana WorldMap plugin: Grafana panel with also an OpenStreet map for visualizing data.

Both have pros and cons, but the main differences between the two is that Node-Red Worldmap is suited more to real time display, and the Grafana plugin is better adapted to display data based on some time based query. Other major difference is that Node-Red Worldmap would require some coding, but, at least I consider it, at an easy level, and the Grafana plugin is much harder to make it work.

Mapping data using Node-Red Worldmap:
One of the easiest ways for mapping data in real time is using Node Red Worldmap node. The map is plotted and updated in real time.

cd ~
cd .node-red
npm install node-red-contrib-web-worldmap

After restarting and deploying a worldmap node, the map should be available at: http://server:1880/worldmap or other URL depending on the Node-Red base configuration.

One thing to keep in mind is that Node-Red is single user, so all instances of world maps (several different clients/browsers) will always have the same view.
The simplest way to start using the worldmap is just to copy and deploy the demo workflow provided by the node, but the key concept is that each point has a name and a set of coordinates.

msg.payload = {};
msg.payload.name = "CentralLX";
msg.payload.lat = 38.7223;
msg.payload.lon = -9.1393;
msg.payload.layer = "SensorData";
msg.payload.UVLevel = getUVLevel() ;
msg.payload.Temperature = getTemp();

The cool thing is that if we inject repeatedly the above message (keeping the same name) but with different coordinates, the data point will move across the map in real time, and as I said earlier, the move will be reflect onto every client.

So all we need is an Inject node to a function node with the above code and feed it to the world map:

At the end we get this at the URL http://server:1880/worldmap:

Mapping data using Grafana Worldmap plugin:

The Grafana Worldmap plugin can get the location data in several ways. One of them is to use geohash data that is associated to the values/measurements.
There is a Node Red Geohash node that generates the geohash value from the latitude and longitude of data location. As usual we install the node:

cd ~
cd .node-red
npm install node-red-node-geohash

and then the Grafana plugin. We just follow the plugin instructions:

cd ~
grafana-cli plugins install grafana-worldmap-panel
/etc/init.d/grafana-server restart

With this per-requisites installed we can now feed data onto the database, in our case InfluxDB, that will be used by Grafana. We just need make sure that we add the geohash field. The geohash node will calculate from the node-red message properties lat – latitude and lon – longitude the required info:

A simple example:

Using the Influx tool, we can query our database and see that the geohash localization is now set:

> select * from SensorData limit 2
name: DemoValue
time                Temp UVLevel geohash    lat       lon        
----                ---- ------- -------   ---------- ----------  
1490706639630929463 22   8       eyckpjywh 38.7045055 -9.1754669  
1490706651606553077 21   7       eyckpjzjr 38.7044008 -9.1746488 

Anyway for setting up the World map plugin to display the above data was not straight forward, so the following instructions are more for a startup point rather than a solution.

The first thing to know is that the plugin is waiting for two fields: geohash and metric. With this in mind, before wasting too much time with the map plugin, a table panel that is filled with the required query is a precious tool to debug the query:

After we infer from the table that the data is more or less the data we want, we just transfer the query to WorldMap plugin:

Notice two important things: The aliasing for the query field to metric with the alias(metric) instruction, and the Format as: Table.

We can now setup the specific Worldmap settings:

On the Map Visual Options , I’ve centered the map in my location and set the zoom level. Fiddling around here can be seen in real time.

On the Map Data Options for this specific example, the Location Data comes from a table filled with the previous query (hence the format as table on the query output), and we want to see the current values with no aggregation.

When hoovering around a spot plotted on the map we can see a label: value, and the label used is obtained from a table field. In my case I just used geohash (not really useful…). Anyway these changes only work after saving and reloading the panel with F5 in my experience.

At the end we have now graphed data and localized data:

If we drag the selector on the left graphic panel, or select another time interval on top right menu of the grafana dashboard, the visualized information on the map changes.

Setting up a Grafana Dashboard using Node-Red and InfluxDB – Part 1: Installing

A more or less standard software stack used for control, processing and displaying data, has emerged that is almost used by everyone when hacking around on Arduinos, ESP8266, Raspeberry Pi’s and other plethora of devices. This “standard” software stack basically always includes the MQTT protocol, some sort of Web based services, Node-Red and several different cloud based services like Thingspeak, PubNub and so on. For displaying data locally, solutions like Freeboard and Node-Red UI are a great resources, but they only shows current data/status, and has no easy way to see historical data.

So on this post I’ll document a software stack based on Node-Red, InfluxDB and Graphana that I use to store and display data from sensors that I have around while keeping and be able to display historical memory of data. The key asset here is the specialized time-series database InfluxDB that keeps data stored and allows fast retrieval based on time-stamps: 5 minutes ago, the last 7 days, and so on. InfluxDB is not the only Time-Series database that is available, but it integrates directly with Grafana the software that allows the building of dashboards based on stored data.

I’m running an older version of InfluxDB on my ARM based Odroid server, since a long time ago, ARM based builds of InfluxDB and Grafana where not available. This is now not the case, but InfluxDB and Grafana have ARM based builds so we can use them on Raspberry PI and Odroid ARM based boards.

So let’s start:

Setting up Node-Red with InfluxDB
I’ll not detail the Node-Red installation itself since it is already documented thoroughly everywhere. To install the supporting nodes for InfluxDB we need to install the package node-red-contrib-influxdb

cd ~/.node-red
npm install  node-red-contrib-influxdb

We should now restart Node-red to assume/detect the new nodes.

Node Red InfluxDB nodes

Installing InfluxDB
We can go to the InfluxDB downloads page and follow the installation instructions for our platform. In my case I need the ARM build to be used on Odroid.

cd ~
wget https://dl.influxdata.com/influxdb/releases/influxdb-1.2.0_linux_armhf.tar.gz
tar xvzf influxdb-1.2.0_linux_armhf.tar.gz

The InfluxDB engine is now decompressed in the newly created directory influxdb-1.2.0-1. Inside this directory there are the directories that should be copied to the system directories /etc, /usr and /var:

sudo -s
cd /home/odroid/influxdb-1.2.0-1

Copy the files to the right location. I’ve added the -i switch just to make sure that I don’t overwrite nothing.

root@odroid:~/influxdb-1.2.0-1# cp -ir etc/ /etc
root@odroid:~/influxdb-1.2.0-1# cp -ir usr/* /usr
root@odroid:~/influxdb-1.2.0-1# cp -ir var/* /var

We need now to create the influxdb user and group:

root@odroid:~/influxdb-1.2.0-1# groupadd influxdb
root@odroid:~/influxdb-1.2.0-1# useradd -M -s /bin/false -d /var/lib/influxdb -G influxdb influxdb

We need now to change permissions on /var/lib/influxdb:

cd /var/lib
chown influxdb:influxdb influxdb

We can now set up the automatic start up script. On the directory /usr/lib/influxdb/scripts there are scripts for the systemctl based Linux versions and init.d based versions that is my case. So all I have to do is to copy the init.sh script from that directory to the /etc/init.d and link it to my run level:

root@odroid:~# cd /etc/init.d
root@odroid:/etc/init.d# cp /usr/lib/influxdb/scripts/init.sh influxdb
root@odroid:/etc/init.d# runlevel
 N 2
root@odroid:/etc/init.d# cd /etc/rc2.d
root@odroid:/etc/init.d# ln -s /etc/init.d/influxdb S90influxdb

And that’s it. We can now start the database with the command /etc/init.d/influxdb start

root@odroid:~# /etc/init.d/influxdb start
Starting influxdb...
influxdb process was started [ OK ]

We can see the influxdb logs at /var/log/influxdb and start using it through the command line client influx:

root@odroid:~# influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> show databases
name: databases
name
----
_internal

> 

Installing Grafana
We need now to download Grafana. In my case for Odroid since it is an ARMv7 based processor, no release/binary is available.
But a ARM builds are available on this GitHub Repository: https://github.com/fg2it/grafana-on-raspberry for both the Raspberry Pi and other ARM based computer boards, but only for Debian/Ubuntu based OS’s. Just click on download button on the description for the ARMv7 based build and at the end of the next page a download link should be available:

odroid@odroid:~$ wget https://bintray.com/fg2it/deb/download_file?file_path=main%2Fg%2Fgrafana_4.1.2-1487023783_armhf.deb -O grafana.deb

And install:

root@odroid:~# dpkg -i grafana.deb
Selecting previously unselected package grafana.
(Reading database ... 164576 files and directories currently installed.)
Preparing to unpack grafana.deb ...
Unpacking grafana (4.1.2-1487023783) ...
Setting up grafana (4.1.2-1487023783) ...
Installing new version of config file /etc/default/grafana-server ...
Installing new version of config file /etc/grafana/grafana.ini ...
Installing new version of config file /etc/grafana/ldap.toml ...
Installing new version of config file /etc/init.d/grafana-server ...
Installing new version of config file /usr/lib/systemd/system/grafana-server.service ...

Set the automatic startup at boot:

root@odroid:~# ln -s /etc/init.d/grafana-server /etc/rc2.d/S91grafana-server

And we can now start the server:

root@odroid:~# /etc/init.d/grafana-server start
 * Starting Grafana Server    [ OK ] 
root@odroid:~# 

We can now access the server at the address: http://server:3000/ where server is the IP or DNS name of our ODroid or RPi.

Conclusion:
This ends the installation part for the base software.

The following steps are:

  • Create the Influx databases –
  • Receive data from sensors/devices and store it on the previously created database
  • Configure and create Grafana data sources and dashboards
  • Add some plugins to Grafana

Setting up an UPS for Synology NAS, Odroid and Arch Linux

To protect data residing on my Synology NAS I’ve bought and installed an UPS, an APC 700U to be exact. The trigger for buying and installing one was the loss of (some) disk data event that happened to family member external hard disk due to power loss. The data recovery cost was higher than buying an UPS and of course the lack of backups added to outcome of that event.

While no Synology NAS was involved on the above data loss, I know first hand that backups by themselves only add one layer of protection to possible data loss, and since from time to time I also have some power loss events, it was just a matter of time that my NAS might be hit by an unrecoverable power event, and, who knows, data loss.

So buying an UPS just might be a good idea…

Anyway the UPS from APC that I bought has an USB port allowing it to be connected to the Synology, which allows the UPS to be monitored and also allows the NAS to gracefully shutdown before UPS battery exhaustion. As a bonus it also allows to run an UPS monitoring server where other devices that share the same UPS power source can be notified of a power event through the network. Just keep in mind that the network switch or router must also be power protected…

Installing the UPS:
Installing the UPS is as simple as power down all devices that will connect to the UPS: Synology NAS, Odroid, external hard disks, PC base unit and network switch, and connecting an USB cable from the UPS to the back Diskstation USB ports.

After starting up, just go to DSM Control Panel and select Hardware & Power and then the UPS tab. Enable the UPS by ticking the Enable UPS support and also enable the UPS server to allow remote clients by ticking Enable UPS Network Server:

UPS Configuration

We can see by pressing the Device Information button that the UPS was correctly detected:

UPS Device Information

To end the configuration we need to press the Permitted DiskStation Devices and add the IP’s address of the devices that also have their power sources connected to the UPS and will also monitor the power status, in my case the IP of Odroid and my home PC.

And that’s it.

Setting up Arch Linux
Interestingly I dind’t found the NUT tools (Network UPS tools) on the core Arch repositories, but they are available at the AUR repository:

 yaourt -S network-ups-tools nut-monitor

The above packages will install the core NUT tools and a graphical monitor.

We can now scan our network for the ups:

nut-scanner -s 192.168.1.16
Scanning USB bus.
Scanning SNMP bus.
Scanning XML/HTTP bus.
Scanning NUT bus (old connect method).
[nutdev1]
        driver = "nutclient"
        port = "ups@192.168.1.16"

With the UPS reference found, we can now query it:

 upsc ups@192.168.1.16
 
Init SSL without certificate database
battery.charge: 100
....
....
battery.voltage: 13.9
battery.voltage.nominal: 12.0
device.mfr: American Power Conversion
device.model: Back-UPS XS 700U  
....

We can check the load, for example, with:

upsc  ups@192.168.1.16 | grep load
Init SSL without certificate database
ups.load: 37

We need now to modify the following files on /etc/ups:

  1. nut.conf
  2. upsmon.conf

First, as root, we copy a file from upsmon.conf.sample to upsmon.conf and add the following line:

MONITOR ups@192.168.1.16 1 * * slave

after the other commented out MONITOR lines. Since I’m only monitory, I’ve just put * at the username and password for authentication.

On nut.conf, we change the line MODE to MODE=netclient

After changing the files, we enable and start the UPS monitoring service:

sudo systemctl enable nut-monitor
sudo systemctl start nut-monitor

We are now monitoring the UPS status through the network. Keep in mind that the hub/switch power should also be connected to the UPS.

For monitoring we can use the nut-monitor application to see the UPS status in a nicer way:

NUT Monitor

To make the application easier to use, we can create a profile and save it, and when calling the application nut-monitor, we pass the profile name with the -F switch.

Setting up Odroid
To allow Odroid to monitor the UPS status through the Synology UPS server we need to also install the nut UPS tools (the same used by Arch Linux and DSM):

 apt-get install nut

The configuration steps for Odroid are the same as the Arch Linux steps, but since Odroid is running an Ubuntu variant, the files are located on a different path: /etc/nut.

To start the monitoring with the new configuration we just do /etc/init.d/ups-monitor restart.

Authentication
If authentication is needed, on the Synology disk, check the NUT configuration files located at /usr/syno/etc/ups/.

The upsd.users file has the user and password defined by default by the NUT tools on DSM.