Using Netbeans, OpenOCD and GDBServer plugin for ARM development

On my previous post we’ve seen how to setup the base software for starting up programming for ARM processor based boards.

Normally these boards have no base code running, like Arduino for example, to allow direct programming through USB, and so we need to have a programmer to flash the code onto the board through a specific interface, either JTAG or SWD. As a bonus the programmer also allows to debug in real time the code on the processor with breakpoints and watchpoints without resourcing to the common printf or Serial.print…

These programmers come on all sizes, shapes and prices… Some of the available are:

  1. STLink/V2 – Standard programmer from 2.5€ and up for clones, 22€ for the original.
  2. BlackMagic probe – An alternative that can be flashed on some hardware to build programmers, and that supports some devices based on the ARM processors, for example the BLE NRF51822 chip.
  3. Segger J-Link – State of the art programmer. A cheaper EDU version is available but with some licensing restrictions.

STLink/V2 is supported by OpenOCD which allows to use openocd to flash and debug code through JTAG or SWD (Single Wire Debug).

Code Sample
The “Hello World” example on the hardware world is the Blink Led example. On this site there is an example for my processor, in my case the STM32F103 processor. Just download and expand the file and import the project into NetBeans:

New Project with existing sources

Just make sure that the correct toolchain is selected:

Project with ARM Toolchain

The build process should run without errors, and at the root of the project a new file should be created: main.elf

Flashing the code
For using OpenOCD to flash the code onto the ARM processor based board we need to configure openocd to know which programmer is using and which target board is programming.
In my case since I’m using a STLink/V2 programmer connected to a target STM32F103 based board, I’ve created the following configuration file:

ebay_board.cfg

set CHIPNAME STM32F103C8T6

source [find interface/stlink-v2.cfg]

transport select hla_swd

set WORKAREASIZE 0x2000
source [find target/stm32f1x.cfg]

Based on this configuration we can flash now our board:

openocd -f /opt/ARM/ebay_board.cfg -c init -c targets -c "halt" -c "flash write_image erase /opt/ARM/Projects/STM32F103VHB6_RevZ_Demo1/main.elf" -c "verify_image /opt/ARM/Projects/STM32F103VHB6_RevZ_Demo1/main.elf" -c "reset run" -c shutdown

And the output is:

Open On-Chip Debugger 0.9.0 (2016-04-27-23:18)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v24 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.208372
Info : STM32F103C8T6.cpu: hardware has 6 breakpoints, 4 watchpoints
    TargetName         Type       Endian TapName            State       
--  ------------------ ---------- ------ ------------------ ------------
 0* STM32F103C8T6.cpu  hla_target little STM32F103C8T6.cpu  halted
auto erase enabled
Info : device id = 0x20036410
Info : flash size = 64kbytes
target state: halted
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000003a msp: 0x20004fd0
wrote 7168 bytes from file /opt/ARM/Projects/STM32F103VHB6_RevZ_Demo1/main.elf in 0.468006s (14.957 KiB/s)
target state: halted
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x20004fd0
verified 6836 bytes in 0.035336s (188.923 KiB/s)
shutdown command invoked

Success! A blinking led (we have to connect one on my board to the correct pin) is now blinking.

Debugging the code:
To be able to debug our code we need to start OpenOCD and connect it to the board:

openocd -f ebay_board.cfg 
Open On-Chip Debugger 0.9.0 (2016-04-27-23:18)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v24 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.211511
Info : STM32F103C8T6.cpu: hardware has 6 breakpoints, 4 watchpoints

And openocd is running.

On the NetBeans size we need to goto Debug->Attach Debugger. A new window should appear and we must change the debugger type from the default GDB Debugger to gdbserver and configure the openocd remote port and project:

gdbserver configuration

Make sure that the target is defined as: extended-remote localhost:3333
and the correct project is selected.

And that’s it. If the code already has some breakpoint defined, and the code passes through it, the execution is stopped and the correct code line is shown. Otherwise we can press the Pause button:

Debugger control

On the openocd output we can see the breakpoints and code pause working:

Info : device id = 0x20036410
Info : flash size = 64kbytes
Info : halted: PC: 0x08000806

Hardware breakpoint

If there are weird errors regarding connection dropped or failed, make sure that under the toolchain for ARM, the correct debugger is selected, meaning it should be the ARM debugger and not the gdb debugger.

And that’s it.

Setting up Netbeans for ARM development

My quick notes for setting up Netbeans, OpenOCD for ARM cortex processor development on Arch Linux. The instructions, excluding the ARCH Linux specific pacman commands, should be the same for any Linux platform.

So I’ve bought some STM32F103 ARM Cortex based boards, and for starting building software for them these are the steps:

Arm toolchain

Download the ARM toolchain from ARM GNU Toolchain. In my case I downloaded the latest available version for Linux 64 bit.

Create a working directory, in my case I just created /opt/ARM and unzip the toolchain there.

cd /opt
mkdir ARM
cd ARM
tar xvf ~/Downloads/gcc-arm-none-eabi-6_2-2016q4-20161216-linux.tar.bz2

Add now the ARM toolchain to your path, by editing the .bashrc file at our home directory:

cd
vi .bashrc

Add at the end the following line:

export PATH=$PATH:/opt/ARM/gcc-arm-none-eabi-6_2-2016q4/bin

and then execute the following command for assuming the new setting, on the current terminal window:

source ~/.bashrc

To the new path to be globally available we have to logout and login again, but we won’t do that right now.

We can now test the ARM toolchain installation by calling, for example the command: arm-none-eabi-gcc -v

arm-none-eabi-gcc -v
Using built-in specs.
COLLECT_GCC=arm-none-eabi-gcc
COLLECT_LTO_WRAPPER=/opt/ARM/gcc-arm-none-eabi-6_2-2016q4/bin/../lib/gcc/arm-none-eabi/6.2.1/lto-wrapper
Target: arm-none-eabi
Configured with: /tmp/jenkins-GCC-6-build...
...
...

Success!

OpenOCD
OpenOCD is a tool that will allow to flash the ARM processors and also allow to debug code. Netbeans by itself won’t be able to flash code on the processor. So regarding OpenOCD we need to do the installation and some configurations first:

On ARCH Linux it goes more or less like this:

sudo -s
pacman -S openocd
cp /usr/share/openocd/contrib/99-openocd.rules /etc/udev/rules.d

groupadd plugdev
usermod -a -G plugdev pcortex

udevadm control --reload-rules

Replace pcortex on the usermod command with your user name. Now we can logout and logon again to assume the new user group and path.

Setting up Netbeans
After starting up NetBeans, I’m using the latest version 8.2 (at the date of this post), we select Tools->Plugins and try to search and install the gdbserver plugin. If it fails searching for it, just download it from the GDBServer Plugin home page and install it manually.

Then we need to add the ARM toolchain to the available C++ Tools Collection. Just go to Tools->Options->C++ and press the Add button:

Add ARM Toolchain

Then add the correct path to the binaries for C/C++ and very important for the ARM debugger:
ARM tools

We don’t need to do nothing for anything else, since Code Completion tab will be filled automatically.

And that’s it, we can now use Netbeans to develop for ARM based boards.

In the next post we will see how to flash and debug code on these ARM based boards.

Upgrading NodeJs and Node Red on Odroid

I run many services on my Odroid C1+ including Node-Red. But since NodeJs on Odroid C1+ is version v0.10 is starting to be seriously old for running Node-Red or other NodeJS dependent software.

So my quick instructions for upgrading NodeJS and Node-Red on the Odroid C1+

Upgrading NodeJS

First verify what version is available/installed on the Odroid:

odroid@odroid:~$ node -v
v0.12.14
odroid@odroid:~$ nodejs -v
v0.10.25

Since I’ve already had previously installed a more recent version of NodeJS (the node command), the version used by Node-Red is v0.12.14 while the default NodeJS version is v0.10.25.

We can also, and should, check the npm version:

odroid@odroid:~$ npm -v
2.15.1

We also need to find what architecture we are using, just for completeness since ODroid C1+ is an ARM7 based architecture:

odroid@odroid:~$ uname -a
Linux odroid 3.10.96-151 #1 SMP PREEMPT Wed Jun 15 18:47:37 BRT 2016 armv7l armv7l armv7l GNU/Linux

This will allow us to download the correct version of the NodeJS binaries from the NodeJS site: NodeJS downloads.
In our case we choose the ARM7 architecture binaries, which at the current time is file: node-v6.9.2-linux-armv7l.tar.xz
So I’ve just copied the link from the NodeJS site and did a wget on the Odroid:

wget https://nodejs.org/dist/v6.9.2/node-v6.9.2-linux-armv7l.tar.xz

I then created a working directory and “untared” the file:

odroid@odroid:~$ mkdir nodework
odroid@odroid:~$ cd nodework
odroid@odroid:~/nodework$ tar xvf ../node-v6.9.2-linux-armv7l.tar.xz
odroid@odroid:~/nodework$ cd node-v6.9.2-linux-armv7l/
odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l$ 

Since there isn’t an install script we need to move the new NodeJS files to the correct locations:

  1. Binaries to /usr/bin
  2. Include files to /usr/include
  3. Libs files to /usr/lib

Copy the binaries, replacing, if existing the older versions:

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/bin
$ sudo cp -i node /usr/bin
cp: overwrite ‘/usr/bin/node’? y
odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/bin$ 

Copy the include files:

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/include
$ sudo cp -R node  /usr/include/

and copy the libraries

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/lib
$ sudo cp -R node_modules /usr/lib

and finally:

odroid@odroid:~/nodework/node-v6.9.2-linux-armv7l/share
$ sudo cp -R . /usr/share

We need now to make npm to point to the correct nodejs script so, we need to delete the npm link at the /usr/bin and /usr/local/bin directories:

odroid@odroid:~$ sudo rm /usr/bin/npm
odroid@odroid:~$ sudo rm /usr/local/bin/npm

and re-create the correct links:

odroid@odroid:~$ sudo ln -s /usr/lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm
odroid@odroid:~$ sudo ln -s /usr/lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm

Running now the node and npm commands should report the latest versions:

odroid@odroid:~$ node -v
v6.9.2
odroid@odroid:~$ npm -v
3.10.9
odroid@odroid:~$ 

Success!

Upgrading Node-Red

From the Node-Red startup log, we can see the previous versions of node-red and nodejs used:

Welcome to Node-RED
===================

28 Dec 17:55:40 - [info] Node-RED version: v0.15.2
28 Dec 17:55:40 - [info] Node.js  version: v0.12.14
28 Dec 17:55:40 - [info] Linux 3.10.96-151 arm LE
28 Dec 17:55:42 - [info] Loading palette nodes
28 Dec 17:55:50 - [info] Dashboard version 2.1.0 started at /ui
28 Dec 17:55:54 - [warn] ------------------------------------------------------
28 Dec 17:55:54 - [warn] [rpi-gpio] Info : Ignoring Raspberry Pi specific node
28 Dec 17:55:54 - [warn] ------------------------------------------------------

we can upgrade now Node-Red according to the Node Red upgrading instructions:

odroid@odroid:~$ sudo npm cache clean
odroid@odroid:~$ sudo npm install -g --unsafe-perm node-red

and after a while the upgrade should be done.

Before starting up node-red I went to the node-red module directories, and did an update:

odroid@odroid:~/.node-red$ npm update
/home/odroid/.node-red
└── crypto-js@3.1.8 

Starting up Node-Red should show now the new software versions:

Welcome to Node-RED
===================

1 Jan 20:35:46 - [info] Node-RED version: v0.15.2
1 Jan 20:35:46 - [info] Node.js  version: v6.9.2
1 Jan 20:35:46 - [info] Linux 3.10.96-151 arm LE
1 Jan 20:35:47 - [info] Loading palette nodes
1 Jan 20:35:54 - [info] Dashboard version 2.2.1 started at /ui

Done!

Some thoughts on Lorawan

It cropped up on my news feed here on WordPress an article regarding three major flaws on the Lorawan protocol. Since I’m following the Lorawan topic I was interested to see what those “Major” flaws where, and found it rather interesting, that the text, also derived from a commercial vendor, looked like the conundrum story like if a glass is half full or half empty.

First there is an huge misconception regarding low power communications protocols (LPWAN), being either Lorawan, Sigfox or anything else. The key is power, and lower power usage, the less the better. Asking for LPWAN protocols to do the same that other high energy, higher bandwidth protocols can do, is mixing apples with oranges.

Anyway the “flaws”, if we can call then that, are as follow:

– All uplink messages are unacknowledged.
This not true. Lorawan supports three classes of devices. Class A, the less power hungry of them, opens two receive windows for downlink messages from the server, that can be used for acknowledging the uplink message. If it makes sense to have acknowledge, it depends on the business case… If it is required probably LoraWan neither Sigfox or other LPWAN protocols are adequate to be used…

Also any protocols that use the ISM radio bands must obey the defined rules by the government bodies that control the radio spectrum. These rules apply to any protocol on those bands, so it’s not a flaw specific to any protocol. The 1% duty cycle applies to Lorawan and SigFox. In fact Sigfox will enforce the 1% duty cycle by refusing messages that exceed that percentage, and the LoraWan backend provider The Things Network will do the same.

Also there is some confusion regarding the medium access protocol, in this case the radio spectrum. The medium is shared by anyone, so collisions and interference will happen. Sigfox adds some resilience to this this by transmitting each message three times in three different ISM band frequencies, for example. As Lorawan it also doesn’t check for the medium before transmitting but only transmits once, since the physical modulation CSS (Chirp Spread Spectrum) has more resilience to interference. Also due to the availability of what is called Spread Factor, several transmissions can happen at the same time at the same frequency, and successfully be decoded at the other end.

On radio protocols checking for medium occupancy before transmitting only makes sense for non constrained devices, since the process of checking the medium before transmitting will consume power (a lot by having the radio on) and without any sort of guarantee that interference will not start right away or the interferance is happening not at the node side, but on the gateway side. So since one of the engineering requirements for LPWAN is low power, then the exchange between power and medium access control is made, which means ALOHA and let’s transmit!. So now we can have devices that have batteries that last years.

– All gateways in range see all uplink traffic which is not safe
I find this one rather funny since, turning on my radio can catch any available radio broadcasters, or any radio scanner can receive anything. Just check out Web SDR for hours of amusement.

The fact that all gateways see all traffic is a direct consequence of the radio medium, not an issue with the protocol. Any protocol that uses radio has the same “flaw” and it applies to Lorawan, SigFox, UNB, Weightless, 3G, LTE, you name it.

To solve this, encryption is used and at least in Lorawan there are several encryption keys and ways of providing them.
Lorawan can use fixed provided keys (ABP – Activation by personalisation) or variable keys through OTAA (over the air activation).

Anyway the gateways can receive any Lorawan packets, but without at least the 128 bit Network key and 128 bit application key, can’t do anything with the data. Gateways only forward data to network backend servers, and there, if they have the correct keys, decryption can be done and data forward to the correct application servers.

Check out this for more information.

– LoRaWAN requires an enormous amount of bandwidth
Well, yes, it is a SPREAD spectrum technology and it makes part of the holy war between Narrow band supporters vs Wide band supporters. Ones say that UNB is better, others don’t, and so on. Spread spectrum technology exists since a long time ago. Lorawan bandwidth can be 150Khz, 250Khz and 500Khz vs the 200Hz of Sigfox.

While SST can be used and detected below the noise floor level and accepts variations on the frequencies (reflections, Doppler effects), UNB is, on the receiver side way more complicated since it requires very precise crystals for frequency reference and higher power levels on the spectrum.

So in the article that I read it seems that Lorawan SST is just bad, without any consideration of the advantages vs UNB, which by itself is a discussion on different technologies which have advantages and disadvantages each.

Conclusion:
Nothing is perfect in engineering. Trade offs need to be made to achieve the requested requirements, and then based on the initial implementations, improve on it. So Lorawan, as Sigfox, solve the same issues by different means with the associated advantages and disadvantages.

So the above flaws, are just engineering trade offs that can be applied to any protocol.

LPWAN – Starting up with LoraWAN and The Things Network

LPWAN Networks – A simple introduction
Low Power Wide Area Communications (LPWAN) classifies a group of communication protocols featuring low power usage and a high communication range. For Internet of Things communications, where battery powered devices and constrained devices (weak CPU, low RAM/ROM) are the norm, LPWAN use as the communication protocol for IoT makes sense, since it makes possible to have standalone devices with batteries that last years, instead of hours or days, and might be hundreds of meters to Kms away from a base station/gateway.

But LPWAN protocols, in contrast have low communication bandwidth, since from power, range and bandwidth, we can only have two of those, and while this might be a problem for certain applications, IoT devices don’t require high bandwidth and since most have sparse communication requirements, and when they do communicate they don’t need to transmit a lot of data.

Starting up with LoraWan and The Things Network
One of the possible ways of starting to use LPWAN networks in our IoT devices, is to use a LPWAN protocol named LoraWan, which is based on the Lora physical protocol, and if available at our area, the crowd sourced LPWAN network The Things Network as the backend network for receiving data.

Lora uses the 868Mhz ISM radio band and it uses a form of signal transmission called CSS. The ISM radio band can be used freely by everybody, but under certain rules that depends of the country where the devices are located. So to use that band, rules like maximum emission power and duty cycle (usage of the available spectrum) are defined, and in Europe, the maximum power is 20mW and 1% duty cycle per day. Additionally the back end operator can add other restrictions.

Over Lora, the Lorawan protocol is an open standard and source code for implementing it is available and already ported to several devices, including the Arduino, Raspberry PI and the ESP8266, among others.

The software for implementing the LoraWan protocol at the end devices (nodes) is IBM’s LMIC library, available at Github and on Platformio libs:

[ ID  ] Name             Compatibility         "Authors": Description
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[ 852 ] IBM LMIC framework arduino, atmelavr, atmelsam, espressif8266, intel_arc32, microchippic32, nordicnrf51, teensy, timsp430 "IBM, Matthijs Kooijman": Arduino port of the LMIC (LoraWAN-in-C, formerly LoraMAC-in-C) framework provided by IBM. | @1.5.0+arduino-1

Based on this library we can build code and nodes that are able to transmit data through LoraWan to a network backend. Specifically for the Arduino, the Atmega328 is the bare minimum to begin with, since the library, due to use of AES encryption functions occupies a lot of memory.

The backend can be provided by really anyone, including Telco operators, or private and crowd source operators like The Things Network (TTN). TTN provides the backend and management services, but depends on crowd sourced gateways, and might not be available at our area. Still it is possible, for testing, to build our own gateways, our buy them, and connect them to the Things Network. TTN doesn’t require any access fees (yet).

So with LoraWan an the Things Network, we can build our own nodes and gateways since code and hardware can be easily obtained, connect them and use it to build applications.

Regarding LoraWan we can also read this great introduction from Design Spark.

Lora hardware:

Anyway the easiest way for starting up using Lora and Lorawan, is to buy an Dragino Lora Shield and connect it to an Arduino Uno or Mega.

Dragino Lora Shield

Dragino Lora Shield

This is a great shield to startup since doesn’t need any soldering or complex configuration, just plug it on an Arduino shield and use the LMIC library and some code samples. Also the Dragino Shield can be used to build a simple gateway by connecting it to a Raspberry PI or ESP8266 to build what is called a single channel gateway, that allows to test gateway functionality, but it isn’t quite compatible with the LoraWan specifications. Anyway it gets the job done if there is no gateway nearby. Just make sure that you by version 1.3 or 1.4 of the shield. Mine also came with an SMA antenna.

Other alternatives to start using Lorawan are available at eBay/Aliexpress and other renowned name shops, namely Lora radio transceivers at around 8/16€, for example the HopeRF RFM95. Those we can also use them with Arduino or ESP8266 to build our nodes or single channel gateways.

Just make sure that the frequency for these modules and shields must match the allowed radio transmission frequency in your area. In my case, in Europe is 868Mhz, but for example at USA is 900Mhz.

Dragino Lora Shield Jumpers
The Shield has some jumpers and only looking at the schematic and cross referenced them with the RFM95 module (used by the shield as the Lora transceiver) I could see what they where for:

– There are two set of jumpers:

  • One defines the pins for SPI communication: SV# Jumpers;
  • The other set defines which data pins of the RFM95 module are exposed to the shield allowing them to be connected to the Arduino: JP# Jumpers.

The JP# jumpers connect the following if closed:

  • JP1 – RFM95 DI1 – Arduino pin D6
  • JP2 – RFM95 DI2 – Arduino pin D7
  • JP3 – RFM95 DI5 – Arduino pin D8

The RFM95 pin DI0 is permanently connected to Arduino pin D2.
The RFM95 pin RST (Reset) is permanently connected to Arduino pin D9.

Their function for Lora might be the following (The function depends how the RFM95 module is used)

  • RFM95 DI0 – Indicates if data was received and is ready to be read through the SPI bus. Indicates end of transmission for data that was previously sent: RXReady/TXReady
  • RFM95 DI1 – Receive timeout for Lora mode.
  • RFM95 DI2 – Receive timeout for FSK mode.
  • RFM95 DI5 – Used by Semtech Library. LMIC library doesn’t use it.

The SV# jumpers connect:

  • SV2 – SPI Clock line. Default on pin D13, otherwise on Arduino SPI CLK on the ICSP header.
  • SV3 – SPI Data line In (MOSI). Default on pin D11, otherwise on Arduino MOSI pin on the ICSP header.
  • SV4 – SPI Data line Out (MISO). Default on pin D12, otherwise on Arduino MISO pin on the ICSP header.

The SPI Chip Select line is always at pin D10.

So now we know that D10, D9 and D2 are used permanently connected and used by the shield, and the others can be connected or disconnected if needed or not.

LMIC software with Dragino Lora Shield:

To start using the Dragino Lora Shield so it connects to a LPWAN network, we can start using the following example: TTN node by ABP. ABP means Activation by Personalization, which means that all data for joining the network will be explicitly defined on the code. Other alternative would be OTAA: Over the air activation, where the gateway sends the configuration data to the node. Since we don’t know if we have a gateway in range, let’s start first with ABP mode.

The above code uses the LMIC library for implementing the LoraWan stack.
Since LMIC library doesn’t use DI05, we can remove the JP3 jumper, and free this IO line for other things, like another shield.

To use the LMIC library we must define first the pins that are used according to the shield configuration:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, 7}             //DIO pin mapping for DIO0, DIO1 and DIO2  
};

But DIO2 pin is only used for FSK modulation, and so if only using LoraWan we can also open up the JP2 jumper, and our LMIC pin configuration can be as follows:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, LMIC_UNUSED_PIN} //DIO pin mapping for DIO0, DIO1. DIO2 is not used  
};

Connecting to TTN
Connecting to The Things Network (TTN) depends of course of an available TTN gateway at the nodes range. Still we need to configure some parameters to allow the node to connect.

On this example the connection is done through Activation by Personalization. This means that we should put on our code the Network Session key and Application Session key on the code. To obtain these values we need to register on the TTN site, TTN Dashboard and add an ABP device..

From this site, then we can get the three configuration parameters that we need:

  • Device ID
  • Network Session key
  • Application Session key

Note by pressing the icon we can get the values in a form ready to paste it on the code:

/*************************************
 * TODO: Change the following keys
 * NwkSKey: network session key, AppSKey: application session key, and DevAddr: end-device address
 *************************************/
static u1_t NWKSKEY[16] = { 0xAA, 0x0F, 0x29, 0xD3, 0x9D, 0x7A, 0xAE, 0x3B, 0x54, 0xCF, 0xDF, 0x2F, 0x2A, 0x23, 0x55, 0xB5 };
static u1_t APPSKEY[16] = { 0x2F, 0x85, 0x43, 0x5B, 0x34, 0x9C, 0x80, 0xC6, 0xA8, 0xFA, 0x27, 0x49, 0x5A, 0x36, 0x82, 0x21 };
static u4_t DEVADDR = 0x95337738;

And that’s it, when running the sketch, and if in range of a gateway, the sent message should appear at the Dashboard:

Messages received for the device

Messages received for the device

Final thoughts:
In my area there are at least, supposedly, two active TTN gateways and I’m around 2Km’s away from them in a dense urban area.
But when running the sketch the first times, I had no results what so ever.

One of the configurations options for LoraWan is what is called a Spread Factor that, in a simplest way, exchanges range with on-air time for transmission, being SF7 the fastest speed and shorter range and SF12 the slowest speed and higher range. The sketch default value for the sprea factor was SF7, and changing it to SF12:

 ...
    // TTN uses SF9 for its RX2 window.
    LMIC.dn2Dr = DR_SF9;

    // Set data rate and transmit power for uplink (note: txpow seems to be ignored by the library)
    LMIC_setDrTxpow( DR_SF12 ,14);

    // Start job
    do_send(&sendjob);

With SF12 the message started be received and the sequence numbers of the messages received where continuous, so no lost messages.

Dropping to SF11, also worked fine, and the message sequence received and shown on the TTN Dashboard where still continuous.

At SF10, some of the messages where lost, almost 75% of them. In this case, arranging the antenna position (to exactly vertical) and placement did alter the reception successes.

Processing received data
After the data is at the TTN backend there are several ways of getting it. For reference on the TTN site there are instructions in how to access the data.

Crash course on Lora
For further information the next one hour video is a great starting point:

The slides for the presentation are available here:

https://www.dropbox.com/s/87htha4dq5b32wa/ttn-lora-crash-course.pdf?dl=0

MQTT Mosquitto broker – Client Authentication and Client Certificates

After seeing how to set up transport layer security for the Mosquitto MQTT broker by using the Transport layer security on this post, we need to see how to setup client authentication (only authorized clients can connect to the broker) either by using the common user/password based authentication method or using client certificates.

Authentication by user and password:
First let’s enable authentication to the broker by setting up user and password authentication. For enabling this kind of authentication we need to modify the broker configuration (/etc/mosquitto/mosquitto.conf in my case) file and change the following entries:

allow_anonymous false
password_file /etc/mosquitto/passwd_mqtt

Before restarting the MQTT broker we need to add some users to the passwd_mqtt file with the command mosquitto_passwd:

mosquitto_passwd -c passwd_mqtt user1

The -c parameter is for creating the initial password file, if doesn’t exist, otherwise it will overwrite it!. For adding new users or update the passwords just run the command without the -c parameter.

After restarting the broker will should not be able to logon anonymously:

# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d
Client mosqpub/25688-pcortex sending CONNECT
Client mosqpub/25688-pcortex received CONNACK
Connection Refused: not authorised.
Error: The connection was refused.

But providing the user name and password:

# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d -u user1 -P secret
Client mosqpub/25709-pcortex sending CONNECT
Client mosqpub/25709-pcortex received CONNACK
Client mosqpub/25709-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (7 bytes))
Client mosqpub/25709-pcortex sending DISCONNECT 

For subscription, the same has to be done, again by providing an user and password:

# mosquitto_sub -t \$SYS/broker/bytes/\# -v --cafile /etc/mosquitto/certs/ca.crt -p 8883 -u user1 -P secret
$SYS/broker/bytes/received 217
$SYS/broker/bytes/sent 20

Authentication by using client certificates
Using client certificates, signed by a certificate authority, assures the client identity. The certificate authority used must be the same used by the server certificates and is only supported over TLS/SSL.
For using client certificates for authentication, we need to change the listener configuration for TLS/SSL by adding the following directives:

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/hostname.crt
keyfile /etc/mosquitto/certs/hostname.key
require_certificate true
use_identity_as_username true

The require_certificate directive with the value true means that clients must now provide a client certificate to connect.
The use_identity_as_username means that the user name of the connecting user is taken from the CN (Common Name) property of the certificate, otherwise we still need to provide an user and password.

With the above new configuration, we now can’t access the broker with user/password token:

mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d -u user1 -P secret
Client mosqpub/26549-pcortex sending CONNECT
Error: A TLS error occurred.

the result on the log file is:

1478537500: New connection from 127.0.0.1 on port 8883.
1478537500: OpenSSL Error: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
1478537500: Socket error on client , disconnecting.

For accessing now the broker we must provide a client certificate and a private key. The client certificate must be generated from the same CA (Certificate authority) that created the server certificate, otherwise, the client certificate can’t be validated, and the connection fails:

1478537629: New connection from 127.0.0.1 on port 8883.
1478537629: OpenSSL Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
1478537629: Socket error on client , disconnecting.

So on the same directory where the ca.crt and ca.key are residing execute the script to create the client certificate. For example for creating a client certificate for user1, we need to execute the following command:

# ./gen_CA.sh client user1

And we should have three files, two of them the user1.crt, the user certificate, and the user1.key, the user1 private key.

We can now logon to the broker:

# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d  --cert user1.crt --key user1.key 
Client mosqpub/30264-pcortex sending CONNECT
Client mosqpub/30264-pcortex received CONNACK
Client mosqpub/30264-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (7 bytes))
Client mosqpub/30264-pcortex sending DISCONNECT

If we check the log, we have the following:

1478539830: New client connected from 127.0.0.1 as mosqpub/27159-pcortex (c1, k60, u'user1').
1478539830: Client mosqpub/27159-pcortex disconnected.
1478601507: New connection from 127.0.0.1 on port 8883.

We can see that the broker extracted the username from the certificate property CN. If we don’t use the use_identity_as_username as true we need to provide the username and password and the client certificate. In this case, the certificate is validated, and the user used to logon might not be the same as the one defined on the CN certificate property. So without this directive we either need to allow anonymous logon again or define user and passwords.

Additional thoughts:
With user and password authentication we can revoke access to an user, by deleting it from the password file or changing the password.
But what about the authentication based on client certificates? As long the certificate is valid, the user can logon at will, since the broker will always accept it until the end of the certificate validation date.
The solution for this is to revocate the client certificate so when it is used, the broker rejects it. For this functionality, most Certificate Authorities provide the revocation list by providing either by CRL (Certificate Revocation list) file, or by OSCP (Online Status Certificate Protocol), and the server checks the client certificate on this list before allowing access. Mosquitto broker only works with CRL files.

We need to modify the listener configuration to verify the client certificates if it was revocated or not:

# MQTT over TLS/SSL
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/hostname.crt
keyfile /etc/mosquitto/certs/hostname.key
require_certificate true
use_identity_as_username true
crlfile /etc/mosquitto/certs/ca.crl

The issue now is that the script that we are using for generating certificates doesn’t generate CRLs, and so we can’t revocate certificates by using this script.

We can do it by hand, but that wouldn’t make our life much easier, since revocating certificates means that the CA server must now track all certificates and their status.
One possible solution is to use scripts that allow to create (a not very secure) CA: easy-ca. These scripts will replace the genCA.sh script and allow the revocation of certificates.

Getting the easy-CA scripts:

git clone https://github.com/fcgdam/easy-ca.git

I’m omitting a lot of output that these scripts do, namely asking for the Organization name, CA passwords and so on.

Somewhere where we find suitable we create our CA:

Generating the CA:

# cd ~ 
# ~/easy-ca/create-root-ca -d CA
# cd CA

Then at the CA directory we created, we can now create our broker certificate and user certificate. Make sure that the names make sense.

Generating the broker certificate:

# bin/create-ssl -s brokername
# bin/create-client -c user1

At the end we have the following files:

ca/ca.crt

We now have all the needed files for setting up our broker with support for CRL.

# sudo -s
# cp ~/CA/ca/ca.crt /etc/mosquitto/certs
# cp ~/CA/crl/ca.crl /etc/mosquitto/certs
# cp ~/CA/certs/brokername.server.crt /etc/mosquitto/certs/brokername.crt
# cp ~/CA/private/brokername.server.key /etc/mosquitto/certs/brokername.key

And restarting the broker we have the new certificates in place.
We can test now by using our CA user certificate:

# cd ~/CA
# mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d  --cert certs/user1.client.crt --key private/user1.client.key 
Client mosqpub/7424-pcortex sending CONNECT
Client mosqpub/7424-pcortex received CONNACK
Client mosqpub/7424-pcortex sending PUBLISH (d0, q0, r0, m1, 'test', ... (7 bytes))
Client mosqpub/7424-pcortex sending DISCONNECT

It works. Now let’s revoke the user certificate:

# bin/revoke-cert -c certs/user1.client.crt 

Revoking certificate 'certs/user1.client.crt'

Reason for revocation: 

1. unspecified
2. keyCompromise
3. CACompromise
4. affiliationChanged
5. superseded
6. cessationOfOperation
7. certificateHold

Enter 1-7 [1]: 2
You are about to revoke this certificate with reason 'keyCompromise'.
Are you SURE you wish to continue? [y/N]: y
Using configuration from conf/ca.conf
Revoking Certificate 02.
Data Base Updated
Using configuration from conf/ca.conf

Server certificate revoked.

We need to copy the new CRL to the directory where mosquitto expects the CRL:

# sudo cp crl/ca.crl /etc/mosquitto/certs

And after restarting the broker we have:

mosquitto_pub --cafile /etc/mosquitto/certs/ca.crt -h localhost -t "test" -m "message" -p 8883 -d  --cert certs/user1.client.crt --key private/user1.client.key  
Client mosqpub/7640-pcortex sending CONNECT
Error: A TLS error occurred.

And on the logs:

1478625909: OpenSSL Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
1478625909: Socket error on client , disconnecting.
1478625910: New connection from 127.0.0.1 on port 8883.

Cloud based deployment for IOT devices

Following up on my previous post Cloud based CI with Platformio, after we have the build output from the Continuous Integration process, we are able now to deploy to our devices.

This last deploy phase of the cycle Develop, CI, Deliver using Cloud infrastructure, only makes sense to devices that are powerful enough to have permanent or periodic network connectivity and have no problems or limitations with power usage, bandwidth, are in range and are able to remotely be updated.

In reality this means that most low power devices, devices that use LPWAN technologies like LoraWan or SigFox, devices that are sleeping most of the time and are battery powered are not able to be easily updated. For these cases the only solution is really out of band management by upgrading locally the device.

So the scope of this post is just to simply build a cloud based process to allow ESP8266 devices to get update firmware from the CI output. On it’s simplest form all we need is to create a web server, make the firmware available at the server and provide the URL for OTA updates to the ESP8266 that use the HTTP updater.

One can already use from the squix blog the PHP file to be deployed on PHP enabled web server that delivers the latest builds for devices requesting over the air updates.

Openshift PaaS Cloud Platform

The simplest way of making the Squix PHP page available on the cloud is to use the great Platform as a Service Openshift by RedHat. The free tier allows to have three applications (gears) available and the sign up is free. At sign up time we need to name our own domain suffix so that, for example I choose primal I’ll have URL’s such as application-primal.rhcloud.com.

Openshift offers a series of pre-configured applications ready to be deployed such NodeJs, Java, Python and PHP.

Openshift preconfigured platforms

So after sign up, all we need is to create a new application based on the PHP 5.4 template, give it an URL (it can be the default PHP), and that’s it: we have our PHP enabled web server.

Deploying code to Openshift

To deploy code to Openshift we use the Git tool for manipulating our application repository on the PaaS cloud platform.

So we must first clone our repository locally, modify it and then upload the changes.

For obtaining the repository URL and connection details, we must first setup our local machine with the rhc command line tool and upload our public SSH key to the Openshift servers:

 [pcortex@pcortex:~]$ gem install rhc

If the gem tool is not available, first install Ruby (sudo pacman -S ruby).

We then setup the rhc tool with the command rhc setup. Complete details here.

The command rhc apps should list now our Openshift applications:

[pcortex@pcortex:~]$ rhc apps
nodejs @ http://nodejs-primal.rhcloud.com/ (uuid: 9a72d50252d09a72d5)
-----------------------------------------------------------------------------
  Domain:     primal
  Created:    Aug 26  3:43 PM
  Gears:      1 (defaults to small)
  Git URL:    ssh://9a72d50252d09a72d5@nodejs-primal.rhcloud.com/~/git/nodejs.git/
  SSH:        9a72d50252d09a72d5@nodejs-primal.rhcloud.com
  Deployment: auto (on git push)

  nodejs-0.10 (Node.js 0.10) 
----------------------------             
    Gears: 1 small 
                    
php @ http://php-primal.rhcloud.com/ (uuid: c0c157c41271b559e66) 
-----------------------------------------------------------------------                    
  Domain:     primal          
  Created:    12:16 PM  
  Gears:      1 (defaults to small) 
  Git URL:    ssh://c0c157c41271b559e66@php-primal.rhcloud.com/~/git/php.git/                
  SSH:        c0c157c41271b559e66@php-primal.rhcloud.com 
  Deployment: auto (on git push) 

  php-5.4 (PHP 5.4)
  -----------------
    Gears: 1 small

You have access to 2 applications.

We pull now the remote repository to our machine:

[pcortex@pcortex:~]$ mkdir Openshift
[pcortex@pcortex:~]$ cd Openshift
[pcortex@pcortex:Openshift]$ git clone ssh://c0c157c41271b559e66@php-primal.rhcloud.com/~/git/php.git/
[pcortex@pcortex:Openshift]$ cd php
[pcortex@pcortex:php]$ wget https://raw.githubusercontent.com/squix78/esp8266-ci-ota/master/server/firmware.php 

We should now change the PHP file so it uses our repository to bring up our firmware:

 <?php
    $githubApiUrl = "https://api.github.com/repos/squix78/esp8266-ci-ota/releases/latest";
    $ch = curl_init();

And then it’s just to commit the change to Openshift:

[pcortex@pcortex:php]$ git add firmware.php
[pcortex@pcortex:php]$ git commit -m "Added firmware.php file"
[pcortex@pcortex:php]$ git push
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 924 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Stopping PHP 5.4 cartridge (Apache+mod_php)
remote: Waiting for stop to finish
remote: Waiting for stop to finish
remote: Building git ref 'master', commit a72403a
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is 8fdecb3f
remote: Activating deployment
remote: Starting PHP 5.4 cartridge (Apache+mod_php)
remote: Application directory "/" selected as DocumentRoot
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://php-primal.rhcloud.com/~/git/php.git/
   321e48b..a72403a  master -> master

And that’s it: the link for HTTP OTA is available at http://php-primal.rhcloud.com/firmware.php

Final notes:

With the above firmware.php file we can deliver a single firmware file to any device that calls the page.

But a better solution is needed if we want to:

– Deliver multiple firmware files to different devices
– Deliver different versions of firmware files, for example be able to lock a specific version to some devices
– Know which devices have updated
– Know which version of firmware the devices are running

and of course, add some security.