LPWAN – Starting up with LoraWAN and The Things Network

LPWAN Networks – A simple introduction
Low Power Wide Area Communications (LPWAN) classifies a group of communication protocols featuring low power usage and a high communication range. For Internet of Things communications, where battery powered devices and constrained devices (weak CPU, low RAM/ROM) are the norm, LPWAN use as the communication protocol for IoT makes sense, since it makes possible to have standalone devices with batteries that last years, instead of hours or days, and might be hundreds of meters to Kms away from a base station/gateway.

But LPWAN protocols, in contrast have low communication bandwidth, since from power, range and bandwidth, we can only have two of those, and while this might be a problem for certain applications, IoT devices don’t require high bandwidth and since most have sparse communication requirements, and when they do communicate they don’t need to transmit a lot of data.

Starting up with LoraWan and The Things Network
One of the possible ways of starting to use LPWAN networks in our IoT devices, is to use a LPWAN protocol named LoraWan, which is based on the Lora physical protocol, and if available at our area, the crowd sourced LPWAN network The Things Network as the backend network for receiving data.

Lora uses the 868Mhz ISM radio band and it uses a form of signal transmission called CSS. The ISM radio band can be used freely by everybody, but under certain rules that depends of the country where the devices are located. So to use that band, rules like maximum emission power and duty cycle (usage of the available spectrum) are defined, and in Europe, the maximum power is 20mW and 1% duty cycle per day. Additionally the back end operator can add other restrictions.

Over Lora, the Lorawan protocol is an open standard and source code for implementing it is available and already ported to several devices, including the Arduino, Raspberry PI and the ESP8266, among others.

The software for implementing the LoraWan protocol at the end devices (nodes) is IBM’s LMIC library, available at Github and on Platformio libs:

[ ID  ] Name             Compatibility         "Authors": Description
[ 852 ] IBM LMIC framework arduino, atmelavr, atmelsam, espressif8266, intel_arc32, microchippic32, nordicnrf51, teensy, timsp430 "IBM, Matthijs Kooijman": Arduino port of the LMIC (LoraWAN-in-C, formerly LoraMAC-in-C) framework provided by IBM. | @1.5.0+arduino-1

Based on this library we can build code and nodes that are able to transmit data through LoraWan to a network backend. Specifically for the Arduino, the Atmega328 is the bare minimum to begin with, since the library, due to use of AES encryption functions occupies a lot of memory.

The backend can be provided by really anyone, including Telco operators, or private and crowd source operators like The Things Network (TTN). TTN provides the backend and management services, but depends on crowd sourced gateways, and might not be available at our area. Still it is possible, for testing, to build our own gateways, our buy them, and connect them to the Things Network. TTN doesn’t require any access fees (yet).

So with LoraWan an the Things Network, we can build our own nodes and gateways since code and hardware can be easily obtained, connect them and use it to build applications.

Regarding LoraWan we can also read this great introduction from Design Spark.

Lora hardware:

Anyway the easiest way for starting up using Lora and Lorawan, is to buy an Dragino Lora Shield and connect it to an Arduino Uno or Mega.

Dragino Lora Shield

Dragino Lora Shield

This is a great shield to startup since doesn’t need any soldering or complex configuration, just plug it on an Arduino shield and use the LMIC library and some code samples. Also the Dragino Shield can be used to build a simple gateway by connecting it to a Raspberry PI or ESP8266 to build what is called a single channel gateway, that allows to test gateway functionality, but it isn’t quite compatible with the LoraWan specifications. Anyway it gets the job done if there is no gateway nearby. Just make sure that you by version 1.3 or 1.4 of the shield. Mine also came with an SMA antenna.

Other alternatives to start using Lorawan are available at eBay/Aliexpress and other renowned name shops, namely Lora radio transceivers at around 8/16€, for example the HopeRF RFM95. Those we can also use them with Arduino or ESP8266 to build our nodes or single channel gateways.

Just make sure that the frequency for these modules and shields must match the allowed radio transmission frequency in your area. In my case, in Europe is 868Mhz, but for example at USA is 900Mhz.

Dragino Lora Shield Jumpers
The Shield has some jumpers and only looking at the schematic and cross referenced them with the RFM95 module (used by the shield as the Lora transceiver) I could see what they where for:

– There are two set of jumpers:

  • One defines the pins for SPI communication: SV# Jumpers;
  • The other set defines which data pins of the RFM95 module are exposed to the shield allowing them to be connected to the Arduino: JP# Jumpers.

The JP# jumpers connect the following if closed:

  • JP1 – RFM95 DI1 – Arduino pin D6
  • JP2 – RFM95 DI2 – Arduino pin D7
  • JP3 – RFM95 DI5 – Arduino pin D8

The RFM95 pin DI0 is permanently connected to Arduino pin D2.
The RFM95 pin RST (Reset) is permanently connected to Arduino pin D9.

Their function for Lora might be the following (The function depends how the RFM95 module is used)

  • RFM95 DI0 – Indicates if data was received and is ready to be read through the SPI bus. Indicates end of transmission for data that was previously sent: RXReady/TXReady
  • RFM95 DI1 – Receive timeout for Lora mode.
  • RFM95 DI2 – Receive timeout for FSK mode.
  • RFM95 DI5 – Used by Semtech Library. LMIC library doesn’t use it.

The SV# jumpers connect:

  • SV2 – SPI Clock line. Default on pin D13, otherwise on Arduino SPI CLK on the ICSP header.
  • SV3 – SPI Data line In (MOSI). Default on pin D11, otherwise on Arduino MOSI pin on the ICSP header.
  • SV4 – SPI Data line Out (MISO). Default on pin D12, otherwise on Arduino MISO pin on the ICSP header.

The SPI Chip Select line is always at pin D10.

So now we know that D10, D9 and D2 are used permanently connected and used by the shield, and the others can be connected or disconnected if needed or not.

LMIC software with Dragino Lora Shield:

To start using the Dragino Lora Shield so it connects to a LPWAN network, we can start using the following example: TTN node by ABP. ABP means Activation by Personalization, which means that all data for joining the network will be explicitly defined on the code. Other alternative would be OTAA: Over the air activation, where the gateway sends the configuration data to the node. Since we don’t know if we have a gateway in range, let’s start first with ABP mode.

The above code uses the LMIC library for implementing the LoraWan stack.
Since LMIC library doesn’t use DI05, we can remove the JP3 jumper, and free this IO line for other things, like another shield.

To use the LMIC library we must define first the pins that are used according to the shield configuration:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, 7}             //DIO pin mapping for DIO0, DIO1 and DIO2  

But DIO2 pin is only used for FSK modulation, and so if only using LoraWan we can also open up the JP2 jumper, and our LMIC pin configuration can be as follows:

// Pin mapping
const lmic_pinmap lmic_pins = {
    .nss = 10,                   //Chip Select pin. In our case it is always D10.
    .rxtx = LMIC_UNUSED_PIN,     //Antenna selection pin: not used in our case.
    .rst = 9,                    //Reset pin used to reset the RFM95: D9
    .dio = {2, 6, LMIC_UNUSED_PIN} //DIO pin mapping for DIO0, DIO1. DIO2 is not used  

Connecting to TTN
Connecting to The Things Network (TTN) depends of course of an available TTN gateway at the nodes range. Still we need to configure some parameters to allow the node to connect.

On this example the connection is done through Activation by Personalization. This means that we should put on our code the Network Session key and Application Session key on the code. To obtain these values we need to register on the TTN site, TTN Dashboard and add an ABP device..

From this site, then we can get the three configuration parameters that we need:

  • Device ID
  • Network Session key
  • Application Session key

Note by pressing the icon we can get the values in a form ready to paste it on the code:

 * TODO: Change the following keys
 * NwkSKey: network session key, AppSKey: application session key, and DevAddr: end-device address
static u1_t NWKSKEY[16] = { 0xAA, 0x0F, 0x29, 0xD3, 0x9D, 0x7A, 0xAE, 0x3B, 0x54, 0xCF, 0xDF, 0x2F, 0x2A, 0x23, 0x55, 0xB5 };
static u1_t APPSKEY[16] = { 0x2F, 0x85, 0x43, 0x5B, 0x34, 0x9C, 0x80, 0xC6, 0xA8, 0xFA, 0x27, 0x49, 0x5A, 0x36, 0x82, 0x21 };
static u4_t DEVADDR = 0x95337738;

And that’s it, when running the sketch, and if in range of a gateway, the sent message should appear at the Dashboard:

Messages received for the device

Messages received for the device

Final thoughts:
In my area there are at least, supposedly, two active TTN gateways and I’m around 2Km’s away from them in a dense urban area.
But when running the sketch the first times, I had no results what so ever.

One of the configurations options for LoraWan is what is called a Spread Factor that, in a simplest way, exchanges range with on-air time for transmission, being SF7 the fastest speed and shorter range and SF12 the slowest speed and higher range. The sketch default value for the sprea factor was SF7, and changing it to SF12:

    // TTN uses SF9 for its RX2 window.
    LMIC.dn2Dr = DR_SF9;

    // Set data rate and transmit power for uplink (note: txpow seems to be ignored by the library)
    LMIC_setDrTxpow( DR_SF12 ,14);

    // Start job

With SF12 the message started be received and the sequence numbers of the messages received where continuous, so no lost messages.

Dropping to SF11, also worked fine, and the message sequence received and shown on the TTN Dashboard where still continuous.

At SF10, some of the messages where lost, almost 75% of them. In this case, arranging the antenna position (to exactly vertical) and placement did alter the reception successes.

Processing received data
After the data is at the TTN backend there are several ways of getting it. For reference on the TTN site there are instructions in how to access the data.

Crash course on Lora
For further information the next one hour video is a great starting point:

The slides for the presentation are available here:


Communication over 433Mhz links

I’ve bought, about a year ago, maybe more, some 433Mhz transmitter and receivers , so I could build a mailbox monitoring solution, loosely based on LOFI Project. The base idea was when someone put something in my mailbox I was notified. Anyway, the ESP8266 came along, and after some experiments, there is no way that the ESP8266 can be used for implementing the monitoring mailbox project I was thinking due to power consumption but mainly because the high distance across several floors and walls between the mailbox and my access point. So back to basics and to the original simpler idea of using plain 433Mhz link for transmitting data.

I’ve started to do some experiments using these devices:

eBay 433Mhz RX TX

All I can say about these is while the transmitter is ok and works fine across floors and walls (I can see the signal clearly using my SDR), the receiver is absolute garbage and useless for the intended purpose.

The receiver is only able to receive in line of sight with the emitter when there are no obstacles, and even in this scenario with both emitter an receiver with attached antennas, the maximum distance between the emitter and the receiver is 4/5 meters at maximum.

The solution is to use the much better and higher cost RXB8 receiver that has according to the datasheet -114dBm sensitivity, but it does work and is very good.


The code for interfacing with this receiver is the same code that works with the cheaper receiver and is the Arduino Manchester encoding library.

Using the RXB8 with an attached 433Mhz antenna (I’m using an antena with 5dBi and a SMA connector), the results are simply superior, with the end result that the signal/messages when the emitter is located several floors down and across several walls, are received correctly and are able to be decoded by the Arduino.

Anyway, one of the interesting things on the LOFI project is/was the Hamming Error Correction implementation found here in the RobotRoom blog.

So I’ve forked the original mchr3k Arduino Manchester encoding library and added the Hamming Code Error correction support: Arduino Manchester encoding library with Hamming EC support.

The usage is quite simple, and can be seen on the examples Hamming_TX e Hamming_RX.

The two new functions are:

uint8_t Manchester::EC_encodeMessage( uint8_t numBytes, uint8_t *data, uint8_t *ecout)

where the input data buffer is provided by the data parameter with the buffer size also provided by the numBytes parameter. The output buffer ecout should at least be one third bigger than the original buffer since for every two input bytes an additional parity byte is added.

For receiving the buffer with the data and parity data is decoded and corrected if any errors are detected. Errors are corrected if possible.

uint8_t Manchester::EC_decodeMessage( uint8_t numBytes, uint8_t *ecin, uint8_t *bytesOut, uint8_t *dataout )

The input data is the size and the buffer with the data and parity, namely the numBytes and ecin input buffer pointer, and the output is returned on the dataout buffer, with the decoded size also returned on the bytesOut parameter.

Above these functions we can build some logical protocol for the received data, but that really depends on the way we want to use the library.

ESP8266 – Developer board for the ESP-01 and ESP-12

Not sure what might be the expression in English to express my level of frustration, but I finally gave up on Lua and the Nodemcu firmware for the ESP8266. The combo of ESPlorer and Nodemcu with the fact that isn’t required to flash the ESP8266 to deploy and test new code is in fact a great advantage. But this comes with a price, a hefty price. Above certain levels of complexity or code size, the Nodemcu firmware just starts to reboot randomly or hangs, and this adds a high level of frustration to developing medium size program. Add one more instruction, or line, and presto. It doesn’t work anymore, and I’m starting to delete comments, spaces, empty lines, to make it work again… Why bother… I’m not alone on this one: 4 reasons for leaving nodemcu…. My opinion Nodemcu is great for simple things and fine for testing, even if there is the need to learn Lua. Anything above that, there are better alternatives now and on the horizon.

Solving the problem
The problem that Nodemcu solves is the fact that we don’t need to flash any new firmware to test new programs or code. To flash new firmware we need to pull GPIO0 to Ground and pulse reset and then use a program, like esptool.py to push the new firmware to the chip. With Nodemcu we do this only once, on the initial ESP8266 configuration.
For other alternative firmwares, we need to always flash the firmware with the new code/program.
But the esptool.py holds a secret, it can pulse the physical serial port lines DTR and RTS to command GPIO0 and RESET to initiate the ESP8266 in flash mode, transfer the firmware, and when it ends releases the lines and allows the ESP8266 to run the new code. All without changing any connections on the ESP8266 at the physical level. In fact with this we can have the same functionality as the Nodemcu witth other firmwares. Off course off the shelf (ebay…) chips don’t do that, but I’m aware that Nodemcu and Olimex boards do support this kind of flashing mode control through the serial port. I could just bought one of these, but after programming the ESP8266 remains attached to that support electronics (serial chip) that might not be needed anymore.
So off we go building a developer board that allows the easy flashing and programming the ESP8266 but then allows the chip, namely ESP-01 and ESP-12 to detach and be used on their final board.

The board
My idealized board and in semi state of final construction (proto board) supports the esptool.py firmware flashing, powers up either from a power supply or trough mini-usb, and can program the ESP-01 or ESP-12.
My design is based on a circuit available in the esp8266.ru site, with some modifications. Also, due to the fact the DTR and RTS lines do have a function now, it’s necessary that the serial terminal programs do control these lines correctly to see, if desired, the serial port output. The serial console of ESPlorer works just fine and it has specific buttons to control these lines. In fact by pressing the RTS line button we can reset by hardware the ESP8266…
All components of the board are sourced from available items on ebay, which means that building the board is just to solder and connect some components on a protoboard with connector headers for the ESP-01 and ESP-12. Also the board can program both types, but only one at the time…

The USB serial adapter
The USB serial adapter must be one that exposes the RTS and DTR lines, and provides the serial lines at 3.3V, the level for the ESP8266. There are CP2102 based boards available on ebay or DX, that expose all the lines and are 3.3v level.
I bought mine from Alice1101983 on ebay:
It’s data and control pins are 3.3v level, just right for the ESP8266, so no need to do any level conversion, and exposes the DTR and RTS pins. Notice the board holes on the side that have the other exposed pins:
alixpress_cp2102 (Image from Alixpress).
We need to solder a wire from the RTS pin to the ESP8266 RESET pin.

Power source
I’m not powering up the board through the USB/serial adapter. The serial port adapter does have a 3.3v power pin and also a 5v power pin. But the ESP8266 can demand high currents and the 3.3v regulator on the USB/serial adapter is almost certain not to be able to provide that current, even knowing that the USB port can provide 0.5A.
So I’m not using the power provided by the CP2102 serial adapter, and instead I’m using an MB102 bread board power supply with mini usb connector that can provide 5V and 3.3v outputs. I’ve also bought mine on alice1101983: alice_mb102
This specific board receives power through the usual DC jack, but with the mini-usb connector it also can receive power from an USB power supply (mobile charger), or, even better, an USB power bank.
One of the MB102 board outputs is set to 3.3v to power up the ESP8266 boards and any other 3.3v devices, and the other other power output is set to 5V and connected to an IIC I2C level converter allowing the use of I2C 5V level devices like the 16×02 LCD screens.
I’m using these level converter devices: ebay_i2c

The schematics that I’m using are based on this schematics:


I’ve got this from the ESP8266.RU site, the same from ESPlorer, and done some modifications.
With this schema, I can use the Arduino IDE and esptool.py to program the ESP8266 without the need of explicitly enter the firmware flash mode.
Also I’m not showing in my schematics the ESP-12 connections, but they are in parallel for the RX, TX, RTS and DTR lines.

A word of caution
Since now the RTS and DTR lines do have a function, some serial programs to access the ESP8266 serial console do not work. ESPlorer terminal works fine, and it has a handy DTR and RTS toggle buttons.
I’m using my own brewed python pyserial based serial program, that correctly set the RTS and DTR lines, so that after flashing the firmware we can see the serial output if any.
Also the way that reset switch resistor is in series with the switch AFTER the reset pin, means that with the serial port adapter is connected, the reset switch doesn’t work. Without being connected to the USB port, the reset switch works fine. The resistor is there to protect the CP2102 RTS line.

Final thoughts
I’m still waiting for the female headers to connect and start programming the ESP-12 on this developing board.
But so far, every thing is good. I’ve already tested the Arduino firmware ESP8266 Arduino Firmware, Sming, and off I go for testing https://github.com/cesanta/smart.js/tree/master and Espruino, which is a Javascript interpreter for the ESP8266.

1.8 SPI TFT display for Arduino

If you need to buy a display to connect to Arduino (or other boards as the Raspberry PI) don’t waste your money on the Nokia 5110 display. While these Nokia displays are quite cheap, around 2.5€, they are monochrome, and suffer from an assembly issue (at least mine) that the display contacts to the supporting board become loose (it uses some kind of adhesive) and the display looses contrast. Pressing the display against the supporting board makes it ok, but only for a while. The only valid reason that I can see for using one of these displays, if you need one, is memory size available for your program, since driving it is quite easy.

So, for a bit more money, and at less than 4€, we can buy these gorgeous 1.8 SPI TFT display. It is a world apart from the Nokia 5110 display. It has a higher resolution, it has color, it is bright, it is much better.

I’ve bought mine from eBay but there are several versions of this display regarding the controller chip. These versions from the exterior are virtually indistinguishable. The eBay seller of mine, shows a QDTech picture and also provided a Arduino library, that off course didn’t work…

Anyway , to use this display:

Connect the VCC and BL (backlight) pin to your power source (in my case, 5V), and connect the GND to ground.

Connect the RST pin to the RST Arduino Pin directly.

The other data lines should be connected to Arduino through 1K resistors.

We can drive the display by using any pin from Arduino, but to have the best performance we should use the hardware SPI pins 13 and 11.

Note that the pins names on the TFT board can change, depending in what version was shipped, but in my case was:

  • TFT SCLK -> Arduino Pin 13  (Serial Clock)
  • TFT DIN -> Arduino Pin 11 (Data)
  • TFT D/C -> Arduino Pin 8 (Data/Command)
  • TFT CS -> Arduino Pin 10 (Chip Select)

All these pins are connected in serial with an 1K resistor.

After the connection it took a while to find out the correct library to use to drive my TFT version.

The eBay description provided the QDTech driver and UTFT library which didn’t work. In fact the UTFT library was too big that I couldn’t even upload the code to my Arduino Uno without cutting out some of the sample code.

In my case the correct library for my display was the Adafruit ST7735 library available here: https://github.com/adafruit/Adafruit-ST7735-Library  But this doesn’t mean that it is the correct library for your display. It seems there are some different chips for this display like the above ST7735, the S6D02A1 and the QDTech. We need to test several of the libraries to get the right version.

ESP8266: Setting up native SDK’s on Linux and exploring some applications

The information for installing and setting up the native SDK for the esp8266 from Expressif is readily available in the Internet:

With the native SDK we can develop our own firmware using the Espressif SDK and the C or C++ language. Then we can use directly the internal esp8266 processor, allowing to do interesting things in a more direct way, like implementing MQTT, driving hardware, and so on.

So why this post?  Basically is to put into writing some of my own annotations to the issues that I had installing the SDK and the cross compiler tools on my Linux machines.

Starting up:

The original instructions that I’ve followed where these: https://github.com/esp8266/esp8266-wiki/wiki/Toolchain
I installed the tool chains in two computers, both running Linux: an Arch Linux distribution and a Kubuntu 14.04 distribution (and time is up for this one because I’m moving it also to Arch Linux. All I’m missing is to install a SSD).

1st) Installed the required pre-requisites that are documented on the above link. On Arch Linux the package installation tool is pacman and probably the package names can be different. I didn’t need to install anything on Arch, but on Kubuntu I needed to install the 64-bit pre-requisites.

2nd) Create the /opt/Espressif directory. The owner should already be your user, except if you have no permissions on the /opt directory.

3rd) Install the Xtensa Crosstool-NG that has the cross compiler and tools for the esp8266 processor:

cd /opt/Espressif
git clone -b lx106-g++ https://github.com/jcmvbkbc/crosstool-NG.git
cd crosstool-NG
./bootstrap && ./configure --prefix=`pwd` && make && make install
./ct-ng xtensa-lx106-elf
./ct-ng build

Some notes regarding this:

  •  We are going to clone the git branch lx106-g++ that has the C compiler and the C++ compiler. There is another branch named lx106 that only has the C compiler.  There is some software out there like Ardunet that need the c++ compiler and so that is the reason why I’ve chosen this git branch.
  • I’ve also changed the url for the git repository from git:// to https:// because in I’m behind a web proxy that needs authentication. Just make sure that the http_proxy, https_proxy and ftp_proxy variables are set. The ftp_proxy variable is needed because at the ct-ng build step will connect to a ftp server to retrieve files. One example for setting these environment variables: export https_proxy=http://myproxyuser:myproxypass@proxyip:port. If you have an internet direct connection, like at home, no need to set these variables.
  • The ./ct-ng build step will take a long time, around 30 minutes in one of my computers and 15 in another.
  • The tool will connect to the ftp server on IP: where are located the Gnu compiler sources.

At the end we need to add the path to the crosstools to our PATH variable.
export PATH=/opt/Espressif/crosstool-NG/builds/xtensa-lx106-elf/bin/:$PATH
You may want to add this line at your .bashrc file. In my case (I’m using KDE desktop, I’ve created a Konsole profile for this type of work and add the new settings to the PATH variable,)

Installing the Expressif SDK

The instructions on the above link are for the 0.9.3 version of the SDK. I’ve installed the most recent version at this time that was 0.9.4:

cd /opt/Espressif
wget -O esp_iot_sdk_v0.9.4_14_12_19.zip https://github.com/esp8266/esp8266-wiki/raw/master/sdk/esp_iot_sdk_v0.9.4_14_12_19.zip
unzip esp_iot_sdk_v0.9.4_14_12_19.zip
mv esp_iot_sdk_v0.9.4 ESP8266_SDK
mv License ESP8266_SDK/
cd /opt/Espressif/ESP8266_SDK
wget -O lib/libc.a https://github.com/esp8266/esp8266-wiki/raw/master/libs/libc.a
wget -O lib/libhal.a https://github.com/esp8266/esp8266-wiki/raw/master/libs/libhal.a
wget -O include.tgz https://github.com/esp8266/esp8266-wiki/raw/master/include.tgz
tar -xvzf include.tgz

Except the change of the SDK version, the instructions are the same as the original reference post.

Installing the esptool

The esptool will pick up the compiler output and extract/create the files needed for flashing the esp8266. The instructions for this step are the same as in the original reference post. In my case, I’ve got the tool from the sources and not installed the deb file:

cd /opt/Espressif
wget https://github.com/esp8266/esp8266-wiki/raw/master/deb/src/esptool_0.0.2.orig.tar.gz
tar xvzf esptool_0.0.2.orig.tar.gz
cd esptool

Let’s put the tool on the search path. In my case I’ve choose the Xtensa bin directory. (Edit: it should be put at /usr/bin. See below at building the blinky example)

cd /opt/Expressif/esptool
chmod +w /opt/Espressif/crosstool-NG/builds/xtensa-lx106-elf/bin
ln -s /opt/Espressif/esptool/esptool ../crosstool-NG/builds/xtensa-lx106-elf/bin
sudo cp /opt/Espressif/esptool/esptool /usr/bin

Installing the firmware uploader tool esptool-py

cd /opt/Espressif
git clone https://github.com/themadinventor/esptool esptool-py
chmod +w /opt/Espressif/crosstool-NG/builds/xtensa-lx106-elf/bin
ln -s /opt/Espressif/esptool-py/esptool.py crosstool-NG/builds/xtensa-lx106-elf/bin/

With this tool we can upload the generated firmware into the esp8266.


So now we are ready and we can follow the instructions here: https://github.com/esp8266/esp8266-wiki/wiki/Building to starting building things.

There are some code examples, one of them named blinky but the make file assumes that the esptool is located /usr/bin. This is because I’ve not installed the esptool through the deb package. So we can copy the esptool command to the /usr/bin directory: sudo cp  /opt/Espressif/esptool/esptool   /usr/bin.

We can now build the AT demo and the IoT demo. Just follow the instructions. For example for the IoT example just do the following (Make sure that the path is set: export PATH=/opt/Espressif/crosstool-NG/builds/xtensa-lx106-elf/bin/:$PATH):

cd /opt/Espressif/ESP8266_SDK
sed -i -e 's/xt-ar/xtensa-lx106-elf-ar/' -e 's/xt-xcc/xtensa-lx106-elf-gcc/' -e 's/xt-objcopy/xtensa-lx106-elf-objcopy/' Makefile
mv examples/IoT_Demo .
cd /opt/Espressif/ESP8266_SDK/IoT_Demo 
cd .output/eagle/debug/image
esptool -eo eagle.app.v6.out -bo eagle.app.v6.flash.bin -bs .text -bs .data -bs .rodata -bc -ec 
xtensa-lx106-elf-objcopy --only-section .irom0.text -O binary eagle.app.v6.out eagle.app.v6.irom0text.bin 


For uploading just follow the instructions here: https://github.com/esp8266/esp8266-wiki/wiki/Uploading but basically is using the esptool.py to upload the firmware to the esp8266. Just make sure tha GPIO0 is grounded at boot time.

The IoT RTOS based example:

There is a beta SDK for the esp8266 based on the freeRtos according to this: https://github.com/espressif.

To use this SDK (Make sure that the PATH environment variable is correctly defined):

cd /opt/Expressif
git clone https://github.com/espressif/esp_iot_rtos_sdk.git
git clone https://github.com/espressif/esp_iot_rtos_sdk_lib.git
cp esp_iot_rtos_sdk_lib/lib/*  esp_iot_rtos_sdk/lib
cd esp_iot_rtos_sdk


We should have now the IoT RTOS main image ready. To generate the firmware files:

cd app/.output/eagle/debug/image/
esptool -eo eagle.app.v6.out -bo eagle.app.v6.flash.bin -bs .text -bs .data -bs .rodata -bc -ec
xtensa-lx106-elf-objcopy --only-section .irom0.text -O binary eagle.app.v6.out eagle.app.v6.irom0text.bin

And we can flash it:

/opt/Espressif/esptool-py/esptool.py --port /dev/ttyUSB0 write_flash 0x00000 eagle.app.v6.flash.bin 0x40000 eagle.app.v6.irom0text.bin

The default baud rate for the IoT example is 74880…

 Exploring some application:

Now that we have all the tools for developing natively we can check out some applications:

  • esp-mqtt – A native MQTT client for the esp8266
  • Ardunet – A work in progress that mimics the Arduino way of making applications on the esp8266.

The download links are:


For building it, first change your connection parameters at include/user_config.h

cd /opt/Espressif
git clone https://github.com/tuanpmt/esp_mqtt.git
cd esp_mqtt
cp Makefile.linux Makefile
make flash


For building it:

cd /opt/Espressif
git clone https://github.com/Karang/Ardunet.git
cd Ardunet
cp -R /opt/Espressif/esp_iot_rtos_sdk/ld .
cd /opt/Espressif/Ardunet/src/.output/eagle/debug/image
esptool -eo eagle.app.v6.out -bo eagle.app.v6.flash.bin -bs .text -bs .data -bs .rodata -bc -ec
xtensa-lx106-elf-objcopy --only-section .irom0.text -O binary eagle.app.v6.out eagle.app.v6.irom0text.bin

And we can flash it.


So after this long post I hope that things now are a bit more documented from the installation and setting up point of view.

Powering up Arduino boards (or others) with USB power banks

Arduino boards can use the serial USB to provide the power source when connected to a computer either directly, like the UNO and the MEGA, or through the FTDI adapter like the Mini Pro. But if we want to use them when disconnected from the computer and/or in standalone mode we need another form of providing power to them.

When using my Arduino boards in standalone mode, I’m powering them up using rechargeable Ni-MH AA batteries, which works fine, but has the issue that when the batteries are exausted, recharging them can take a long time. In fact to fully recharge some of the 2600mAh batteries that I have can take as much as 12 hours… A common solution for this issue is to have several groups/packs of batteries, but I wanted something better.

In a first approach I started to look at LiPO batteries commonly used in powering up RC toys, drones, and so on. These batteries pack a lot of energy, come at several power ratings and sizes, and need specialized chargers for charging them. They also have the issue that can be quite dangerous and can bring houses down in uncontrolled fires if not properly used. They should not be overcharged and over discharged, otherwise they get destroyed, which means proper chargers and proper electronics to shut them down when bellow a certain voltage level is a necessity.

These batteries have normally per cell/battery 3.7V and are packed in parallel or serial with notations like 7.2V 2S that mean two connected Serial 3.7 cells. (2P means two packed in parallel). Also there is a discharge and charge rate, like for example 20C that means that the battery can discharge at 20 times it’s nominal capacity, for example 2600mAh 20C would mean that a peak current of 46A for 10s can be achieved. Normally charging them is at 1C rate, which means 2.6A for a 2600mAh battery, which should take less than an hour to fully charge, a far cry away from the safer Ni-Mh 12 hours charging time….

The fact is that for using LiPO batteries for powering up my Arduino boards mean that I needed specialized chargers and specialized control electronics to avoid over discharge. These chargers and monitoring chips/boards are readily available and are easy to find. Also if I wanted to use the 1S 3.7V batteries on my 5V Arduinos I also needed to have a step up DC converter from 3.7V to 5V, or in the case of a 2S (or above) a step down to 5V converter.

The fact is that while it might be cool to add a LiPO battery, charging board, protection and step up/down converter to power up an Arduino, it might be overkill at least for my endeavours… If my bet was on battery physical dimensions, than size/vs power might be the only available solution.

USB power banks normally use a form of LiPO batteries known as 18650. And these power banks have an all in one board that allows charging through USB, keep the batteries safe above their ratting discharge voltage, and a step-up or step down converter to 5V provided on another USB port: it’s an all in one. In fact we can buy them just for the single board solution for charging and monitoring.

These power banks can be bought with a single 18650 battery (starting at 2600mAh) and multiple 18650 batteries providing much larger power output, while keeping the charging times decent.

Most of the capacities of these batteries that are sold on eBay do not have truly their announced capacity, but hey, if needed we can swap the provided battery for a real Panasonic/Sanyo ones.

So I’ve just bough aluminium encased single cell power banks and an USB to DC male jack cable and let’s see how it works out…

ESP8266 NodeMcu and Lua language and some Arduino issues.

When using the ESP8266 chip with the original firmware, AT based command set, to perform operations, implies that something external, like an Arduino or RPi (Raspberry Pi), must be used to drive/command the chip through the serial port. This external device sends out AT commands and waits for responses.

While for some operations/applications, using the AT commands  might just work when using the Software Serial library from Arduino. In another words, using a software based serial port to communicate might work. The issue is that even at a slow rate of 9600 baud’s, Arduino has some trouble to follow the esp8266 output, due to the lack of flow control.

For example listing the available access points (AT+CWLAP) or even receiving simple web page from the internet, most of the time the Software Serial port only receives incomplete messages or garbled information. This can be worked around by increasing the software serial port buffer size, but is not a solid solution.

In fact without using the hardware port on the Arduino to connect to the ESP8266, I have doubts that some useful work can be done if we simultaneous also want to use the hardware port for debug ( connecting to the computer) and the software serial port to connect to the esp8266. We must use the hardware port for connecting to the esp8266 and use the software serial library for debugging purposes which brings other issues. This situation only arises on the Arduino Uno or Mini where only one hardware based serial port is available. The Arduino Mega has 4 hardware serial ports, so it doesn’t have this issue.

So why all about this ranting regarding Arduino, serial ports and the esp8266?

The fact is that the esp8266 is by itself a microprocessor and so it is able to do some processing on it’s own. So it’s is possible to only send or receive the already processed information needed from/for Arduino, without the Arduino being the processor that is controlling and defining the behaviour of the esp8266 chip like connecting to WIFI and so on. The AT command set is fine for some applications, but is so 80’s … 🙂

One of the solutions available is to replace the original based AT command set firmware for the ESP8266 with the NodeMcu firmware, that’s now open sourced, that allows to add some intelligence to the esp8266 chip, namely connecting, receiving, sending and waiting for data (server mode) using the Lua language running on the ESP8266 chip itself and not on an external device.

After flashing and restarting the esp8266 with the NodeMcu firmware, the esp8266 tries to load a file named init.lua.

Several words of warning regarding  when developing with this firmware:

– If init.lua file crashes, the NodeMcu restarts, and calls again init.lua that crashes again, that restarts NodeMcu and so on. You get the idea.  At initial stages do not develop using the init.lua file. The only way to recover from this crash loop is to flash the firmware again. Do your first developing in a file named testing.lua or init2.lua for example. When ready, just rename it to init.lua.

– NoceMcu is single threaded and the esp8266 has a WatchDog timer. Tight loops will trigger the watchdog timer and restart the firmware. If needed we can we use the tmr.wdclr() which resets the watchdog timer, but then other bad things happen due to the single threaded nature of the firmware, like losing wifi connection, timer triggers not activating, and so on.

A great way to develop is to use the ESPlorer IDE tool. It needs Java JDK 1.8 (Edit: It seems not anymore. JDK 7 might be fine), so make sure that version is installed in your Operation System. Then we can just do /opt/jre1.8/bin/java  -jar ESPlorer.jar to launch the IDE.

Let’s see how in Lua we can connect to WIFI.

The simplest code for init.lua is

-- This is a comment line
  print("Configuring WIFI....")   -- Print something to the serial port if connected
  wifi.setmode( wifi.STATION )    -- Client/Station mode. Other mode could be Access Point
  wifi.sta.config( SSID , APPWD)  -- Connect to SSID with password APPWD.
  -- Example: wifi.sta.config( "MYWNET" , "QuantumEncryptedPassword" )
  print( "Ip Address: " .. wifi.sta.getip() ) -- The .. is the string concatenate string operator. 
 dofile ( "RunMyCode.lua")

This code snippet will connect to WIFI, if successful print the IP, and then lauch the RunMyCode.lua file.

But what happens if the connection fails, or takes sometime? We may end lauching the RunMyCode.lua file without WIFI connectivity.

So I’ve lost a couple of hours, learning Lua and esp8266 quirks to develop my init.lua file, with a lot of help of the esp8266 forum where there are a lot of Lua code snippets and examples.

Due to the single threaded nature of the NodeMcu firmware is not possible to do this (a tight loop waiting for the wifi to connect):

isConnected = false
   -- Lets see if we are already connected by getting the IP
   ipAddr = wifi.sta.getip()
   if ( ( ipAddr ~= nil ) and  ( ipAddr ~= "" ) )then  -- Check if IP is valid
      isConnected = true

until ( isConnected == true )

This will hang the esp8266 and will kick the watch dog timer, restarting the firmware.  Also the only thread running is the the thread executing this repeat/until loop, and so this code snippet will never connect because the background functions that need to be run, so that the connection does happen, never have a chance to run and the isConnected will always be false. The fact is that after the firmware restart, on the lua prompt running the print ( wifi.sta.getip() ) shows that it connected… So the loop is in fact blocking everything else.

Another example:

isConnected = false
   -- Lets see if we are already connected by getting the IP
   ipAddr = wifi.sta.getip()
   if ( ( ipAddr ~= nil ) and  ( ipAddr ~= "" ) )then  -- Check if IP is valid
      isConnected = true
   tmr.wdclr()    -- Reset the watch dog timer.
until ( isConnected == true )

This is even worse because this code segment explicitly resets the watchdog, and so we might thing that the loop will succeed some time in the future, but it won’t. The only way to leave this is to reset by hardware. It seems that the only way that background functions can work, if we can say that, is to let the script end…

So how do we test the connection and we need to end the script? Well timers are the solution:

tmr.alarm ( Alarm_number , Number of mS , Repeat , function )

So this:  tmr.alarm ( 0 , 2500 , 1 , checkWifi )  will use timer #0 to call periodically each 2.5S the checkWifi function (NOTE the lack of () on the checkWifi parameter (checkWifi is a function)).  The Repeat parameter if zero is a one shot call, otherwise it will reset the alarm and trigger it again and again.

So we let the script end and periodically check if we have Wifi. If we have connectivity then we can lauch our code. And so this is my init.lua code:

-- Constants
APPWD   = "QuantumPassword"
CMDFILE = "ping.lua"   -- File that is executed after connection

-- Some control variables
wifiTrys     = 0      -- Counter of trys to connect to wifi
NUMWIFITRYS  = 200    -- Maximum number of WIFI Testings while waiting for connection

-- Change the code of this function that it calls your code.
function launch()
  print("Connected to WIFI!")
  print("IP Address: " .. wifi.sta.getip())
  -- Call our command file every minute.
  tmr.alarm(0, 60000, 1, function() dofile(CMDFILE) end )

function checkWIFI() 
  if ( wifiTrys > NUMWIFITRYS ) then
    print("Sorry. Not able to connect")
    ipAddr = wifi.sta.getip()
    if ( ( ipAddr ~= nil ) and  ( ipAddr ~= "" ) )then
      -- lauch()        -- Cannot call directly the function from here the timer... NodeMcu crashes...
      tmr.alarm( 1 , 500 , 0 , launch )
      -- Reset alarm again
      tmr.alarm( 0 , 2500 , 0 , checkWIFI)
      print("Checking WIFI..." .. wifiTrys)
      wifiTrys = wifiTrys + 1

print("-- Starting up! ")

-- Lets see if we are already connected by getting the IP
ipAddr = wifi.sta.getip()
if ( ( ipAddr == nil ) or  ( ipAddr == "" ) ) then
  -- We aren't connected, so let's connect
  print("Configuring WIFI....")
  wifi.setmode( wifi.STATION )
  wifi.sta.config( SSID , APPWD)
  print("Waiting for connection")
  tmr.alarm( 0 , 2500 , 0 , checkWIFI )  -- Call checkWIFI 2.5S in the future.
 -- We are connected, so just run the launch code.
-- Drop through here to let NodeMcu run

So we can see that to program the esp8266 in Lua using the NodeMcu firmware we need to change a bit our programming paradigms…