ESP8266 Firmware messing arounds…

So after I checked how to restore the base firmware that my board came with, It was time to do some messing with alternative firmwares…

The first one that I’ve heard was the Lua base NodeMCU firmware, now open sourced available at https://github.com/nodemcu/nodemcu-firmware.git .  The official web site seems to be here: http://nodemcu.com/index_en.html

This one is rather interesting that allows not to control the ESP8266 by using AT commands (tedious…) but to use some of the processing power on the chip itself (it has a processor, ram and flash) using the Lua language. So for example we can use Lua to set up the wifi communication and use the serial port only for data transfer between the Arduino and the ESP8266 without using tons of AT commands.

To use this firmware, use the following procedure:

1st) Download the NodeMcu firmware. Just do a git clone https://github.com/nodemcu/nodemcu-firmware.git on a working directory.

2nd) Put the Esp8266 in firmware update mode (tie GPIO0 to GND and power cycle)

3rd) With the esptool.py upload the NodeMcu firmware located at the pre_build/latest/nodemcu_512k_latest.bin directory, by using the following command:
/pathtoesptool/esptool.py -p /dev/ttyUSB0 write_flash 0x000000 nodemcu_512k_latest.bin

After the firmware is uploaded, disconnect the GPIO0 from ground and reset the module.

Connect through the terminal and it should show this after (another reset):
NodeMcu 0.9.4 build 20141226 powered by Lua 5.1.4'
And pressing ENTER should show the > prompt.

We can now program the Esp8266 in Lua. The chip when reset calls an init.lua file if it exists in flash storage.

The .lua files can be programmed on our PC and then uploaded to the esp8266 chip, without any need to enter the firmware upgrade mode. There are several tools to do that namely the luatool available at https://github.com/4refr0nt/luatool.git . Just make sure that no other terminal is using the serial port when uploading the files. Just check the readme of the luatool for some samples on using the Lua language.

We can also see some Lua samples and discussion forum here: http://www.esp8266.com/viewforum.php?f=17 and some Lua samples for the ESP8266 here: http://bbs.espressif.com/viewtopic.php?f=15&t=45&p=175&hilit=lua#p175

There is another esp8266 firmware, MicroPython where the language is not Lua but Python, but it hasn’t yet support for the WiFi part of the esp8266 chip I think, so not very useful for the moment.

Finally there are firmwares based on the original esp8266 firmwares, using the AT command set.

These normally are named eagle_something and are uploaded into the esp8266 also with the esptool.py tool.

I’ve tried this one: http://bbs.espressif.com/viewtopic.php?f=15&t=105 that adds NTP (Network time protocol) support to the command set. Please note that after flashing this firmware, the base baud rate changes to 115200 instead of 9600.

ESP8266 Firmware update and recovery

So after I’ve received my ESP8266 I’ve built a support protoboard with pin headers for serial connection, Voltage Shifter (used one for I2C voltage shifting), a LM317 voltage regulator (The board uses an external power source) and a reset button so that I can simply reset the esp8266 easily. I’ve also put the GPIO0 and GPIO2 pins available on pin headers, mainly so that I can short the GPIO0 to Ground if I needed to update the firmware, and of course, I glad I did this…

After some tinkering around I came across these page for firmware update using the interner/cloud:
http://blog.electrodragon.com/cloud-updating-your-wi07c-esp8266-now/

With the following AT command: AT+GMR I’ve determined that my version was 0018000902-AI03, and supposedly it supports the Cloud update.

Also as far the documentation is saying, there is no need for the GPIO0 pin to be grounded to do the firmware upgrade, so I wrote the dreaded command: AT+CIUPDATE after I did the basics, namely connected to an access point and with available internet connection (no mid path proxys).

The upgraded started, but not at it is shown in the above link. For example it shows the command execution something like: +CIUPDATE:1 found server but all I’ve got was +CIUPDATE:1 and +CIUPDATE:2 and so on and no OK at the end. Just garbage…

At this point I though that the ESP8266 was bricked… because I always got garbage despite of configuring my terminal with several different baud rates, including the baud rate 76800 that according to some sources (well google…) it is the default boot loader baud rate. But I had no success and so I thought it was bricked.

Well that is not exactly the case because we can flash a new firmware using the old procedure of tying the GPIO to ground while the ESP8266 is directly connected to the computer via FTDI.

Please note that these instructions are for Linux:

So what is need is the esptool.py available here: https://github.com/themadinventor/esptool/blob/master/esptool.py
and the latest firmware available here: https://drive.google.com/folderview?id=0B3dUKfqzZnlwR2pVWjg3NXRVdW8&usp=drive_web&tid=0B3dUKfqzZnlwRjFaNTUzZFptbzg#list

Download and unzip the firmware file at the same directory where the esptool.py is located.
I’ve tried both firmwares: ESP8266_AT_V00180902_02_baudrate watchdog added.zip and ESP8266_AT_V00180902_04 (0018000902-AI03)_unconfirm version.bin, and kept this last one, the original that my board came with. With both firmwares the cloud update, at least with me fails.

So with GPIO pin grounded, power up the ESP8266, and execute the following command for uploading the 0018000902-AI03 firmware:
./esptool.py -p /dev/ttyUSB0 write_flash 0x000000 ESP8266_AT_V00180902_04\ \(0018000902-AI03\)_unconfirm\ version.bin

Where /dev/ttyUSB0 is my FTDI usb port. Some times when trying to flash I had the following error (While I was in the Cloud update testing cycle):
Connecting...
Traceback (most recent call last):
File "./esptool.py", line 353, in
esp.connect()
File "./esptool.py", line 138, in connect
raise Exception('Failed to connect')
Exception: Failed to connect

where the esptool.py failed to connect. But pressing my (handy) reset button, and trying again solved the issue.
As the firmware is uploading we can see the progress of flash erasing and then the firmware upload progress. At the end something like this shows up:
Connecting...
Erasing flash...
Writing at 0x0007ec00... (100 %)

Leaving…

Now power off, by disconnecting the USB cable and the external power source to the ESP8266, disconnect the GPI0 pin, so now it floats (not connected to anything) and power up again.

The ESP8266 should be functional again at baud rate 9600.

At the end I’m still unable to do a firmware cloud update…

Hope this helps

Using KDE Kate editor as an Arduino IDE

The Arduino IDE is the tool of choice for building and programming for the Arduino platform. It has all that is need for casual programming, namely a simple way to choose and select the Arduino board and serial port, a simple interface, and a simple editor.

But those of us used to other tools for programming, like NetBeans, IntelliJ IDEA (I use it as the Android Studio version) or Eclipse, can find the editor lacking in little things that makes programming easier.

Anyway I was able to use NetBeans for programming and deploying arduino sketchs, with a great feature that is/was command auto completion for the Arduino C “language”. But Netbeans is big, and all it does is to call the Atmel tools for compiling and uploading sketchs into the arduino boards.

So I’m a KDE user, and KDE has a great extensible editor named Kate. Why not use it for programming for the Arduino?

As I said, Kate is a powerful extensible editor and  there are two plugins for making Kate “friendlier” in this task as a replacement for the Arduino IDE.

What we need to accomplish this? We need the following files/software:

Syntax highlighting for ino and pde files: https://github.com/jeroendoggen/Arduino-KDE-syntax-highlighting/blob/master/ArduinoKate.xml

Base project at: https://github.com/jeroendoggen/Arduino-KDE-syntax-highlighting

Just download the ArduinoKate.xml file and as root copy it to the /usr/share/kde4/apps/katepart/syntax folder. Just make sure that on your distribution that the path is correct.

Now if opening an .ino or .pde file with the Kate editor, we should have syntax highlight.

Next we need to download an Arduino Make file from https://github.com/sudar/Arduino-Makefile. Just download it and unzip it to a directory that won’t be deleted or moved. You also need the Arduino IDE to make this work, as this make file will get the tools and libraries from the Arduino IDE installation directory.

Second, change the .bashrc file so the following environment variables are added:

# Base directory where the arduino IDE is installed. Change to suit your directory
export ARDUINO_DIR=/home/pcortex/arduino
# Base directory where the Sudar Arduino Makefile files where installed
export ARDUINO_DIR=/home/pcortex/00.Develop/Arduino-Makefile-master
# Base Arduino tools directory
export AVR_TOOLS_DIR=/usr

We have almost everything ready:

Just launch now the Kate editor and goto Settings->Configure Kate->Plugins and enable the Build and Project plugin.

We are now ready. Let’s develop:

1) Create a directory for your project: mkdir ~/Arduino-project

2) Change to that directory and create a makefile. The content is simple:

BOARD_TAG = uno
MONITOR_PORT = /dev/ttyACM*
#include $(ARDMK_DIR)/Arduino.mk
include /home/pcortex/00.Develop/Arduino-Makefile-master/Arduino.mk

Save the file. This makefile is for the Arduino Uno. To compile for other boards, on this directory and with this make file saved, just run the command make show_boards to see the available boards. Then change the BOARD_TAG appropriately to your board. For example, for the Arduino Mini Pro 5v the BOARD_TAG is pro5v328 and not uno.

3) Create the Kate project file:  This file MUST be named .kateproject :

{
"name": "My Arduino Project Name"
, "files": [ { "git": 1 }, {"directory": ".", "filters": ["*.ino","*.pde","*.c","*.cpp", "*.h" ] } ]
, "build": {
                  "directory": "."
                   , "build": "make all"
                   , "clean": "make clean"
                   , "upload": "make upload"    
            }
}

The JSON “name” tag will define the name that shows up at the Kate Project dropdown box selector.

4) If using Git (as I do) just add a .gitignore file where the *.o and *.d files are ignored.

That’s it.

Create your main.ino file and add code.

To compile, just go to Build->Build Default Target.  This is equivalent to Verify on the Arduino IDE.

To upload, on the Build plugin window, just select (or add) the make upload option, and the Makefile will upload the sketch/program to the board.

If any errors happen, clicking on the Build plugin output tab, on the lines with errors and line numbers, will jump to the exact location of the error.

For more information refer to Sudar documentation for the Makefile.

Happy coding!

 

Arch Linux and PyQwt installation

Some things that we take for granted on other distributions, seem to take a bit of work on Arch Linux. Basically is because that Arch Linux is more or less a bleeding edge distribution, and some packages lag behind in that continuous update cycle .

So I needed to use python to do some plotting with the QwtPlot widget available at the Qwt library. To use the Qwt widgets  from python, we need the Python bindings PyQwt.  The problem with this is that installing PyQwt from AUR uses the deprecated build system based on the configure.py script and the pqtconfig object that no longer exists on the most recent version of PyQT on Arch Linux.

So when we try to install PyQwt from Arch Linux AUR repository, the following error happens:

Requires at least PyQt-4.2 and its development tools.

An the installation fails. This is due to the missing pqtconfig python module. We can see that by running python and executing the following line:

import pyqtconfig as pyqtconfig

This error can be seen a fault of either: PyQwt hasn’t had no updates for the new PyQt releases, or PyQt, in it’s newest version, didn’t keep backward compatibility.

The fact is that it seems that is the PyQt package on Arch Linux that didn’t keep that backward compatibility, by dropping out the pqtconfig class/object on the package installation. Who knows…

This blog entry: http://vogelchr.blogspot.pt/2014/08/building-pyqwt-520-by-manually.html has the solution and it solves the issue with the PyQt package by installing the missing module.

Just build PyQT from sources, and at the end just copy the missing module.

  1. Download PyQt from http://www.riverbankcomputing.co.uk/software/pyqt/download
  2. Extract the package at a temporary directory
  3. Build it: python2 configure.py -v /usr/share/sip –qsci-api -q /usr/bin/qmake-qt4 
  4. Install the missing module: sudo install -m644 pyqtconfig.py /usr/lib/python2.7/site-packages/PyQt4/pyqtconfig.py

From that point, installing PyQwt from AUR works just fine.

SSH over HTTP Proxy

Using SSH to connecting to an host when an HTTP Proxy is between the client and the host, can not be done directly without some configuration.

On Linux based machines the solution is to install and run corkscrew, a program that can tunnel the SSH protocol through an HTTP Proxy.

So how to do the configuration?

1) First install the corkscrew program with your package manager. On Ubuntu family: apt-get install corkscrew

2) Then you need to configure SSH to use corkscrew when connecting to the host that has a http proxy between.

3) Goto to your home directory and change to the hidden directoy .ssh within a command shell window.

4) Create or edit a file named config. The name is just config. No extensions.

5) Add the following lines to the config file

Host <IP_of _remote_host>  
 ProxyCommand corkscrew <IP_of_HTTP_Proxy> <HTTP_Proxy_Port> %h %p <auth_file>

Where the <IP_of_remote_host> is the public ip address of the host where you wish to connect.

The <IP_of_HTTP_Proxy> and <HTTP_Proxy_Port>  are the IP address and Port of you local http proxy server that you wish to go through.

And finally, if your proxy server requires authentication, by username and password, just give a complete path to a file where Proxy credentials are stored, for example /home/primalcortex/.corkscrew_auth

This file content must be something like:

username:password

For example a complete config file example:

Host 78.0.1.3
    ProxyCommand 192.168.1.1 8080 %h %p /home/primalcortex/.corkscrew-auth

and the .corkscrew-auth file:

myproxyuser:rtwertjwe4

6) Just connect now:

ssh myremoteuser@78.0.1.3

or when not using the default ssh port:

ssh -p 12345 myremoteuser@78.0.1.3

7) Done!

So why we need this?

Well, first is of course, to access a remote machine, but ssh can forward local ports to remote ports, and this is important because, with this feature we can use Thunderbird to directly connect to a remote server by using the standard IMAP and SMTP protocols through an HTTP proxy.

MySQL on a Ubuntu VPS

Using the great site lowendbox.com I’ve “bought” a Ubuntu based VPS (Virtual Private Server) so that I can use for my testings…

Anyway, I needed to install MySQL database on this Ubuntu Server based VPS, which is simply done by running the following command:

apt-get update
apt-get install mysql-client mysql-server

During the installation process a password for the root user is required. Just make sure that it’s strong enough (Hint: use keypass password generator…)

After installing and running the MySQL server is available at port 3306 and normally only available at the loopback address. But anyway I’ve changed the local firewall rules to block all connections to port 3306 from outside the loopback adapter: Just edit the /etc/rc.local file and add the following lines before the exit 0 command

iptables -A INPUT -p all -s localhost -d localhost -j ACCEPT
iptables -A INPUT -p tcp --destination-port 3306 -j REJECT

Then as the root user just run the file: /etc/rc.local and make sure that the rules are active:

root@vpss:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  localhost.localdomain  localhost.localdomain 
ACCEPT     all  --  localhost.localdomain  localhost.localdomain 
ACCEPT     all  --  localhost.localdomain  localhost.localdomain 
ACCEPT     all  --  localhost.localdomain  localhost.localdomain 
REJECT     tcp  --  anywhere             anywhere             tcp dpt:mysql reject-with icmp-port-unreachable
ACCEPT     all  --  localhost.localdomain  localhost.localdomain 
ACCEPT     all  --  localhost.localdomain  localhost.localdomain 
ACCEPT     all  --  localhost.localdomain  localhost.localdomain 
ACCEPT     all  --  localhost.localdomain  localhost.localdomain 
REJECT     tcp  --  anywhere             anywhere             tcp dpt:mysql reject-with icmp-port-unreachable

And that’s it.

Now we need a backup policy so that anything goes wrong, at least we have some data to recover…

 

 

Kubuntu upgrade from 12.04 to 13.04

Despite Kubuntu 12.04 being a LTS release, and after some weeks upgrading to 13.04 on my personal desktop computer, I decided to do the same on my Work Laptop.

Things didn’t ran as expected…

First the upgrade from 12.04 to 12.10 deleted a bunch of packages, by my command :( and I ended up with no graphical display… A quick look at Xorg.0.log file showed me that my Xorger’s driver for my Intel graphic card was gone.

Anyway, I’ve installed the xservers-xorg-video-intel package and proceeded to upgrade to 13.04.

At the end despite having a graphical desktop, after login on KDE, a qdbus error appeared… qdbus package was missing… (apt-get install qdbus).

Also on 12.04 the transition from the login greeter to the desktop is silky smooth (it is on my desktop), on my laptop it blanks showing a black screen with a mouse cursor, but the desktop shows up abruptly. I don’t have now the KDE logon progress icons…

And finally, DNS settings from my DHCP servers didn’t worked, I had to manually add the dns servers IP to resolv.conf…  This issue was also a missing package, namely dnsmaq. After adding up the nameserver 127.0.0.1 to my /etc/resolv.conf file, everything is up….

Let’s see what is waiting again in the dark…