Barometric and Temperature Sensor BMP180

Since I’m doing some experiments with my OpenWeatherMap LUA based client on the ESP8266 running the Nodemcu firmware, I’ve bought a BMP180 sensor.

The thing is that even starting with 30K of free heap on the Nodemcu firmware it is almost impossible to run all the code for querying OpenWeatherMap Json interface, driving the 16×02 LCD AND also using the bmp180 sensor for local data.

Anyway, this post isn’t about this, but to one simple thing: The bmp180 values for pressure where OK around the 1000s, but the temperature was completely bonkers… Negative values, high positive values. The issue? Simple, the SDA and SCL line where swapped…

So if using the bmp180 LUA libraries that are available and seem to work for everybody except you, and me ( ūüôā ), check and double check the connections.

ESP8266 NodeMCU firmware updating

My ESP-01 board has not being used for a few weeks due to other projects, and today, after updating my local nodemcu firmware repository I’ve found out that the latest build directory where the binary for the firmware was located was gone.
No problem.
We can find now the pre-compiled binaries on this link: NodeMcu Firmware Releases

There is also a cloud compiling option on this link: NodeMcu custom builds
If by any chance you are having second thoughts in providing your e-mail, use one from Fake Mails and wait for the firmware on the provided (temporary) e-mail.

Anyway, the procedure is always the same:

  • Close anything that using the serial port, for example the ESPlorer tool
  • Put the ESP8266 into firmware flashing mode by connecting the GPIO0 to ground and reset the board or power cycling.
  • Flash the firmware with ./esptool.py -p /dev/ttyUSB0 write_flash 0x000000 nodemcu_float_0.9.6-dev_20150704.bin
  • Remove the GPIO0 from GND and power cycle or reset
  • Connect to serial port and check the NodeMCU version

Update done!

MQTT Publishing on the ESP8266

I’ve done some tests with esp8266 and NodeMcu Lua based firmware, and I’ve come to the following conclusions:

  • We can only publish again in a MQTT topic if, and only if the previous topic call has finish. That’s why we need to use a semaphore to stop multiple publish request at the same time.
  • MQTT based code consumes a lot of heap. So much, that to make it work I have to strip empty lines and comments from the code. It seems that the latest NodeMcu firmware does have a node.compile() function that we can use.
  • Using unbound arrays for storing MQTT messages while waiting for their turn to be publish, will end up using all available heap space, and the esp8266 will reboot, because setting previous array entries to nil won’t free the used space at all.

So I’ve now a sample code that uses a circular buffer to publish messages. The code below has some flaws, namely:

  • Doesn’t check if the buffer is full, it will just overwrite any older unpublished messages.
  • Entries on the buffer have no QOS defined.
  • Has no “intelligence” for example to only override messages with QOS=0 that might get lost anyway, and keep QOS>0 messages. That would mean that eventually the buffer might get full, and all publishing from the user code can fail.

Here is the code:

mqtt_cbuf.lua

-- Configuration to connect to the MQTT broker.
BROKER = "192.168.1.16"   -- Ip/hostname of MQTT broker
BRPORT = 1883             -- MQTT broker port
BRUSER = "user"           -- If MQTT authenitcation is used then define the user
BRPWD  = "pwd"            -- The above user password
CLIENTID = "ESP8266-" ..  node.chipid() -- The MQTT ID. Change to something you like
BFSIZE = 16

-- MQTT topics to subscribe
topics = {"topic1","topic2","topic3","topic4"} 

-- Control variables.
pub_sem = 0         
current_topic  = 1  
topicsub_delay = 50

-- Publishing structures
pub_topic = {}
pub_message = {}
pub_head = 1
pub_tail = 1

print("heap: " .. node.heap() )
m = mqtt.Client( CLIENTID, 120, BRUSER, BRPWD)

print "Connecting to MQTT broker. Please wait..."
print("heap: " .. node.heap() )
m:connect( BROKER , BRPORT, 0, function(conn)
     print("Connected to MQTT:" .. BROKER .. ":" .. BRPORT .." as " .. CLIENTID )
     mqtt_sub() 
end)

function mqtt_sub()
     if table.getn(topics) < current_topic then
          run_main_prog()
     else
          m:subscribe(topics[current_topic] , 0, function(conn)          
          end)
          current_topic = current_topic + 1  
          tmr.alarm(5, topicsub_delay, 0, mqtt_sub )
     end
end

function _publish_data()
   if pub_head ~= pub_tail then
     if pub_sem == 0 then
       pub_sem = 1
       m:publish( pub_topic[pub_head], pub_message[pub_head] ,0 ,0 , function(conn) 
          print("Sending to " .. pub_topic[pub_head] ..": " .. pub_message[pub_head])
          pub_sem = 0
          pub_topic[pub_head] = nil
          pub_message[pub_head] = nil
          pub_head = (pub_head + 1) % BFSIZE
       end)
     end  
   end
end

function publish_data(topic , message )
  pub_topic[pub_tail] = topic
  pub_message[pub_tail] = message
  pub_tail = (pub_tail + 1) % BFSIZE
  _publish_data()
end

function publish_data1()
 publish_data("Out1","Mensagem para out1" )
 publish_data("Out2","Mensagem para out2" )
end

function run_main_prog()
     tmr.alarm(2, 5000, 1, publish_data1 )
     m:on("message", function(conn, topic, data)
        print(topic .. ":" )
        if (data ~= nil ) then
          print ( data )
        end
      end )
end

Apparently this code can run for a long time.

Also at the beginning I print the heap space before running and connecting to a MQTT broker.

  • So after a reset: node.heap = 19600 ( with init.lua file)
  • Running the above file ( dofile(“mqtt_cbuf.lua”)): node.heap = 10560 after compiling and before connecting to the MQTT broker
  • After connecting: node.heap = 8136

If we use the new node.compile function with the function node.compile(“mqtt_cbuf.lua”) we should have now a new file named mqtt_cbuf.lc. So:

 

  • So after a reset: node.heap = 19472 ( with init.lua file)
  • Running the above file ( dofile(“mqtt_cbuf.lc”)): node.heap = 12160 after compiling and before connecting to the MQTT broker
  • After connecting: node.heap = 9720

A bit better. Not much, but makes a difference.

Since a small code as the above can fail to run due to lack of heap space, solving some of the issues that I’ve referred on the beginning of this post might only be feasible if really needed. Let’s see if I can improve this.

ESP8266 NodeMcu Heap: Limitations and a small hack to graph heap usage

So I’m messing around with ESP8266 and the NodeMcu firmware that uses the Lua language.

The fact is that I’m hitting some limitations that are hindering development of something more complex than simple web servers or anything else for that matter. Limitations like it can’t be a network server and a client at the same time, no network sockets, like opening up a socket to a server and keep that connection open a use it across Lua files,¬† makes building things much harder when in fact this is a communication centred device… So at the end for more complex projects the native SDK and using the C language is the way to go.

Anyway, I’ve made a small hack that uses an Apache Web server with PHP installed and Dygraphs library to draw out the heap usage in a nice line graph:

Setting up the Apache Web Server: First make sure that you have Apache with PHP installed (for this hack I use PHP because it was already installed and configured). In my case, the Apache document root directory is /var/www .

Create a directory named data under /var/www and make sure that the user running the apache server process has rights to write on the directory. In my case the user is www-data:

ps -ef | grep apache
….
www-data 12118 697 0 Jan04 ? 00:00:00 /usr/sbin/apache2 -k start
….

So I needed to do chown www-data /var/www/data so that Apache can write data to that directory.
On that directory create a file named esp8266heap.csv where the first line has the following content:

Data , Heap

Just make sure that you add a new line at the end of this line.
Make sure that the rights are ok:

chmod o+w esp8266heap.csv
chown www-data:www-data esp8266heap.csv.

Setting up the code for receiving data:

Now the esp8266 will call a file named espdata.php passing through the query string the heap size: http://server_ip_name/espdata.php?heap=12345. This php file will get the heap query parameter and write it on the esp8266heap.csv file along with a time stamp. Here’s the code:

<?php
$now = new DateTime();
parse_str( html_entity_decode( $_SERVER['QUERY_STRING']) , $out);

if ( array_key_exists( 'heap' , $out ) ) {

echo "\n\r\n\rValor: " . $out['heap'];
$handle = fopen("/var/www/data/esp8266heap.csv", "a");

fwrite( $handle , $now->format('Y-m-d H:i:s') . " , " . $out['heap'] . "\n" );
fclose ( $handle ) ;

}
?>

Changing the esp8266 code:

My ping.lua file (check my older posts), that is called every minute looks like this now:

conn=net.createConnection(net.TCP, false)
conn:on("receive", function(conn, payload) print("Get done.", payload) end )
conn:connect(80,"xx.xx.xx.xx")
conn:send("GET /esp8266data.php?heap=" .. node.heap() .." HTTP/1.1\r\nHost: xx.xx.xx.xx\r\n" .. "Connection: keep-alive\r\nAccept: */*\r\n\r\n")

And that’s it. Every single minute an esp8266 timer triggers and calls the above ping.lua file that makes the espdata.php store the current heap size in the esp8266heap.csv file.

Graphing the data:

For graphing the data I’m using the dygraphs javascript library called remotely from my page that I’ve called esp8266graph.html:


<html>
<head>
<script src="//cdnjs.cloudflare.com/ajax/libs/dygraph/1.1.0/dygraph-combined.js"></script>
</head>
<body>
<div id="graphdiv2"
style="width:500px; height:300px;"></div>
<script type="text/javascript">
g2 = new Dygraph(
document.getElementById("graphdiv2"),
"data/esp8266heap.csv", // path to CSV file
{}          // options
);
</script>
</body>
</html>

And that’s it.

Seeing it all:

We can go now to http://server_ip_name/espgraph.html and see the esp8266 NodeMcu heap size variations in an interactive way. The final result show that with only the single¬†file, heap keeps constant, then drops, and keeps constant again, then drops again, and so on, until the esp8266 reboots, and then the heap recovers and the process starts again…

Selection_042

We can see where it is heading… The sharp change at around 3AM was the NodeMcu reboot.

Anyway, this hack with dygraphs can be used for anything e can imagine…