IoT device provisioning and tracking

If we are deploying several IoT devices, above a certain number it becomes a hard task to know which ones are alive and working, their location, and since they may have different functions/purposes, to keep track of their configuration. This last problem is exacerbated if we need to keep and maintain different code bases and firmwares for each device.

Device provisioning and some cleaver firmware code, can help to bring these issues under control. Device provisioning is nothing more than a fancy name for a central server that holds information for each device, it’s configuration and status and even might allow some form of monitoring. In advanced usage it might let an operator to push firmware or configuration updates to certain devices, without the need to physical get them.

So this post is about my simple device provisioning that allows to keep track of my ESP8266 devices, running the Sming firmware, and control, in this initial stage, some of it’s configuration.

The solution is based in three components:

  • NodeJs based REST server that holds information in a SQLite database
  • An Angular.Js based web application to managed the device configuration
  • Sming based firmware for the ESP8266

The REST server
The REST server is based in javascript running on a NodeJs server, which uses Express and Body-Parser to create a simple and quick REST server. The REST protocol uses the HTTP operations GET, POST and PUT to define operations like get data, create data or update data. Due to the Express modules, we can build a simple interface to a database that translates REST operations to database operations. The Body-Parser module allows to communicate purely using JSON, and so takes care of all the details of coding and decoding JSON data communicated over HTTP.
Finally there is one important thing, at least for the browser front-end that is the CORS protection. CORS, Cross Origin Resource Sharing is a security feature that browsers use to not connect to anything else other than the originating domain of the page that are showing. To allow our REST server to be called by a running application on a browser (in our case, our Angular.Js frontend), the server must explicitly allow this. So we also need a CORS module for our REST server.
Finally I’ve chosen a SQLITE database due to it’s simpler configuration. No need to setup servers…
To allow our REST server to run we need to install locally to our application the folowing modules: npm install express node-sqlite3 body-parser cors

// Define and call the packages necessary for the building the REST API.
var sqlite3    = require('sqlite3').verbose();
var db         = new sqlite3.Database('data/restDB.sqlite');
var express    = require('express');
var app        = express();
var bodyParser = require('body-parser');
var cors       = require('cors');

After the module instantiation, and note that the database is created in the data subdirectory, we can create if needed our tables and fill them up with default data if needed:

db.serialize(function() {  // All database operations will be executed in series.
    // some code here    
    db.run("CREATE TABLE IF NOT EXISTS devices ( deviceid TEXT , name TEXT , lastseen TEXT , ssid TEXT, ipaddr TEXT, cfgsn INTEGER, flags INTEGER, data TEXT, datec TEXT )");
    // some more code here
});

The db.serialize makes the database functions to run in series, one after each other.
The db.run will execute a SQL command, and in the above case the creation of the device provisioning table.

The schema is the following:

  • deviceid -> the device id as sent by the device. It should be unique. I use the MAC address of the ESP8266 that should be unique.
  • name -> a human friendly name for the device. It can be anything.
  • lastseen -> The device should periodically call the REST server to let us know that’s is alive and connected. The lastseen field is the last time that information was received.
  • ssid -> Shows the WiFi SSID to which the device is connected.
  • ipaddr -> Shows the device IPv4 address in use. Might not be reachable if the device is behind a NAT router.
  • cfsn -> Config serial number. To let the device know if the configuration it holds is older than the configuration on the table row.
  • flags -> For future use, but the idea is to allow pagination, and command/control/data separation.
  • data -> Configuration data for the device. In my case a JSON array of properties and values user defined.
  • datec -> The date that the record was create, which means when the device was first seen.

Now knowing what information we are using, we can build the REST server:

//Configure Express
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());
app.use(cors());

var router = express.Router();

// Root URI.
router.get('/', function(req, res) {
        res.json( {message: 'REST API' });
});

///// OUR CODE here

// Our base url is /api
app.use('/api', router);
app.listen(3000);

console.log("========================================");
console.log("Server started at http://localhost:3000/");

Excluding the ///// OUR CODE here this is the standard skeleton for the REST server that runs at port 3000 and hears for REST request on the /api base url.

For testing I recommend the FireFox plugin HTTPRequester that allows to simulate all the REST calls.

The devices REST interface

For the device provisioning I’ve implemented the REST end point devices which mean that we should call our REST server with the following URL’s: http://rest_server:3000/api/devices and http://rest_server:3000/api/devices/deviceid where deviceid is the device generated id.

For the first URL, so far I only implement the GET verb, which returns all registered devices:

router.route('/devices')
        .get ( function(req, res) {
           console.log("Devices list");
           db.all("SELECT ROWID, deviceid, name, lastseen, ssid, ipaddr, cfgsn, flags, data, datec  FROM devices",
              function( err , row ) {
                if (err) {
                  res.json( { status:'NOK', message: 'ERROR selecting data'} );
                } else {            
                  console.log('API: GET devices -> (' + row.length + ') records -> ' + JSON.stringify(row));
                   res.json( row );
                }                  
        });
      })

The function db.all returns an empty array if no results, or an array with row set. The row set is returned directly as a JSON array object to the calling application/device.

Selection_237

For the last URL, the implemented verbs are GET, to get the latest configuration and update the lastseen field, POST to register the device and/or returning the configuration, and PUT to update device information (used by the front end):

router.route('/devices/:id')
    .get( function(req, res) {
        var currdate = new Date();
        db.run("UPDATE devices SET lastseen='" + currdate + "' WHERE deviceid='" + req.params.id + "'");

        db.get("SELECT cfgsn, data FROM devices WHERE deviceid='"+req.params.id+"'",
          function( err , row ) {
            if (err) {    
              res.json( {status: 'NOK', message: 'Device get information failed' } );
            } else {
              if ( (typeof row == 'undefined') ) { 
                res.json( {status: 'NOK', message: 'Device NOT Registered' } );
              } else {
                res.json( row );
              }
            }
        });
    })    
    .put( function(req, res) {
        db.run("UPDATE devices SET name='"+req.body.name+"', cfgsn="+req.body.cfgsn.toString()+", data='"+req.body.data+"' WHERE deviceid='"+req.params.id+"'");  
        res.json( {status: 'OK', message: 'Device Updated' } );
    })    
    .post( function(req, res) {
        console.log("DEVICES -> POST Device request received: " + req.params.id);
        console.log("Body: " + JSON.stringify(req.body) );
        console.log("IP address: " + req.body.ipaddr );
        console.log("SSID: " + req.body.ssid );

        db.get("SELECT ROWID , deviceid, name, lastseen, ssid, ipaddr, cfgsn, flags, data, datec FROM devices WHERE deviceid='"+req.params.id+"'",
          function( err , row ) {
            if (err) {
              res.json( {status: 'NOK', message: 'Device Registration failed' } );
            } else {
              if ( (typeof row == 'undefined') ) {    // Device is new... if row is undefined (no results)
                var deviceid  = req.params.id;
                var name      = req.params.id;
                var lastseen  = new Date();
                var ssid      = req.body.ssid;
                var ipaddr    = req.body.ipaddr;
                var cfgsn     = "1";
                var flags     = "0";
                var data      = "";
                var datec     = lastseen;
                db.run("INSERT INTO devices ('deviceid','name','lastseen','ssid','ipaddr','cfgsn','flags','data','datec') VALUES ('" + deviceid + "', '" + name + "', '" + lastseen + "', '" + ssid + "', '" + ipaddr + "', " + cfgsn + ", " + flags + ", '" + data + "', '" + datec + "')");
                res.json( {status: 'OK', message: 'Device Registed' } );
             } else {
                res.json( row ); // Device already registered
             }
          }
       });
    })

The above REST verbs, namely the POST and PUT need a request body, to pass data from the source to the database.

In case of the POST verb the, body should be the following JSON: { “ipaddr”:”x.x.x.x”, “ssid”:”wifissid” }. For example:

Selection_238

We save the above code on a file, let it be restserver.js for example, and start our server with node restserver.js.

On the front end then we can have now a list of devices that are registered:

Selection_239

And selecting the device we can change the human readable name, and add configuration properties:

Selection_240

The front end, based on Angular Js isn’t completly ready, since it’s main purpose is to graph data from Sparkfuns Phant server.

Sming ESP8266 Code

To make this complete, we only need to start coding the ESP8266. I’ve abandoned NodeMcu and Lua language, due to several factors. Sming is a great firmware and already has support for a lot of devices and also includes a JSON library.

The sample Sming code is as simple as this:

#include <user_config.h>
#include <SmingCore/SmingCore.h>

#ifndef WIFI_SSID
    #define WIFI_SSID "defineme" // Put you SSID and Password here
    #define WIFI_PWD "passwd"
#endif

HttpClient provServer;  // Provisioning server client
String deviceid;

void onProvSent(HttpClient &client, bool successful )
{
    char cbuf[2048];
    StaticJsonBuffer<2048> jsonBuffer;
        
    if (successful)
        Serial.println("Registration request sent sucessufully");
    else
        Serial.println("Registration request has failed");
        
    String response = client.getResponseString();
    Serial.println("Server response: '" + response + "'"); 
        response.toCharArray( cbuf , 2048 );      
        JsonObject& config = jsonBuffer.parseObject( cbuf );
        
        if ( !config.success() ) {
            Serial.println("Error decoding JSON.");
        } else {
            Serial.println("Received config: " + config.toJsonString() );
        
            const char* name = config["name"];
            const char* datec= config["datec"];
            
            Serial.println ("Device Name: " + String(name) );
            Serial.println ("Date: " + String(datec) );
            // Process the JSON Array with the configuration data...

        }
}

void registerDevice ()
{
        String devIP;
        StaticJsonBuffer<256> jsonBuffer;
                
        devIP = WifiStation.getIP().toString();
        
        Serial.println("Sending device ID " + deviceid + " to provisioning server..." );
        Serial.println("Device IP : " + devIP );      
        JsonObject& config = jsonBuffer.createObject();
 
        config.addCopy("ipaddr", devIP );
        config.add("ssid"  , WIFI_SSID );
        
        provServer.setRequestContentType("application/json");
        provServer.setPostBody(config.toJsonString());
        provServer.downloadString("http://192.168.1.17:3000/api/devices/" + deviceid , onProvSent );
}

// Will be called when WiFi station was connected to AP
void connectOk()
{
    Serial.println("I'm CONNECTED");
        
        // Let's get the Mac address, that should be unique...
        deviceid = WifiStation.getMAC();
        deviceid.toUpperCase();
        Serial.println("Device ID: " + deviceid );
        
        // Register Device on the provisioning Server
        registerDevice();

    // Start send data loop
    // procTimer.initializeMs(25 * 1000, sendData).start(); // every 25 seconds
}

// Will be called when WiFi station timeout was reached
void connectFail()
{
    Serial.println("I'm NOT CONNECTED. Need help :(");

    // Start soft access point
    WifiAccessPoint.enable(true);
    WifiAccessPoint.config("CONFIG ME PLEEEEASE...", "", AUTH_OPEN);

    // .. some you code for configuration ..
}

void init()
{
    Serial.begin(SERIAL_BAUD_RATE); // 115200 by default
    Serial.systemDebugOutput(false); // Disable debug output to serial

    WifiStation.config(WIFI_SSID, WIFI_PWD);
    WifiStation.enable(true);
    WifiAccessPoint.enable(false);

    // Run our method when station was connected to AP (or not connected)
    WifiStation.waitConnection(connectOk, 20, connectFail); // We recommend 20+ seconds for connection timeout at start
}

And that’s it. After startup, the device after connecting to WiFi, makes a request to the provisioning server. If the request is from a new device it registers, if not it receives it’s configuration data. From this point we can do anything, namely pooling periodically the provisioning server to check for new configuration (by comparing the cfgsn hold by the device with the received one).

Advertisements

One thought on “IoT device provisioning and tracking

  1. Pingback: SparkFun Phant server data stream graphing | Primal Cortex's Weblog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s