Cloud based continuous integration and delivery for IOT using PlatformIO

After finding out the PlatformIO for IoT development I started to read some of the Platformio documentation and also what other users have written about it.

One of the most interesting features of Platformio is that it supports to be used on a https://en.wikipedia.org/wiki/Continuous_integration process for any PlatformIO based project. This is important for using automated build systems for CI (Continuous Integration), and so, allows early detection of possible build problems. CI makes sense when several contributors/team are working on the same code repository, and we need to make sure that the project is able to build with all the team/contributors code inputs/changes. At the end, deliverables can be pushed to their destination:

CI

What Platformio CI enables is that for our IOT projects we can have automatic builds after code commits on the code repository (for example, GitHub). When the automatically build is triggered, PlatformIO is able to pull all dependencies and frameworks to build our project on the automated build system.

After the automatically build is triggered and the build is successful we can then deliver the output.

One of the most interesting examples of this workflow is the following post http://blog.squix.org/2016/06/esp8266-continuous-delivery-pipeline-push-to-production.html that shows the process of developing, committing code to the repository, triggering automatic builds and, at major releases, deploy firmware updates over the air (OTA) to the ESP8266. All of this using Platformio and 100% cloud infrastructure for IOT deployment.

Starting up with Platformio and TravisCI

Platformio supports several CI systems, and one of them is Travis.CI that integrates out of the box with GitHub. To enable Travis.CI on your GitHub projects/repository, just sign in on GitHub, and on (another browser tab) goto the TravisCI site and press the Sign in with GitHub button, accept the authorization requests, and select the repositories to enable for CI by going to your user profile.

Enabled Repository

After enabling the repositories, every commit will trigger an automatic build on the Travis CI platform. As a build status indicator we can add a IMG tag to the README.md file so we can have the build status without going to the Travis site, for example: https://travis-ci.org/fcgdam/RFGW_SensorNodes.svg?branch=master.

Setting up the build
Travis.CI will start the build process according to instructions defined on the hidden repository file .travis.yml that is unique and specific for each repository.

This Travis Documentation link explains in detail the logic and options behind the travis.yml file and build process.

Fortunately PlatformIO when initializing a project creates a sample travis.yml file.
Based on that sample, here is one of mine for compiling two sample Arduino projects on the same code repository:

language: python
python:
     - "2.7"

sudo: false
cache:
     directories:
         - "~/.platformio"

install:
     - pip install -U platformio

script:
     - cd RFGW && platformio run -e nanoatmega328
     - cd ../ATTINNY_MailMonitor_TX && platformio run

The tags language:, sudo: and cache: are not changed in this case.

The install: and script: tags are customized so that our project can be built.

On the install: tag, the first command is always the installation of the platformio tools, followed, if necessary, by installation of other dependencies. For example if our project depends on libraries from the Platformio library registry we can do the following:

install:
     - pip install -U platformio
     - platformio lib install 64

This, before building, will install first platformio, and then it will install the ArduinoJson (Id 64) library. We can add as much commands as we want prefixed by the dash character.
Also this is one way of doing it, but this means that we need to change .travis.yml file every time we add/remove libraries.

Another way is to add the library dependency on the project file platformio.ini like this:

[env:nanoatmega328]
platform = atmelavr
framework = arduino
board = nanoatmega328
lib_install= 64

And in this case all the dependencies are associated within the project file. But in this case the build commands are different.

So one example with multiple libraries could be as follow:

install:
     - pip install -U platformio
     - platformio lib install 64
     - platformio lib install 652
script:
     - platformio run

And this is the same as above:

install:
     - pip install -U platformio
script:
     - platformio run -e nanoatmega328

where nanoatmega328 is the environment configuration on platformio.ini file:

[env:nanoatmega328]
platform = atmelavr
framework = arduino
board = nanoatmega328
lib_install= 64,652

To end this topic, notice that we can have several builds on the same repository, just add several command lines to the script: tag:

script:
     - cd RFGW && platformio run -e nanoatmega328
     - cd ../ATTINNY_MailMonitor_TX && platformio run

I’m always using relative paths related to the project root in the above example.

Further information can be found on the Platformio Travis integration guide and on the Travis.CI site.

Continuous delivery

Since every commit to our repository triggers the Travis build process, we need now to distinguish between working commits and release commits so that on this last type, the build output is made available to be deployed to end devices/platforms for OTA updates (or not).

This can be easily achieved by using git tags and conditional deploy process that only works when a git tag is defined.

With this scheme the normal cycle of git add, commit and push will create a working commit that triggers as usual the CI build process but not the deployment phase of copying the build output (binaries, firmware) to the GitHub releases tab.

Creating a tag and a release can be done either by command line or by using the Github web interface, being this the easiest way of doing it.

But there are some pre-requisites to allow this to happen:
– Generate an OAuth personal GitHub token so that Travis can copy the output to the Releases GitHub tab.
– Encrypt the OAuth token with the travis command line tool.
– Change the .travis.yml file so that it deploys the build output to the releases tab only at tagged commits.

The GitHub token is generated by going to your Github Profile, selecting settings and then Personal Access tokens.
Press Generate new token, enter your password and add permissions to access your repositories.
The permissions should be at least full repo access:

GitHub Personal token permission

Make sure that at the end you copy the OAuth token, otherwise you must generate another one from the beginning.

The Github token must be kept secret at all times, and since we need to have it on the travis.yml file which can be read by everyone, we must make sure that we encrypt it in such a way that only Travis.CI can use it.
This is achieved by using the travis command line tool on our machine so we need to:

[pcortex@pcortex:RFGW_SensorNodes|master]$ travis encrypt GH_TOKEN="7d54...df5977" --add 

The GH_TOKEN is the name that must hold the OAuth token so that Travis can use it securely.

With the above command, the .travis.yml file is modified and the following entry is added:

env:
  global:
    secure: WqroI5PtWWm94svvau5G3LFz4PMBU...fY=

We can now add the final configuration to the Travis.CI configuration file, so that at tag releases, the build output is automatically added.

deploy:
  provider: releases
  api_key: ${GH_TOKEN}
  file:
    - $TRAVIS_BUILD_DIR/.pioenvs/nanoatmega328/firmware.hex
    - $TRAVIS_BUILD_DIR/.pioenvs/digispark-tiny/firmware.elf
  skip_cleanup: true
  on:
    tags: true  

The provider: tag defines that we want to deploy to GitHub Releases, and the api_key: tag contains the secure Oauth token to access GitHub.

The file: tag define which files we want to deploy, and in this case we use the $TRAVIS_BUILD_DIR environment variable to locate our build directory root. The skip_cleanup: will avoid cleaning all build outputs.

The on: tag is the most important because it conditionally defines that the deploy process only happens at tagged release.

So after this configuration, if we commit without tagging, the build is made, but no deploy to the Releases happens:

Travis Build without tagging

If we want to trigger a tagged commit we can do it purely on the command line:

[pcortex@pcortex:RFGW_SensorNodes|master]$ git tag -a v0.3 -m "Version 0.3"
[pcortex@pcortex:RFGW_SensorNodes|master]$ git push origin v0.3
....
To https://github.com/fcgdam/RFGW_SensorNodes.git
 * [new tag]         v0.3 -> v0.3

And that’s it: Automatic build process and release:

Tagged build process output

And the final result:

Tagged build output

We have now a tagged release with source code and binaries automatically created and packaged.

Deployment

At this point we have the deliverables for a release, and we should now distributed/deploy it. This is by itself another process that can be done through Cloud services or locally, it really depends of the end architecture.

The most important issue here is related to security: making sure that the correct build is delivered, was not changed in any way and reaches the intended devices.

One thought on “Cloud based continuous integration and delivery for IOT using PlatformIO

  1. Pingback: Cloud based deployment for IOT devices | Primal Cortex's Weblog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s