Docker container web interface – Portainer and Riot-OS Development

This post is a follow up of starting up with RIOT-OS. To be able to develop with RIOT-OS an easy (and easier) way to do so is just to install docker and web UI docker interface Portainer to control docker.

So we will install Docker, Portainer, and finally the RIOT-OS building environment.

Installing Docker and Portainer, is an initial stepping stone for using the dockerized development environment for RIOT-OS, since I don’t want to install all the development environments in my machine.

Installing Docker:
On Arch-Linux is as simple as installing the Docker package using pacman, enabling the services and rebooting.
Basically we need to run, as root the following commands:

pacman -S docker
systemctl enable docker.service
reboot

After rebooting the following command should return some information

docker info

A sample output is:

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 9
Server Version: 18.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
...
...
...

Installing Portainer
Installing the Docker Portainer Web UI is as simple as:

docker pull portainer/portainer

To run Portainer a set of complete instructions on this page, but basically on the simplest way is:

$ docker volume create portainer_data
$ docker run -d -p 9000:9000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer

We can now check if the docker image is up:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS                    NAMES
7a38ae7fc922        portainer/portainer   "/portainer"        4 seconds ago       Up 3 seconds        0.0.0.0:9000->9000/tcp   portainer

Since I have already ran the Portainer container, the initial setting up steps when accessing the URL HTTP://localhost:9000 do not appear, but we need to choose:

  1. A set of credentials to use as de administrator for portainer
  2. The local machine registry to connect to the local docker containers.

1- At initial access we define an user and password:

Portainer Credentials

2- Then we connect to our local docker instance:
Portainer Local Docker

Press Connect and then we can now access our Docker instance from Portainer:
Portainer Main Screen

Pressing the Local Docker Connection we can now manage our docker resources.

Installing the build environment for RIOT-OS
We can do it by two ways:

From the command line:

docker pull riot/riotbuild

or use Portainer:

This container is very big, so we need to wait some time for the container image download. The command line shows in greater detail the download process.

After the image is downloaded, we can follow these instructions for building our apps using the docker container as the build environment.

After the image is installed:

To use is is as simple as going the the examples directory and do:

make BUILD_IN_DOCKER=1

From this we are now able to build based RIOT-OS applications for several targets, including the ESP8266/ESP32.

As we can see we even don’t need to have a running container, just the image.

Advertisements

Arch Linux with full encrypted disk

So, I’ve bought a new light weight laptop, an HP Envy 13.3 1.2Kg I7, where I wiped out Windows 10 and installed Arch Linux.

Just for security reasons I’ve decided to do a full disk encryption install, including boot.

There are several instructions on the Web, including videos on youtube in how to do it, and so on this gist I have my instructions for the installation, based off course in other gists and instructions.

In this post is the configuration instructions that worked for me and also to allow better find from google/bing/ddg.

# WARNING: WORK IN PROGRESS, USE THESE STEPS WITH CAUTION. IT WILL CLEAR ALL DISK DATA!!
I REALLY recommend first to use a Virtual Box machine with EFI support enabled to test everything before doing it on a real machine.

# Arch installation on a HP ENVY 13 inch laptop (ah0006np part number: 16GB Ram, 512GB SSD)

OBJECTIVE:
Install Arch Linux with encrypted boot, root and swap filesystems and boot from UEFI, completly dumping Windows on the process.
No dual boot.
Windows, if necessary will be run on a Virtual Machine and re-use the Windows key that came with the laptop.

The configuration will be LVM on LUKS. Also a major difference from other tutorials is that the boot partition is also encrypted, and not a standard partition.

# Results do far:

– Disk encryption ok. GRUB boots slow (20s). Otherwise works fine.
– Wireless works ootb, but errors on dmesg output from time to time when high traffic.
– Sound and microfone works ok
– Webcam does work but needs configuration: See below at the end.
– Keyboard special keys work fine (brightness, Sound, Mute), including keyboard background lights, but F6 sound Mute Led does not work.
– Some screen corruption with the Intel Driver either SNA or UXA. Nouveau crashes, nvidia driver didn’t work. To be checked -> Issue with QT 5 and Konsole/Kate applications, not a Intel Driver issue.
– KDE SDDM doesn’t recover well if screen DPMS is activated. I’ve disable it so far to solve it.
– Suspend/resume works fine.
– Battery time so far, around 4/5 hours.


# Desired disk layout:

+---------------+----------------+----------------+----------------+
|ESP partition: |Boot partition: |Volume 1:       |Volume 2:       |
|               |                |                |                |
|/boot/efi       |/boot           |root            |swap            |
|               |                |                |                |
|               |                |/dev/vg0/root   |/dev/vg0/swap   |
|/dev/sda1      |/dev/sda2       +----------------+----------------+
|unencrypted    |LUKS encrypted  |/dev/sda3 encrypted LVM on LUKS  |
+---------------+----------------+---------------------------------+

The final result is to have an Arch Linux Installation with full disk encryption and with a basic set of applications such as the KDE Plasma Desktop.

These instructions have several sources, namely:
https://grez911.github.io/cryptoarch.html
and this WordPress post.

The installation process on this guide is for the Arch Linux installation onto an HP Envy 13, 16GB RAM with 512MB ssd laptop. This laptop comes with Windows 10 Home installed, and as far as my model goes, it comes with an Intel WiFi board and a WD Sandisk SN520 512GB NVME SSD.

The official Arch installation guide contains more details that you should refer to during this installation process.
That guide resides at: https://wiki.archlinux.org/index.php/Installation_Guide

## Boot from image

Download the archlinux-\*.iso image from https://www.archlinux.org/download/ and its GnuPG signature.
Use gpg –verify to ensure your archlinux-\*.iso is exactly what the Arch developers intended. For example
at the time of installation:

$ gpg --verify archlinux-2017.10.01-x86_64.iso.sig
gpg: Signature made Sun 01 Oct 2017 07:29:43 AM CEST using RSA key ID 9741E8AC
gpg: Good signature from "Pierre Schmitz "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 4AA4 767B BC9C 4B1D 18AE  28B7 7F2D 434B 9741 E8AC

Currently the Arch ISO is archlinux-2018.11.01-x86_64.iso.

Burn the archlinux-\*.iso to a 1+ Gb USB stick. You can use the dd command, unetbootin or Etcher.

Connect the USB stick to the usb port and power on/cycle the machine to boot.
If your USB stick fails to boot, ensure that Secure Boot is disabled in your UEFI configuration.

Note: To access the BIOS on the Envy Laptop, turn on the laptop and press several times the ESC key or the F10 key to access the BIOS while the screen is black.
First I moved the boot order to have the USB boot at the top.
Then we need to disable the secure boot option and press F10 to save. Confirm saving it.

Attention now: There is a confirmation screen to really commit the secure boot option change. Enter the requested code and save.

After booting up:

Set your keymap only if not you are not using the default English keyboard.

$ loadkeys pt-latin1

We can now, if required backup the HP recovery partition, that I suppose is the Windows Install Media.

# Connect to the Internet.

Execute the wifi-menu command and select a Wifi network. On this HP Envy, the wireless card (Intel) was detected with no issues.

Check with the “ip a” command if there is network connection.

## Prepare your hard disk

In the next steps we will create necessary partitions and encrypt the main partition.

Find the correct block device

$ lsblk

In my case the correct block device (the NVME SSD of my laptop) is ‘nvme0n1’. (Depends on the machine)

Create and size partitions appropriate to your goals using gdisk.

$ gdisk /dev/nvme0n1

Press p to show the partitions.

In my case I have a 260Mb EFI partition, a 16MB Reserved Microsoft Partition, a 460GB partition and a 980MB and another 15GB partition.

From this point on, everything that is to be done, will destroy the disk data.

# Delete all partitions on disk

Use the d command to delete all partitions. Use d, then partition number. Repeat for all partitions

Press o to create the GPT.

Create three partitions: One for the EFI, one for boot and the other will be used to have the Arch Linux installation. To create a partition, press n:

1. Partition 1 = 512 MB EFI partition (Hex code EF00). Initial Sector: ; End: 512M; Type: EF00
2. Partition 2 = 1GB Boot partition (Hex code 8300)
3. Partition 3 = Size it to the last sector of your drive. (default) (Hex code 8E00 – Linux LVM Partition)

Review your partitions with the ‘p’ command.
Write your gdisk changes with ‘w’.

Check again the names with the blkid command to know the partitions name:

1. EFI: /dev/nvme0n1p1
2. BOOT: /dev/nvme0n1p2
3. Arch: /dev/nvme0n1p3

# Create filesystems
The EFI filesystem must be FAT32:

$ mkfs.vfat -F 32 /dev/nvme0n1p1

The other filesystems are to be encrypted.

(optional) Before creating the partitions we can use the command

cryptsetup benchmark 

to see how fast the different encryption algoritms are.

# Encrypted /boot partition

$ cryptsetup -c aes-xts-plain64 -h sha512 -s 512 --use-random luksFormat /dev/nvme0n1p2
$ cryptsetup open /dev/nvme0n1p2 cryptboot
$ mkfs.ext4 /dev/mapper/cryptboot

The first command will ask for the disk passphrase. Do not forget it!.

ATTENTION:
The first crypsetup command will set the LUKS with default iter-time parameters, which may or may not make grub to boot slow (around 20s). If this is not fine add the following parameter: –iter-time=5000 (This will affect security, so use a large key phrase)

The last command will create a /dev/mapper/cryptboot device.
We can check that it was created with the command ls /dev/mapper

# Create encrypted LUKS device for the LVM

cryptsetup -c aes-xts-plain64 -h sha512 -s 512 --use-random luksFormat /dev/nvme0n1p3
cryptsetup open /dev/nvme0n1p3 cryptlvm


## Create encrypted LVM partitions

These steps will create the required root partition and an optional partition for swap.
Modify this structure only if you need additional, separate partitions. The sizes used below are only suggestions.
The VG and LV labels ‘ArchVG, root and swap’ can be changed to anything memorable to you. Use your labels consistently, below!

$ pvcreate /dev/mapper/cryptlvm
$ vgcreate ArchVG /dev/mapper/cryptlvm
$ lvcreate -L +16G ArchVG -n swap
$ lvcreate -l +100%FREE ArchVG -n root

Again, we can see on /dev/mapper if the logical volumes where created.

## Create filesystems on your encrypted partitions

$ mkswap /dev/mapper/ArchVG-swap
$ mkfs.ext4 /dev/mapper/cryptboot
$ mkfs.ext4 /dev/mapper/ArchVG-root

Mount the new system

mount /dev/mapper/ArchVG-root /mnt
swapon /dev/mapper/ArchVG-swap
mkdir /mnt/boot
mount /dev/mapper/cryptboot /mnt/boot
mkdir /mnt/boot/efi
mount /dev/nvme0n1p1 /mnt/boot/efi

# Install the Arch system

This installation command provides a decent set of basic system programs which will also support WiFi when initally booting into your Arch system.

At this point we need to have a network connection. Since the HP only has Wifi connection, we need to setup the WiFi connection. Other alternative is to use an Ethernet USB dongle that is recognized by the Arch boot iso.
Also, if you are behind a proxy, you can set the http_proxy and https_proxy variables to access the internet.

(Optional) Use reflector to speedup download (credit goes to u/momasf)

Change COUNTRY to (surprise) your country name.

pacman -Sy reflector
reflector --country 'COUNTRY' --age 12 --protocol https --sort rate --save /etc/pacman.d/mirrorlist

I won’t install base-dev here to save time at the installation.

$ pacstrap /mnt base grub-efi-x86_64 efibootmgr dialog wpa_supplicant vim

# Create and review FSTAB
The -U option pulls in all the correct UUIDs for your mounted filesystems.

 $ genfstab -U /mnt >> /mnt/etc/fstab
 $ nano /mnt/etc/fstab  # Check your fstab carefully, and modify it, if required.
 

Enter the newly installed system

$ arch-chroot /mnt /bin/bash

Set the system clock, you can replace UTC with your actual timezone

$ ln -fs /usr/share/zoneinfo/Europe/Lisbon /etc/localtime
$ hwclock --systohc --utc

Assign your hostname

$ echo mylaptop > /etc/hostname

My requirements for the locale are:
– Metric system
– 24h time format
– dd/mm/yyyy date format
– Portuguese language
– A4 paper size
– But all help, error messages are in English

The *pt_PT.UTF-8* plus *en_US.UTF-8* locale meets those requirements. To set up this locale:

– In /etc/locale.gen

en_US.UTF-8 UTF-8
pt_PT.UTF-8 UTF-8

– In /etc/locale.conf, you should **only** have this line:

LANG=en_US.UTF-8

We will change other settings on Bash profile.

Now run:

$ locale-gen

Create a new file vconsole.conf so that the console keymap is correctly set at boot. Create the file and add the following line:

KEYMAP=pt-latin1

Set your root password

$ passwd

Create a User, assign appropriate Group membership, and set a User password.

$ useradd -m -G audio,games,log,lp,optical,power,scanner,storage,video,wheel -s /bin/bash memyselfandi
$ passwd memyselfandi

Configure mkinitcpio with the correct HOOKS required for your initrd image

$ nano /etc/mkinitcpio.conf

Use this HOOKS statement: (I’ve moved keyboard before keymap, encrypt and so on…)

HOOKS="base udev autodetect modconf block keyboard keymap encrypt lvm2 resume filesystems fsck"

Generate your initrd image

mkinitcpio -p linux

## Install and configure Grub-EFI
Since we have the boot partition INSIDE the encrypted disk, we need to add the following option to the Grub options:

Edit the file /etc/default/grub and uncomment the following line:

GRUB_ENABLE_CRYPTODISK=y

And then we can install Grub, which will create an EFI entry named ArchLinux

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ArchLinux

Edit /etc/default/grub so it includes a statement like this:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/nvme0n1p3:cryptlvm resume=/dev/mapper/ArchVG-swap i915.enable_guc=3"

I’ve also added the i915 configuration line.

Other way of doing it is to use UUID:

blkid /dev/nvme0n1p3 -s UUID -o value

And use the UUID outputed on this command line:

GRUB_CMDLINE_LINUX="cryptdevice=UUID=55994-XXXX-xXXXX-XXXXX:cryptlvm resume=/dev/mapper/ArchVG-swap"

Generate Your Final Grub Configuration:

$ grub-mkconfig -o /boot/grub/grub.cfg

At this point there are some errors regarding failing to connect to lvmetad, which are normal and can be ignored.

# Mounting /boot without password request
Grub will ask for passwords to access the encrypted volumes. We can do this automatically:

dd bs=512 count=8 if=/dev/urandom of=/etc/key
chmod 400 /etc/key
cryptsetup luksAddKey /dev/nvme0n1p2 /etc/key
echo "cryptboot /dev/nvme0n1p2 /etc/key luks" >> /etc/crypttab

# Mounting root LVM without password prompt

dd bs=512 count=8 if=/dev/urandom of=/crypto_keyfile.bin
chmod 000 /crypto_keyfile.bin
cryptsetup luksAddKey /dev/nvme0n1p3 /crypto_keyfile.bin
sed -i 's\^FILES=.*\FILES="/crypto_keyfile.bin"\g' /etc/mkinitcpio.conf
mkinitcpio -p linux
chmod 600 /boot/initramfs-linux*

The mkinitcpio.conf FILES line will look like:

FILES="/crypto_keyfile.bin"

# Enable Intel microcode CPU updates (if you use Intel processor, of course)

pacman -S intel-ucode
grub-mkconfig -o /boot/grub/grub.cfg

# Check EFI Boot Manager
Check that the EFI Boot manager has the ArchLinux entry:

$ efibootmgr

For example if ArchLinux entry is Boot0003, check if on the boot order, 0003 is on the head of the list.
If not change the order with:

$ efibootmg -o 0003,0002,0001,0000

Exit Your New Arch System

$ exit

Unmount all partitions

$ umount -R /mnt
$ swapoff -a

Reboot and Enjoy Your Encrypted Arch Linux System!

reboot

___

# Setup system

We need again to connect to the internet, so run again the *wifi-menu*.

Install bash completion for reduced typing effort and other packages if necessary:

$ pacman -S sudo bash-completion base-devel git

To be able to use sudo from your normal user add wheel to sudoers.

$ EDITOR=nano visudo

Uncomment the line

%wheel      ALL=(ALL) ALL

From this point on, it really depends of what need there is for the machine.

# Making the webcam to work.
The webcam id appears at the lsusb output:

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 8087:0a2a Intel Corp. 
Bus 001 Device 002: ID 04ca:7090 Lite-On Technology Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The webcam is the Bus 001:002 device: ID 04ca:7090.
Add the following rule at /etc/udev/rules.d

KERNEL=="video[0-9]*", SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{idVendor}=="04ca", ATTRS{idProduct}=="7090", SYMLINK+="video-cam"

Load the module to activate the webcam:

modprobe uvcvideo

The /dev/video0 and 1 devices should appear.

A simple start with the RiotOS operating system for IoT devices

The RiotOS operating system is an Open Source operating system that targets different embedded platforms while allowing to use the same code base for some aspects of IoT development: Connectivity, protocols and security. It also supports other key aspects such as threads, Inter process communication, synchronization support on the kernel side. It also supports several communications protocols, suche BLE, 6LoWPAN, Lorawan, and so on.

At the target level for the RiotOS operating system, RiotOS supports a wide scope of different platforms, including a special native platform. What does this means? It means that within certain limitations, it is entirely possible to develop an IoT application that runs where the code is developed, this includes our PC and any SBC such as the RaspberryPI. This support opens a new window of opportunities where it is possible to integrate different classes of devices, such as Arduino, STM32 and RPI while leveraging the knowledge on the same supporting code base.

How to start:

This is a very quick start instructions for building a native demo application with network support. As usual all the instructions apply to a Linux Operating system, in my case Arch Linux.

Clone the RiotOS repository:

Just run:

cd /opt
git clone https://github.com/RIOT-OS/RIOT.git

I’ve chosen the /opt directory for the sake of example.

Compile one the example application:

All example applications are under the example applications.
In each directory, there is a Makefile file with a specific content.

For example on the examples/default there is Makefile that targets the native environment.

Of key interest is the line in this file that defines the target board:

BOARD ?= native

If the BOARD variable is not defined on the running environment it will use the native board.

We just need now to run the make command:

[pcortex@pcortex:default|master]$ make
Building application "default" for "native" with MCU "native".

"make" -C /opt/RIOT/boards/native
"make" -C /opt/RIOT/boards/native/drivers
"make" -C /opt/RIOT/core
"make" -C /opt/RIOT/cpu/native
...
...
...
"make" -C /opt/RIOT/sys/shell
"make" -C /opt/RIOT/sys/shell/commands
   text    data     bss     dec     hex filename
  88189    1104   72088  161381   27665 /opt/RIOT/examples/default/bin/native/default.elf

We can see that the compilation output was placed at /opt/RIOT/examples/default/bin/native/default.elf.

But if we ran the default.elf executable, we get:

[pcortex@pcortex:default|master]$ bin/native/default.elf 
usage: bin/native/default.elf  [-i ] [-d] [-e|-E] [-o] [-c ]
 help: bin/native/default.elf -h

Options:
    -h, --help
        print this help message
    -i , --id=
        specify instance id (set by config module)
    -s , --seed=
        specify srandom(3) seed (/dev/random is used instead of random(3) if
        the option is omitted)
    -d, --daemonize
        daemonize native instance
    -e, --stderr-pipe
        redirect stderr to file
    -E, --stderr-noredirect
        do not redirect stderr (i.e. leave sterr unchanged despite
        daemon/socket io)
    -o, --stdout-pipe
        redirect stdout to file (/tmp/riot.stdout.PID) when not attached
        to socket
    -c , --uart-tty=
        specify TTY device for UART. This argument can be used multiple
        times (up to UART_NUMOF)

A little side note, under bin there is a native directory but if we target the compilation to a different platform, a new directory under bin will be created to that platform.

Returning to result to the above command, we notice that the command requires a TAP (terminal interface point) interface. The fact is that the RiotOS code to access the network resources of the hosting operating system doesn’t access the network interface directly, but does it through the TAP interface. We can create only one TAP interface, or several TAP interfaces, which in such case, means that we can have several RiotOS applications using an TAP based network to communicate between themselves and also with the external network. Neat!

RiotOS offers a script to the TAP intefaces, but before running the script, just make sure if any Linux operating system Kernel was done recently, a reboot migh be needed to the interfaces be successfully created.

So:
[pcortex@pcortex:RIOT|master]$ ip a

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp6s0:  mtu 4000 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:24:8c:02:dd:7e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.68/24 brd 192.168.1.255 scope global noprefixroute enp6s0
       valid_lft forever preferred_lft forever
    inet6 fe80::224:8cff:fe02:dd7e/64 scope link 
       valid_lft forever preferred_lft forever

Now we run the tapsetup script that creates the TAP interfaces. This script when called without any parameters, it creates two TAP interfaces. If we pass the -c argument with a number, for example -c 4, it will create 4 TAP interfaces. The -d parameter deletes all interface that where created.

[pcortex@pcortex:tapsetup|master]$ pwd
/opt/RIOT/dist/tools/tapsetup                
[pcortex@pcortex:tapsetup|master]$ sudo ./tapsetup 
creating tapbr0
creating tap0
creating tap1
[pcortex@pcortex:tapsetup|master]$ ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp6s0:  mtu 4000 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:24:8c:02:dd:7e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.68/24 brd 192.168.1.255 scope global noprefixroute enp6s0
       valid_lft forever preferred_lft forever
    inet6 fe80::224:8cff:fe02:dd7e/64 scope link 
       valid_lft forever preferred_lft forever
8: tapbr0:  mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 2e:bd:d8:88:64:55 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2cbd:d8ff:fe88:6455/64 scope link 
       valid_lft forever preferred_lft forever
9: tap0:  mtu 1500 qdisc fq_codel master tapbr0 state DOWN group default qlen 1000
    link/ether 9a:a0:bc:fa:c7:61 brd ff:ff:ff:ff:ff:ff
10: tap1:  mtu 1500 qdisc fq_codel master tapbr0 state DOWN group default qlen 1000
    link/ether 2e:bd:d8:88:64:55 brd ff:ff:ff:ff:ff:ff

We can see that both tap0 and tap1 where created and are down.

Within two different windows we can run the default.elf with each interface now:

Window one: > sudo ./default.elf tap0
Window two: > sudo ./default.elf tap1

The two instances will start communicate and the tap interface have addresses and are up now.

11: tapbr0:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:db:a6:77:3b:e5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::cdb:a6ff:fe77:3be5/64 scope link 
       valid_lft forever preferred_lft forever
12: tap0:  mtu 1500 qdisc fq_codel master tapbr0 state UP group default qlen 1000
    link/ether 0e:db:a6:77:3b:e5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::cdb:a6ff:fe77:3be5/64 scope link 
       valid_lft forever preferred_lft forever
13: tap1:  mtu 1500 qdisc fq_codel master tapbr0 state UP group default qlen 1000
    link/ether be:65:4f:4d:a0:ed brd ff:ff:ff:ff:ff:ff
    inet6 fe80::bc65:4fff:fe4d:a0ed/64 scope link 
       valid_lft forever preferred_lft forever

Conclusion:
The above is a simple example, but RiotOS offers several examples including COAP, MQTT-SN and OpenThread examples.

The next step is to test RiotOS on ESP8266 and ESP32.

MakeBlock STEM mbot Robot – Using nodeJS to control mbot through BLE

A few weeks ago I’ve bought a mbot robot out of curiosity (also as a gift), since they became available at a nearby major electronic retailer and cheaper than buying them online.

The mbot is a robot chassis with two wheels, some external and onboard sensors, including an external ultrasonic sensor, and all this supported on a custom version of Arduino 328 board whish incorporates a motor driver, battery charger and so on. The version that I’ve bought also came with a LED Matrix display where it is possible to draw faces, text and numbers (mbot Face version).

The mbot robot can be controlled or used by either some Android (and IOS) mobile applications, or by using the Scratch programming environment. MakeBlock has a specific mbot version for Scratch called mblock that supports a set of new programming blocks to control the robot. The use of Scratch and mbot makes it ideal combination for teaching kids about programming and robotics.

We caan communicate/interface with mbot either by using an USB cable, or either by Low Power Bluetooth (the mbot BLE version) or through a 2.4GHz radio (the mbot 2.4GHz version). The 2.4Ghz version is more adequate for a classroom environment, since each robot is automatically bounded by radio to the 2.4GHz USB computer stick radio controller, which basically makes it plug and go and no need to fiddle with BLE discovery and bounding.

Anyway, the version that I have is the BLE one, and this post is about how to use NodeJS with the BLE Noble Library to communicate with mbot when using the factory firmware.

Requisites:

To make this work we need to have some requisites first:

Since mbot uses BLE, the computer must support also BLE. In my case I’m using the CSR 4 BLE dongle available on eBay, Ali and so on, to have BLE support on my computer.

The mbot must be loaded with the factory firmware so the this code can work. This is off course just for testing this code.
The factory firmware can be loaded either by using the mblock program when connected by USB cable, or by using the Arduino IDE.

The mbot BLE module is connected to the serial pins of the onboard arduino, so while the factory firmware has a specific interface, nothing stops us from replacing it with our own code and interface. For now we just keep the factory interface that is based on messages that start with 0xFF 0x55 ….

The code was tested on Linux, and it works fine. No idea if it works on windows…

As far it goes today, the NodeJS Bleno library doesn’t work with the latest node version 10, so we need to use this with a previous version of NodeJS. I’m using NodeJS V8, and also use the NodeJS Version Manager to have several versions of NodeJS active and available.

The BLE interface:
Using the Nordic Connect mobile application, we turn on the mbot, and on the application we start the BLE scan:

A device named Makeblock_LE should appear. We can connect to it and see the published services and characteristics:

There are two known services, and two unknown services. After some testing writing data to those services the service ffe1 is the service that connects to the mbot arduino serial port, and the service ffe4 I have no idea what it is for. Probably for controlling something on the BLE module itself.

The characteristics that the service ffe1 service exposes are:

As we can see, on is for reading data: ffe2 and it supports notification. This means we are warned when data is available so we can read it. The other characteristic is ffe3 that is for writing.

Basically if we connect to the Makeblock_LE BLE device, use the ffe1 service and write on the ffe3 characteristic we can control the robot. Data from the robot is automatically sent to us if we have notifications enabled on the ffe2 characteristic.

The mbot protocol:

There is one post that explains the protocol structure to communicate with the mbot.

Basically every command begins with 0xff 0x55 and then a set of bytes to control something.

The responses follow the same principle of starting with 0xff 0x55 and can return several values types.

An easier way to see what to send is to use mblock, program a scratch example in Arduino mode, and on the mblock serial monitor see what is sent to the robot.

My GitHub code source has some command examples for sending to mbot, namely to control the WS2812 RGB leds, the buzzer, the Led Matrix and to read data from the ultrasonic sensor.

How to use it:

Download the code from here MBot_BLE.

git clone https://github.com/fcgdam/mbot_ble

Make sure that you are using NodeJS version 8:

node -v
v8.11.3

If using Node V10, you can try to install the modules since in a future date from this post, the issues with Noble and NodeJS V10 might be solved.

Install the modules dependencies:

npm install

The code to access the BLE device needs root access, or check how to use Noble without root access:

sudo node mbotble.js

If the Ultrasonic distance sensor is connected to port 3, distance data is shown on the terminal.

That’s it!

Sample output:

The sample output for the mbotble.js when running as root on the RPI 3:

root@firefly:/home/pi/BLEMbot# node blembot.js 
- Bluetooth state change
  - Start scanning...
! Found device with local name: Makeblock_LE
! Mbot robot found! 
  - Stopped scanning...
- Connecting to Makeblock_LE [001010F13480]
! Connected to 001010F13480
! mbot BLE service found!
! mbot READ BLE characteristic found.
! mbot WRITE BLE characteristic found.
- End scanning BLE characteristics.
! Subscribed for mbot read notifications
Reading the ultrasound sensor data...
> mbot data received: "ff550002cb3db9410d0a"
Distance: 23.15517234802246
Reading the ultrasound sensor data...
> mbot data received: "ff5500020000bc410d0a"
Distance: 23.5
Reading the ultrasound sensor data...
> mbot data received: "ff5500027c1ab9410d0a"
Distance: 23.13793182373047
Reading the ultrasound sensor data...
> mbot data received: "ff5500028db0c0410d0a"
Distance: 24.086206436157227
Reading the ultrasound sensor data...
> mbot data received: "ff550002ddd398400d0a"
Distance: 4.775862216949463
Reading the ultrasound sensor data...
> mbot data received: "ff5500024f23d8410d0a"
Distance: 27.017240524291992
...
...
...

Skype for Linux – Using corporate proxy

Since, unfortunately, I some times need to use Skype on Linux and because I’m behind a corporate proxy server, Skype doesn’t work or accept, at least on Arch Linux running KDE Plasma desktop, the system proxy settings.

Setting the http_proxy/https_proxy variables have no effect on the Electron (I think…) Skype based app.

Anyway the solution for this is quite simple: Just install ProxyChains.

[pcortex@desktop:~]$ sudo pacman -S proxychains-ng

After installing, edit the file /etc/proxychains.conf and under the [ProxyList] add your proxy:

http  10.1.1.1  3128  my_proxy_username  my_proxy_password

Save the file, and now just run skype with the following command:

[fpcortex@desktop:~]$ proxychains skypeforlinux

Making now the test call works.

Using the Blackmagic probe with Netbeans

Using the Blackmagic debug probe (BMP) with Netbeans is very similar when doing the same but with using Openocd.

The initial steps are the same to use the BMP probe and Openocd. First we set up the ARM toolchain or other, as needed. We also need the gdbserver plugin for Netbeans, and so, we also need to install it. After these two initial steps, the remaining configuration and how to use the probe differs a bit differently from when using Openocd.

One of the main differences is that with Openocd, externally we launch an instance of Openocd and then use the Netbeans gdbserver plugin to connect to that openocd instance. For that we use the Netbeans menu Debug and Attach Debugger and use connection string extended-remote localhost:3333 and the selected project:

For using the BMP probe we don’t need no intermediary software since the code supporting the gdbserver plugin is already on the probe firmware, which means that the debugger can connect directly to the target. In fact due to this we do not need now to attach to a debugger and we can start using the Debug Project (CTRL-F5) to program/flash the code output (optional) and to start debugging the code running on the target device.

For this to happen we need to configure the Debug Command for the project (this must be done for each project if needed) and if wanted we can also configure the Run Command.
Before we are able to do that, we need first to create a command file for the gdb debugger that will pre-execute some commands, namely to attach to the targets probe, before starting the debugging session.

For that I’ve created a file named BMPgdbinit located, in my case, in /opt/ARM:

target extended-remote /dev/ttyACM0
monitor swdp_scan
att 1
load
start
run

This file is invoked by the ARM debugger toolchain (like this: arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit file.out) and it will connect to the Blackmagic probe, that should be located at /dev/ttyACM0, scan the SWD searching for the device, attaches to it, loads/programs the output file, namely the firmware, and starts it. From this point the gdbserver from Netbeans takes control and should stop at the first line of code or the first break point.

So to configure the run command we edit the Project Properties and modify the Run command:

Just define the Run Command as arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit “${OUTPUT_PATH}”

With this configuration we can now press F6 to flash our firmware (because the load command is on the BMPgdbinit file) and run it.

For debugging we need the following configuration:

We must define the Debug Command as ${OUTPUT_PATH} since this will be passed as the arm-none-eabi-gdb file parameter.
We also need to point to the correct location of the BMPgdbinit file.

We can now just press CTRL-F5 to launch a debug session, and again, the firmware will be loaded, and the code will stop at the first break point.

An example of using Netbeans to build a simple program targeting the nRF52832 (the blinky example from pcbreflux ) with an added variable and a watch point set:

That’s it….

Issues:

There are some issues with the above process…

Can’t stop the debugger. Pressing Pause does nothing…
This is a long standing bug apparently so the solution is on a external terminal window send a SIGINT signal to the gdb process:

[pcortex@pcortex:~]$ ps -ef | grep arm-none
pcortex 9792  9405  0 20:22 ?        00:00:00 /tmp/dlight_fdam/5fe1c993/0397644155/pty --dir /opt/nordic/nRF52832/blinky --no-pty /opt/ARM/gcc-arm-none-eabi-6_2-2016q4/bin/arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit -nx --interpreter mi -tty /dev/pts/7
pcortex 9794  9792  0 20:22 ttyACM0  00:00:00 /opt/ARM/gcc-arm-none-eabi-6_2-2016q4/bin/arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit -nx --interpreter mi -tty /dev/pts/7
 

Just send a SIGINT signal to the process associated to the Blackmagic probe:

kill -SIGINT 9794

The control is returned to Netbeans.

The debugger doesn’t seem to follow my code…
Just Clean and build all the project and select run so a clean version of the program is flashed.

I unplug and plug the probe a lot and the tty ports change..

Because of this the BMPgdbinit script can be broken because the probe port changes, for example, to /dev/ttyACM2.

The solution is to create a rules file:

Mine, for Arch Linux is as follows:

# Black Magic Probe
# there are two connections, one for GDB and one for uart debugging
  SUBSYSTEM=="tty", ATTRS{interface}=="Black Magic GDB Server", SYMLINK+="ttyBMP"
  SUBSYSTEM=="tty", ATTRS{interface}=="Black Magic UART Port", SYMLINK+="ttyBMPUART"

This file is named 99-blackmagic.rules located at /etc/udev/rules.d.

We now need to change the port used on the BMPgdbinit file from /dev/ttyACM0 to /dev/ttyBMP. Note that after plugging in the probe to the USB port, it takes a while (2 to 3 seconds) to the ports appear.

This just doesn’t work…

Yes, some times the behaviour seems strange… I’ve worked around some of the issues by flashing again the softdevice and the program mannually from the GDB command line.

Also if something like this happens:

It is due to the fact that the Netbeans environment variables $OUTPUT_PATH is empty… The solution is to explicitly state the outputfile on the RUN and Debug commands instead of using the ENV var. For example change this:

arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit "${OUTPUT_PATH}"

to this

arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit /opt/Projects/nRF/proj1/program.out

or to this:

arm-none-eabi-gdb -x /opt/ARM/BMPgdbinit "${PROJECT_DIR}/file.out"

Final note:
As a final note, some other errors might creep up like:

  • Program is not being run: Check the command line of arm-none-eabi-gdb to see if the BMPgdbinit file is correctly provided.
  • Don’t know how to run. Try \”help target\”.: Same issue. Check if the run and debug commands are correctly defined.
  • Sometimes my target isn’t detected.: Adding an extra monitor swdp_scan line to the BMPgdbinit file might help.

Building a Blackmagic Debug Probe

The Black Magic debug probe is in-application debugger, which allows IDE’s like NetBeans, Eclipse or others to connect directly to the hardware without using bridges like for example Openocd. The Blackmagic Probe software is open source and available on this Github link.

There are several posts/instructions on the internet regarding how to load the Black Magic firmware either on ST-Link hardware debuggers, ST-Link clones or using the “blue pill” STM32F103C8T6 based board.

There are several possible issues with this board:

  1. First they are quite cheap – not an issue
  2. It seems on some versions there is a problem with the USB connector regarding a resistor and bad contacts. In my version I had no issues. It just worked fine.
  3. The on board chip is the STM32F103C8T6 which has 64Kb of flash, but some claim that it can have 128Kb. In my case all the boards that I have, the flash size is 64Kb.

This last issue is a show stopper at the current release of the Black Magic firmware since it won’t fit on the 64Kb flash of the ST32F103C8T6. We must cut things from the firmware to be able to flash it on the STM32F103C8T6 board. But we will see how to do that.

Downloading and compiling the firmware:
This step is just straight forward, just clone the Blacmagic repository and compile. Since we are compiling to ST32F103 board we will assume it is a ST-Link clone.

[pcortex@pcortex:opt]$ git clone https://github.com/blacksphere/blackmagic
Cloning into 'blackmagic'...
remote: Counting objects: 5029, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 5029 (delta 1), reused 2 (delta 0), pack-reused 5020
Receiving objects: 100% (5029/5029), 2.01 MiB | 1.38 MiB/s, done.
Resolving deltas: 100% (3558/3558), done.
[pcortex@pcortex:opt]$ cd blackmagic/
README.md  Makefile  libopencm3/  HACKING  driver/  COPYING  scripts/  src/  upgrade/
[pcortex@pcortex:blackmagic|master]$ make
Initialising git submodules...
Submodule 'libopencm3' (https://github.com/libopencm3/libopencm3.git) registered for path 'libopencm3'
Cloning into '/opt/lixo/blackmagic/libopencm3'...
Submodule path 'libopencm3': checked out '67242de60dec0227739cd549e8a78e1a3c15dbf5'
  GENHDR  include/libopencm3/efm32/efm32gg/irq.json
  GENHDR  include/libopencm3/efm32/efm32g/irq.json
  GENHDR  include/libopencm3/efm32/efm32lg/irq.json
  GENHDR  include/libopencm3/efm32/efm32tg/irq.json
  GENHDR  include/libopencm3/stm32/f2/irq.json
...
...
...

Now we must target the firmware to our blue pill board, and so compiling the firmware as it was for a ST-Link clone:

[pcortex@pcortex:blackmagic|master]$ cd src
[pcortex@pcortex:src|master]$ make clean
 CLEAN
[pcortex@pcortex:src|master]$ make PROBE_HOST=stlink
GIT     include/version.h
  CC      target/adiv5.c
  CC      target/adiv5_jtagdp.c
  CC      target/adiv5_swdp.c
  CC      command.c
  CC      target/cortexa.c
  CC      target/cortexm.c
...
...
...
  CC      platforms/stlink/dfu_upgrade.c
  LD      dfu_upgrade
  OBJCOPY dfu_upgrade.bin
  OBJCOPY dfu_upgrade.hex

Done! On the src there is now at least two files:

  1. blackmagic_dfu.bin – The Device Firmware Upgrade loader.
  2. blackmagic.bin – The Blackmagic firmware.

Checking the file sizes we can see that the blackmagic.bin file (as of today) 57K, and the bootloader is 6.8K.

Flashing the firmware:
Flashing the firmware can be done by several forms:

  1. Using the STM32 embedded loader through a serial port
  2. Using the SWD connection, ST-Link probe and Openocd
  3. Using the SWD connection, ST-Link probe and Texane/st-link/st-util
  4. Using the DFU protocol, but it depends if the board already has a DFU firmware loaded.

Most of the posts on the internet regarding this step use the first aproach, namely using the STM32 boot loader and serial port. This needs a serial to USB converter to be connected and the STM32 boot pins to be modified.

Anyway, I’m flashing the firmware using Openocd and/or st-link utils through a ST-Link debug probe:

The blue pill pinout is as follows where the SWD connector is oposite the USB port:

From the ST-Link probe we need to connect the following pins:

  • Probe SWCLK Blue Pill DCLK
  • Probe SWDIO Blue Pill DIO
  • Probe GND Blue Pill GND

Regarding the power, I advise not to connect it and power up the blue pill board through the USB connector. This will allow to have access to both boards through USB without any possible power clashes.

We can now flash the firmware either with Texane St-Link programs or the AUR Arch Linux package stlink-git or using Openocd.

For using Openocd, create the following file and name it for example bluepill.cfg:

set CHIPNAME STM32F103C8T6

source [find interface/stlink-v2.cfg]
transport select hla_swd
source [find target/stm32f1x.cfg]
set WORKAREASIZE 0x2000

Also make sure that the ST-Link probe is detected so both st-link utils and openocd can work:

[pcortex@pcortex:src|master]$ lsusb
Bus 002 Device 003: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 003: ID 046d:c52f Logitech, Inc. Unifying Receiver
Bus 008 Device 002: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
...
Bus 003 Device 008: ID 0483:3748 STMicroelectronics ST-LINK/V2

We can now flash our board with the following command using Openocd:

openocd -f ../bluepill.cfg -c ‘init_reset halt; program blackmagic_dfu.bin 0x8000000 verify; reset;exit’

[pcortex@pcortex:src|master]$ openocd -f ../bluepill.cfg -c 'init_reset halt; program blackmagic_dfu.bin 0x8000000 verify; reset;exit'
Open On-Chip Debugger 0.10.0
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
0x2000
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v24 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.208372
Info : STM32F103C8T6.cpu: hardware has 6 breakpoints, 4 watchpoints
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x080017b4 msp: 0x20005000
** Programming Started **
auto erase enabled
Info : device id = 0x20036410
Info : flash size = 64kbytes
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000003a msp: 0x20005000
wrote 7168 bytes from file blackmagic_dfu.bin in 0.656939s (10.655 KiB/s)
** Programming Finished **
** Verify Started **
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x20005000
verified 6900 bytes in 0.187018s (36.030 KiB/s)
** Verified OK **
[pcortex@pcortex:src|master]$

Using the st-flash command as documented on STM as BMP

[pcortex@pcortex:src|master]$ st-flash write blackmagic_dfu.bin 0x8000000
st-flash 1.3.1
2017-06-13T20:26:01 INFO src/common.c: Loading device parameters....
2017-06-13T20:26:01 INFO src/common.c: Device connected is: F1 Medium-density device, id 0x20036410
2017-06-13T20:26:01 INFO src/common.c: SRAM size: 0x5000 bytes (20 KiB), Flash: 0x10000 bytes (64 KiB) in pages of 1024 bytes
2017-06-13T20:26:01 INFO src/common.c: Attempting to write 6888 (0x1ae8) bytes to stm32 address: 134217728 (0x8000000)
Flash page at addr: 0x08001800 erased
2017-06-13T20:26:01 INFO src/common.c: Finished erasing 7 pages of 1024 (0x400) bytes
2017-06-13T20:26:01 INFO src/common.c: Starting Flash write for VL/F0/F3 core id
2017-06-13T20:26:01 INFO src/flash_loader.c: Successfully loaded flash loader in sram
  6/6 pages written
2017-06-13T20:26:02 INFO src/common.c: Starting verification of write complete
2017-06-13T20:26:02 INFO src/common.c: Flash written and verified! jolly good!
[pcortex@pcortex:src|master]$

So far so good, but the above is just DFU firmware that allows to use DFU utils later to upgrade the firmware through the USB port without using any other tools.

If unplug and plug our blue pill board and check for the USB ports we have now:

[pcortex@pcortex:src|master]$ lsusb
Bus 002 Device 003: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 003: ID 046d:c52f Logitech, Inc. Unifying Receiver
Bus 008 Device 002: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
...
Bus 003 Device 012: ID 1d50:6017 OpenMoko, Inc. Black Magic Debug Probe (DFU)
Bus 003 Device 008: ID 0483:3748 STMicroelectronics ST-LINK/V2
...

And we can now also use the dfu-util command to upload the firmware.

To upload the firmware we will see that it fails due to the firmware size and the 64Kb flash size. It might not happen to you if your board has more than 64Kb of flash available.

With Openocd command:

openocd -f ../bluepill.cfg -c 'init_reset halt; program blackmagic.bin verify; reset;exit'    
Open On-Chip Debugger 0.10.0
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
0x2000
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v24 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.208372
Info : STM32F103C8T6.cpu: hardware has 6 breakpoints, 4 watchpoints
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x080017b4 msp: 0x20005000
** Programming Started **
auto erase enabled
Info : device id = 0x20036410
Info : flash size = 64kbytes
Warn : no flash bank found for address 0
wrote 0 bytes from file blackmagic.bin in 0.016043s (0.000 KiB/s)
** Programming Finished **
** Verify Started **
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x20005000
Error: checksum mismatch - attempting binary compare
diff 0 address 0x08010000. Was 0xff instead of 0x00
diff 1 address 0x08010001. Was 0xff instead of 0x00
diff 2 address 0x08010002. Was 0xff instead of 0x00
diff 3 address 0x08010003. Was 0xff instead of 0x00
diff 4 address 0x08010004. Was 0xff instead of 0x00
diff 5 address 0x08010005. Was 0xff instead of 0x00
diff 6 address 0x08010006. Was 0xff instead of 0x00

....
diff 123 address 0x0801007b. Was 0xff instead of 0x00
diff 124 address 0x0801007c. Was 0xff instead of 0x00
diff 125 address 0x0801007d. Was 0xff instead of 0x00
diff 126 address 0x0801007e. Was 0xff instead of 0x00
diff 127 address 0x0801007f. Was 0xff instead of 0x00
More than 128 errors, the rest are not printed.
embedded:startup.tcl:476: Error: ** Verify Failed **
in procedure 'program' 
in procedure 'program_error' called at file "embedded:startup.tcl", line 520
at file "embedded:startup.tcl", line 476

With the st-flash command:

[pcortex@pcortex:src|master]$ st-flash --reset write blackmagic.bin 0x8002000
st-flash 1.3.1
2017-06-13T20:37:27 INFO src/common.c: Loading device parameters....
2017-06-13T20:37:27 INFO src/common.c: Device connected is: F1 Medium-density device, id 0x20036410
2017-06-13T20:37:27 INFO src/common.c: SRAM size: 0x5000 bytes (20 KiB), Flash: 0x10000 bytes (64 KiB) in pages of 1024 bytes
2017-06-13T20:37:27 INFO src/common.c: Attempting to write 58084 (0xe2e4) bytes to stm32 address: 134225920 (0x8002000)
2017-06-13T20:37:27 ERROR src/common.c: addr too high
stlink_fwrite_flash() == -1
[pcortex@pcortex:src|master]$ 

And with the dfu util forcing the write above 64Kb:

[pcortex@pcortex:src|master]$ sudo dfu-util -a 0 -s 0x08002000:leave:force -D blackmagic.bin
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

dfu-util: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
Opening DFU capable USB device...
ID 1d50:6017
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #0 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 1024
DfuSe interface name: "Internal Flash   "
Downloading to address = 0x08002000, size = 58084
dfu-util: Last page at 0x080102e3 is not writeable
[pcortex@pcortex:src|master]$ 

So no luck with my board version, and as a solution we need to strip down the firmware.

Striping down the firmware:
To make the BMP firmware able to fit the 64Kb flash we have to cut things. In my case I’ve removed some processors support since my work is with the STM32 and nRF families:

Edit the file named cortexm.c located at target subdirectory, and delete the following lines around line 254 (or search by PROBE) to remove the unneeded support:

#define PROBE(x) \
        do { if ((x)(t)) return true; else target_check_error(t); } while (0)

        PROBE(stm32f1_probe);
        PROBE(stm32f4_probe);
        PROBE(stm32l0_probe);   /* STM32L0xx & STM32L1xx */
        PROBE(stm32l4_probe);
        PROBE(lpc11xx_probe);
        PROBE(lpc15xx_probe);
        PROBE(lpc43xx_probe);
        PROBE(sam3x_probe);
        PROBE(sam4l_probe);
        PROBE(nrf51_probe);
        PROBE(samd_probe);
        PROBE(lmi_probe);
        PROBE(kinetis_probe);
        PROBE(efm32_probe);

Still at the source directory we build again the firmware:

[pcortex@pcortex:src|master]$ make clean
[pcortex@pcortex:src|master]$ make PROBE_HOST=stlink
[pcortex@pcortex:src|master]$ ls -l blackmagic*
-rwxr-xr-x 1 pcortex users 753K jun 13 20:53 blackmagic*
-rwxr-xr-x 1 pcortex users  55K jun 13 20:53 blackmagic.bin*
-rwxr-xr-x 1 pcortex users 325K jun 13 20:53 blackmagic_dfu*
-rwxr-xr-x 1 pcortex users 6,8K jun 13 20:53 blackmagic_dfu.bin*
-rw-r--r-- 1 pcortex users  20K jun 13 20:53 blackmagic_dfu.hex
[pcortex@pcortex:src|master]$

We can seen now that the firmware was “slim down” around 2K, which is enough now to fit the 64Kb flash:

Using Openocd:

[pcortex@pcortex:src|master]$ openocd -f ../bluepill.cfg -c 'init_reset halt; program blackmagic.bin 0x8002000 verify; reset;exit'
Open On-Chip Debugger 0.10.0
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
0x2000
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v24 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.208372
Info : STM32F103C8T6.cpu: hardware has 6 breakpoints, 4 watchpoints
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x080017b4 msp: 0x20005000
** Programming Started **
auto erase enabled
Info : device id = 0x20036410
Info : flash size = 64kbytes
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000003a msp: 0x20005000
wrote 54272 bytes from file blackmagic.bin in 3.792976s (13.973 KiB/s)
** Programming Finished **
** Verify Started **
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000002e msp: 0x20005000
verified 53744 bytes in 0.854957s (61.388 KiB/s)
** Verified OK **

Using st-flash:

[pcortex@pcortex:src|master]$ st-flash --reset write blackmagic.bin 0x8002000
st-flash 1.3.1
2017-06-13T20:58:02 INFO src/common.c: Loading device parameters....
2017-06-13T20:58:02 INFO src/common.c: Device connected is: F1 Medium-density device, id 0x20036410
2017-06-13T20:58:02 INFO src/common.c: SRAM size: 0x5000 bytes (20 KiB), Flash: 0x10000 bytes (64 KiB) in pages of 1024 bytes
2017-06-13T20:58:02 INFO src/common.c: Attempting to write 55720 (0xd9a8) bytes to stm32 address: 134225920 (0x8002000)
Flash page at addr: 0x0800f800 erased
2017-06-13T20:58:05 INFO src/common.c: Finished erasing 55 pages of 1024 (0x400) bytes
2017-06-13T20:58:05 INFO src/common.c: Starting Flash write for VL/F0/F3 core id
2017-06-13T20:58:05 INFO src/flash_loader.c: Successfully loaded flash loader in sram
 54/54 pages written
2017-06-13T20:58:09 INFO src/common.c: Starting verification of write complete
2017-06-13T20:58:09 INFO src/common.c: Flash written and verified! jolly good!

Using the DFU firmware upload:

[pcortex@pcortex:src|master]$ sudo dfu-util -a 0 -s 0x08002000:leave:force -D blackmagic.bin
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

dfu-util: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
Opening DFU capable USB device...
ID 1d50:6017
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #0 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 1024
DfuSe interface name: "Internal Flash   "
Downloading to address = 0x08002000, size = 55720
Download        [=========================] 100%        55720 bytes
Download done.
File downloaded successfully
Transitioning to dfuMANIFEST state

And now if we unplug the blue pill board and connect it again we should have two new tty USB based ports:

[35014.003313] usb 3-2.4: new full-speed USB device number 20 using uhci_hcd
[35019.346373] cdc_acm 3-2.4:1.0: ttyACM0: USB ACM device
[35019.349364] cdc_acm 3-2.4:1.2: ttyACM1: USB ACM device

Success!

Starting using the BMP probe:

To start using our blue pill board as a Blackmagic probe we need the following information and configuration:

  1. Connect a LED (with a resistor…) to pin PA8 to show the probe activity.
  2. The target SWD pins are CLK at pin PA5 and DIO at PB14.
  3. The support for the USB to UART bridge (the second ttyACM1 port) is PA3 – RX and PA2 – TX

So we now can use the BM probe instead of a ST-Link probe, and connect the target board SWD pins to PA5 e PB14: CLK and DIO respectively.

The led will blink when the program is running or it will fast blink when loading a program into the target board.

More information on this video from no other than the builder of the Magic Probe to see more of the probe capabilities. I also recommend the Blackmagic probe wiki for further information.

A quick example with an nRF52 based board connected to the BMP by using the ARM GDB debugger arm-none-eabi-gdb directly:

(gdb) target extended-remote /dev/ttyACM0
Remote debugging using /dev/ttyACM0
(gdb) monitor swdp_scan
Target voltage: unknown
Available Targets:
No. Att Driver
 1      Nordic nRF52
(gdb) att 1
(gdb) monitor
Black Magic Probe (Firmware v1.6.1-25-gaaa7b0e-dirty) (Hardware Version 0)
Copyright (C) 2015  Black Sphere Technologies Ltd.
License GPLv3+: GNU GPL version 3 or later 


Interesting links:

Most of this post wouldn’t be possible without gathering information from around the web. Here is a set of information that might be useful to gather more or missing information from this post:

  1. Black Magic Probe (BMP) on ST-Link v2 clones
  2. STM Discovery as Black Magic Probe
  3. STM32Duino forum