Author	Dave Glover, Microsoft Cloud Developer Advocate
Solution	Creating an image recognition solution with Azure IoT Edge and Azure Cognitive Services
Documentation	README
Platform	Azure IoT Edge
Documentation	Azure IoT Edge, Azure Custom Vision, Azure Speech Services, Azure Functions on Edge, Azure Stream Analytics, Azure Machine Learning Services
Video Training	Enable edge intelligence with Azure IoT Edge
Programming Language	Python
Date	As at April 2019

Raspberry Pi 3A+ running Azure IoT Edge Image Classifier

• Parts Required
• Quick Installation Guide for Raspberry Pi
• Solution Overview
• Azure IoT Edge in Action
• Solution Architectural Considerations
• Creating the Fruit Classification Model
• Exporting an Azure Custom Vision Model
• Azure Speech Services
• Understanding the Project Structure
• Building the Solution
• Deploying the Solution
• Monitoring the Solution on the IoT Edge Device
• Monitoring the Solution from the Azure IoT Edge Blade

Image Classification with Azure IoT Edge

There are lots of applications for image recognition but what I had in mind when developing this application was a solution for vision impaired people scanning fruit and vegetables at a self-service checkout.

Parts Required

1. Raspberry Pi 3B or better, USB Camera, and a Speaker.

   Note, the solution will run on a Raspberry Pi 3A+, it has enough processing power, but the device is limited to 512MB RAM. I would recommend a Raspberry Pi 3B+ as it has 1GB of RAM and is faster than the older 3B model. Azure IoT Edge requires an ARM32v7 or better processor. It will not run on the ARM32v6 processor found in the Raspberry Pi Zero.

2. Alternatively, you can run the solution on desktop Linux - such as Ubuntu 18.04. This solution requires USB camera pass through into a Docker container as well as Azure IoT Edge support. So for now, that is Linux.

Quick Installation Guide for Raspberry Pi

If you do not want to download and build the solution you can use the prebuilt Azure IoT Edge configuration from my GitHub repository and use the associated Docker images.

1. Set up Raspbian Stretch Lite on Raspberry Pi. Be sure to configure the correct Country Code in your wpa_supplicant.conf file.
2. If you don’t already have an Azure account then sign up for a free Azure account. If you are a student then sign up for an Azure for Students account, no credit card required.
3. Follow these instructions to create an Azure IoT Hub, and an Azure IoT Edge device.
4. Install Azure IoT Edge runtime on Raspberry Pi
5. Download the deployment configuration file that describes the Azure IoT Edge Modules and Routes for this solution. Open the deployment.arm32v7.json link and save the deployment.arm32v7.json in a known location on your computer.
6. Install the Azure CLI and the IoT extension for Azure CLI command line tools. For more information, see Deploy Azure IoT Edge modules with Azure CLI
7. Open a command line console/terminal and change directory to the location where you saved the deployment.arm32v7.json file.
8. Finally, from the command line run the following command, be sure to substitute [device id] and the [hub name] values.

az iot edge set-modules --device-id [device id] --hub-name [hub name] --content deployment.arm32v7.json

1. The modules will now start to deploy to your Raspberry Pi, the Raspberry Pi green activity LED will flicker until the deployment completes. Approximately 1.5 GB of Dockers modules will be downloaded and decompressed on the Raspberry Pi. This is a one off operation.

Solution Overview

The system identifies the item scanned against a pre-trained machine learning model, tells the person what they have just scanned, then sends a record of the transaction to a central inventory system.

The solution runs on Azure IoT Edge and consists of a number of services.

1. The Camera Capture Module handles scanning items using a camera. It then calls the Image Classification module to identify the item, a call is then made to the “Text to Speech” module to convert item label to speech, and the name of the item scanned is played on the attached speaker.

2. The Image Classification Module runs a Tensorflow machine learning model that has been trained with images of fruit. It handles classifying the scanned items.

3. The Text to Speech Module converts the name of the item scanned from text to speech using Azure Speech Services.

4. A USB Camera is used to capture images of items to be bought.

5. A Speaker for text to speech playback.

6. Azure IoT Hub (Free tier) is used for managing, deploying, and reporting Azure IoT Edge devices running the solution.

7. Azure Speech Services (free tier) is used to generate very natural speech telling the shopper what they have just scanned.

8. Azure Custom Vision service was used to build the fruit model used for image classification.

What is Azure IoT Edge

The solution is built on Azure IoT Edge which is part of the Azure IoT Hub service and is used to define, secure and deploy a solution to an edge device. It also provides cloud-based central monitoring and reporting of the edge device.

The main components for an IoT Edge solution are:-

1. The IoT Edge Runtime which is installed on the local edge device and consists of two main components. The IoT Edge “hub”, responsible for communications, and the IoT Edge “agent”, responsible for running and monitoring modules on the edge device.

2. Modules. Modules are the unit of deployment. Modules are docker images pulled from a registry such as the Azure Container Registry, or Docker Hub. Modules can be custom developed, built as Azure Functions, or as exported services from Azure Custom Vision, Azure Machine Learning, or Azure Stream Analytics.

3. Routes. Routes define message paths between modules and with Azure IoT Hub.

4. Properties. You can set “desired” properties for a module from Azure IoT Hub. For example, you might want to set a threshold property for a temperature alert.

5. Create Options. Create Options tell the Docker runtime what options to start the module with. For example, you may wish to open ports for REST APIs or debugging ports, define paths to devices such as a USB Camera, set environment variables, or enable privilege mode for certain hardware operations. For more information see the Docker API documentation.

6. Deployment Manifest. The Deployment Manifest pulls everything together and tells the Azure IoT Edge runtime what modules to deploy, from where, plus what message routes to set up, and what create options to start each module with.

Azure IoT Edge in Action

Solution Architectural Considerations

So, with that overview of Azure IoT Edge here were my considerations and constraints for the solution.

1. The solution should scale from a Raspberry Pi (running Raspbian Linux) on ARM32v7, to my desktop development environment, to an industrial capable IoT Edge device such as those found in the Certified IoT Edge Catalog.

2. The solution needs camera input, I used a USB Webcam for image capture as it was supported across all target devices.

3. The camera capture module needed Docker USB device pass-through (not supported by Docker on Windows) so that plus targeting Raspberry Pi meant that I need to target Azure IoT Edge on Linux.

4. I wanted my developer experience to mirror the devices I was targeting plus I needed Docker support for the USB webcam, so I developed the solution on my Ubuntu 18.04 developer desktop. See my Ubuntu for Azure Developers guide.

   As a workaround, if your development device is locked to Windows then use Ubuntu in Virtual Box which allows USB device pass-through which you can then pass-through to Docker in the Virtual Machine. A bit convoluted but it does work.

Azure Services

Creating the Fruit Classification Model

The Azure Custom Vision service is a simple way to create an image classification machine learning model without having to be a data science or machine learning expert. You simply upload multiple collections of labelled images. For example, you could upload a collection of banana images and label them as ‘banana’.

To create your own classification model read How to build a classifier with Custom Vision for more information. It is important to have a good variety of labelled images so be sure to read How to improve your classifier.

Exporting an Azure Custom Vision Model

This “Image Classification” module includes a simple fruit classification model that was exported from Azure Custom Vision. For more information read how to Export your model for use with mobile devices. It is important to select one of the “compact” domains from the project settings page otherwise you will not be able to export the model.

Follow these steps to export your Custom Vision project model.

1. From the Performance tab of your Custom Vision project click Export.

   

2. Select Dockerfile from the list of available options

   

3. Then select the Linux version of the Dockerfile.

   

4. Download the docker file and unzip and you have a ready-made Docker solution with a Python Flask REST API. This was how I created the Azure IoT Edge Image Classification module in this solution. Too easy:)

Azure Speech Services

Azure Speech Services supports both “speech to text” and “text to speech”. For this solution, I’m using the text to speech (F0) free tier which is limited to 5 million characters per month. You will need to add the Speech service using the Azure Portal and “Grab your key” from the service.

Open the deployment.template.json file and update the BingKey with the key you copied from the Azure Speech service.

How to install, build and deploy the solution

1. Clone this GitHub repository.

   git clone https://github.com/gloveboxes/Creating-an-image-recognition-solution-with-Azure-IoT-Edge-and-Azure-Cognitive-Services.git
   

2. Install the Azure IoT Edge runtime on your Linux desktop or device (eg Raspberry Pi).

   Follow the instructions to Deploy your first IoT Edge module to a Linux x64 device.

3. Install the following software development tools.

   1. Visual Studio Code
   2. Plus, the following Visual Studio Code Extensions
      • Azure IoT Edge
      • JSON Tools useful for changing the “Create Options” for a module.
   3. Docker Community Edition on your development machine

4. With Visual Studio Code, open the IoT Edge solution you cloned from GitHub to your developer desktop.

Understanding the Project Structure

The following describes the highlighted sections of the project.

1. There are two modules: CameraCaptureOpenCV and ImageClassifierService.

2. The module.json file defines the Docker build process, the module version, and your docker registry. Updating the version number, pushing the updated module to an image registry, and updating the deployment manifest for an edge device triggers the Azure IoT Edge runtime to pull down the new module to the edge device.

3. The deployment.template.json file is used by the build process. It defines what modules to build, what message routes to set up, and what version of the IoT Edge runtime to run.

4. The deployment.json file is generated from the deployment.template.json and is the Deployment Manifest

5. The version.py in the project root folder is a helper app you can run on your development machine that updates the version number of each module. Useful as a change in the version number is what triggers Azure IoT Edge runtime to pull the updated module and it is easy to forget to change the module version numbers:)

Building the Solution

You need to ensure the image you plan to build matches the target processor architecture specified in the deployment.template.json file.

1. Specify your Docker repository in the module.json file for each module. If you are using a supported Linux Azure IoT Edge Distribution, such as Ubuntu 18.04 as your development machine and you have Azure IoT Edge installed locally then I strongly recommend setting up a local Docker Registry. It will significantly speed up your development, deployment and test cycle.

   To set up a local Docker Registry for prototyping and testing purposes.

docker run -d -p 5000:5000 --restart always --name registry registry:2

1. If pushing the image to a local Docker repository the specify localhost:5000.

   "repository": "localhost:5000/camera-capture-opencv"
   

2. Confirm processor architecture you plan to build for. From the Visual Studio Code bottom bar click the currently selected processor architecture, then from the popup select the desired processor architecture.

3. Next, Build and Push the solution to Docker by right mouse clicking the deployment.template.json file and select “Build and Push IoT Edge Solution”. The first build will be slow as Docker needs to pull the base layers to your local machine. If you are cross compiling to arm32v7 then the first build will be very slow as OpenCV and Python requirements need to be compiled. On a fast Intel i7-8750H processor cross compiling this solution will take approximately 40 minutes.

   

Deploying the Solution

When the Docker Build and Push process has completed select the Azure IoT Hub device you want to deploy the solution to. Right mouse click the deployment.json file found in the config folder and select the target device from the drop-down list.

Monitoring the Solution on the IoT Edge Device

Once the solution has been deployed you can monitor it on the IoT Edge device itself using the iotedge list command.

iotedge list

Monitoring the Solution from the Azure IoT Edge Blade

You can monitor the state of the Azure IoT Edge module from the Azure IoT Hub blade on the Azure Portal.

Click on the device from the Azure IoT Edge blade to view more details about the modules running on the device.

Done!

When the solution is finally deployed to the IoT Edge device the system will start telling you what items it thinks have been scanned.

Congratulations you have deployed your first Azure IoT Edge Solution!