The challenge with TinyML is to extract the maximum performance/efficiency in the lowest footprint for AI workloads on microcontroller class hardware. The TinyML-CAM pipeline, developed by a team of machine learning researchers in Europe, demonstrates what is possible to achieve on relatively low-end camera hardware.
Specifically, they were able to achieve over 80 FPS image recognition on the sub-$10 ESP32-CAM board with the open source TinyML-CAM pipeline, which only takes about 1KB of RAM. It should work on other MCU boards with a camera and the training doesn’t seem complicated as we were told it takes about 30 minutes to complete a custom task.
The researchers note that solutions such as TensorFlow Lite for microcontrollers and Edge Impulse already enable the execution of ML workloads on MCU boards using neural networks (NN). However, they typically take up quite a bit of memory, between 50 and 500 kB of RAM, and take 100 to 600 ms to infer, forcing developers to choose NNs with low complexity and/or low accuracy.
So researchers decided to use more efficient non-NN algorithms (such as decision tree, SVM) using their Eloquent Arduino Libraries. alternative to TFLite for microcontrollers to design a TinyML-CAM pipeline that works in four stages:
- Data collection via a camera server (160×120 resolution) and an MjpegCollector that prompts users for a class name and collects image frames over a period of time until the user logs out
- Feature extraction in four steps:
- Convert collected images from RGB to grayscale
- Run the Histogram of Oriented Gradients (HOG) Feature Extractor to output feature vectors (uses lower resolution 40 x 30 images to speed up the task)
- Run the Pairplot Visualization to get a visual understanding of how informative the extracted features are
- Run the Uniform Manifold Approximation and Projection (UMAP) dimensionality reduction algorithm to take a feature vector and compress it to length 2
- Classifier training
- Porting to C++ – Convert the HOG component and the classifier (Pairplot) to C++ code, or more precisely to a header file (.h)
The header file can then be added to Arduino Sketch, the program compiled and run on the board. You’ll find the source code, instructions for installing it on ESP32 hardware, and a demo video on GitHub.
To test the solution, they train the ESP32-CAM to recognize a Raspberry Pi Pico, an Arduino Portenta H7, or a Wio terminal. The performance is great for this type of board and the footprint is really small, but the researchers note that the Portenta and Pi boards were often mislabeled during Pairplot analysis and expect this problem to be corrected by improving the quality of the data set.
In addition to viewing the source code on GitHub, you’ll also find additional information in a brief 4 page research paper.
Jean-Luc started CNX Software in 2010 as a part-time job before leaving his job as a software engineering manager and starting writing daily news and reviews full-time later in 2011.