What embedded AI means
Embedded AI means running machine learning models directly on an embedded device (a microcontroller, an application processor, or a dedicated accelerator) instead of sending data to a server for analysis. When the device is very small and power-constrained, such as a microcontroller with a few hundred kilobytes of RAM, the field is often called TinyML.
Embedded computer vision is one of the most demanding uses of embedded AI. A camera captures images, and the device itself works out what’s in them: is there a person, is this part defective, which gesture is the user making.
Almost all embedded AI is about inference, meaning running a model that was already trained elsewhere. Training normally happens on desktop GPUs or in the cloud, and the finished model is then optimised and deployed to the device.
Why run AI on the device instead of the cloud
- Latency: decisions happen in milliseconds without a network round trip, which matters for robotics, machine safety and interactive products.
- Privacy: raw audio and video can stay on the device. Only results, such as “person detected,” leave it.
- Bandwidth and cost: sending a count or an alert costs far less than streaming video to a server.
- Power: for battery devices, transmitting data by radio can cost more energy than processing it locally.
- Offline operation: the device keeps working when the network is down or doesn’t exist.
The trade-off is that embedded hardware has tight limits on memory, compute and power, so models have to be smaller and carefully optimised.
Embedded vision vs computer vision vs machine vision
These terms overlap, and people use them loosely:
- Computer vision is the broad field of getting computers to interpret images. It’s not tied to any kind of hardware.
- Embedded vision is computer vision running inside a self-contained device: a smart camera, a drone, a phone, a doorbell. The emphasis is on size, cost and power.
- Machine vision traditionally refers to industrial inspection and guidance systems: cameras, controlled lighting and processing on a production line, used for measuring parts, reading codes or finding defects. Older machine vision systems often used a separate PC. Many are now embedded smart cameras.
Types of vision systems
Vision systems are commonly grouped by what they capture and where processing happens:
- 1D (line-scan) systems read one line of pixels at a time and are used for continuous materials like paper, film or textiles moving past the sensor.
- 2D area-scan systems capture ordinary images. Most smart cameras and inspection systems work this way.
- 3D systems capture depth using stereo cameras, structured light, laser triangulation or time-of-flight sensors, and are used for robot picking and volume measurement.
- Smart cameras put the sensor and processor in one housing. PC-based systems send images to a separate computer.
- Thermal, multispectral and hyperspectral systems capture light outside the visible range for applications like agriculture, building inspection and food sorting.
Hardware tiers for embedded AI and vision
Hardware choice has the most influence on what’s possible. Roughly, the options fall into four tiers:
| Tier | Typical examples | Memory scale | Realistic AI/vision workloads |
|---|---|---|---|
| Microcontroller | Arm Cortex-M4/M7, ESP32-S3, nRF series | Hundreds of KB to a few MB of RAM | Keyword spotting, gesture and motion classification, anomaly detection on sensor data, very low-resolution image classification |
| MCU with NPU | Cortex-M55/M85 parts paired with Arm Ethos-U NPUs, and vendor MCUs with built-in neural accelerators | Similar to MCUs, with much faster inference | Person detection, simple object detection at low resolution, richer audio models, at lower energy per inference |
| Application processor + accelerator | Linux boards such as Raspberry Pi with an AI add-on, NXP i.MX parts with NPUs, Google Coral Edge TPU | Hundreds of MB to several GB | Real-time object detection, multi-camera inputs, pose estimation |
| Embedded GPU module | NVIDIA Jetson family | Several GB | Multiple video streams, larger detection and segmentation models, robotics perception |
Some image sensors now include processing on the sensor itself. The Sony IMX500, used in the Raspberry Pi AI Camera, is one example. This blurs the tiers further.
Moving up a tier generally means more capability but also higher cost, more power, longer boot times and more complex software (often a full Linux system instead of bare-metal firmware).
What can each tier realistically do?
A few honest expectations:
- Microcontrollers handle sensor data very well. Classifying accelerometer, vibration, audio or other time-series data is the TinyML sweet spot. Models are often tens of kilobytes.
- Vision on a plain MCU is possible but limited. Expect low-resolution images (often grayscale or something like 96×96 pixels), simple classification such as “person or no person,” and frame rates of a few frames per second or less, depending on the chip.
- NPUs change the picture for small vision. A microcontroller with a neural accelerator can run models that would be far too slow on the CPU alone, at a fraction of the energy.
- Real-time object detection on video usually calls for an application processor with an accelerator, or a GPU module.
If you’re starting out, Getting Started With TensorFlow Lite for Microcontrollers walks through running a first model on an MCU.
The deployment workflow
Most embedded AI projects follow the same steps, whatever the hardware:
- Define the task narrowly. “Detect whether a person is in frame” is far easier to deploy than “recognise everything in the room.”
- Collect representative data. Capture data with the actual sensor, in the actual environment: same camera, lighting, angles and noise. Data quality often matters more than model choice.
- Train a model on a workstation or in the cloud, typically with TensorFlow/Keras or PyTorch. Start with architectures designed for efficiency, such as MobileNet-style networks or small YOLO variants.
- Optimise the model:
- Quantization converts 32-bit floating-point weights and activations to 8-bit integers, cutting model size roughly fourfold and speeding up inference on hardware with integer acceleration.
- Pruning removes weights that contribute little.
- Knowledge distillation trains a small “student” model to mimic a larger one.
- Convert the model to the format your runtime needs: a
.tflitefile for LiteRT (formerly TensorFlow Lite), ONNX for ONNX Runtime, or a vendor-specific format for an NPU toolchain. - Integrate the model into firmware or an application, including the pre-processing (resizing, normalising) and post-processing (thresholds, non-maximum suppression for detection).
- Measure on the real device: accuracy, latency, memory use and power. Results on a desktop often don’t carry over.
- Plan for updates. Models may need retraining as conditions change, so the device should be able to receive new models, usually as part of OTA firmware updates.
Software for embedded computer vision
Commonly used tools include:
- OpenCV: the long-standing open-source library for classic image processing (filtering, edge detection, colour conversion, feature matching), with a DNN module for running neural networks. Widely used on Linux-class devices.
- LiteRT / TensorFlow Lite: Google’s on-device inference runtime, with a micro version for microcontrollers.
- ONNX Runtime: runs models exported from PyTorch and other frameworks in the ONNX format.
- Vendor toolchains: NVIDIA TensorRT for Jetson, Intel OpenVINO, NXP eIQ, ST’s Edge AI tools, Espressif’s ESP-DL, and others. These typically get the best performance from their own silicon.
- Edge Impulse: a hosted platform covering data collection, training and deployment to many embedded targets. It’s popular for getting a first model running quickly.
- OpenMV: small camera boards programmed in MicroPython for machine vision on microcontrollers.
Python or C++ for embedded AI?
In practice, both:
- Python dominates training, data preparation and experimentation. It’s also used for deployment on Linux-class devices (a Raspberry Pi running a Python inference script is very common), and MicroPython runs on some microcontrollers.
- C and C++ dominate deployment on microcontrollers and anywhere performance, memory and determinism matter. Runtimes like the microcontroller version of LiteRT are written in C++, and the firmware around them is usually C or C++.
A typical workflow is to train in Python and deploy in C++. Learning both is the realistic path for embedded AI work.
Do you need AI at all?
Not every vision problem needs a neural network. Classic image processing (thresholding, blob detection, edge detection, template matching) is often faster, more predictable, easier to debug, and works well under controlled lighting. Many industrial inspection tasks still use it.
Machine learning earns its place when the variation is too complex to describe with hand-written rules: natural scenes, varied lighting, deformable objects, many categories. A sensible approach is to try the simplest method that could work first.
Challenges
- Memory: model weights, the activation buffers used during inference, and camera frames all compete for limited RAM.
- Power: cameras and continuous inference drain batteries quickly. Low-power designs often use a cheap trigger (a motion sensor, say) to wake the vision pipeline only when needed.
- Accuracy in the field: lighting changes, dirty lenses, new product variants and seasonal differences can quietly degrade a model that tested well.
- Safety and determinism: ML outputs are probabilistic. In safety-related systems, ML usually needs to be bounded by conventional logic, monitoring and fallbacks, and it brings significant verification challenges.
- Privacy and regulation: cameras in public or workplace settings raise legal and ethical questions that depend on jurisdiction, even when processing stays local.
Is AI replacing embedded engineers?
AI is changing how embedded engineers work more than whether they’re needed. Coding assistants can draft drivers, boilerplate and tests, but embedded work depends heavily on hardware specifics, timing, debugging with real instruments, and safety judgement, which current AI tools handle poorly without close human review. Meanwhile, embedded AI itself creates work: someone has to fit models into tight memory and power budgets and make them reliable in the field. We look at this more in New Trends in Embedded Systems and Software Engineering.
FAQ
Can an Arduino run AI? Some can. Boards based on more capable microcontrollers, such as Cortex-M4F or ESP32-S3 parts, can run small TinyML models for audio, motion and very simple vision tasks. Classic 8-bit Arduino boards such as the Uno are generally too limited for meaningful neural networks.
What is the cheapest way to start with embedded vision? An ESP32-S3 camera board or an OpenMV camera is an inexpensive way to experiment with low-resolution vision on a microcontroller. A Raspberry Pi with a camera module gives a gentler Linux-based route with full OpenCV and Python support.
How does embedded AI relate to IoT? Edge AI lets connected devices send conclusions instead of raw data, which saves bandwidth, power and cloud cost. Where that processing sits in a connected system is covered in IoT System Architecture.