Concept renderEdge AI means running a machine learning model on the device or a nearby gateway, instead of in the cloud. It cuts latency, keeps raw data local and works offline. The costs are power, memory and accuracy lost to smaller, quantized models. Many devices split the work: a wake word on the device, a large model in the cloud.
"Edge AI" covers a wide range of hardware, from a wake word detector on a microcontroller to a box with an accelerator next to a row of cameras. For a team building a connected device, the useful questions are narrow: which part of the AI work belongs on the device, which chip can run it, and what that costs in battery, memory and accuracy. This guide covers the tiers of edge AI hardware with vendor figures, the reasons to run models on the device, the costs, the common hybrid pattern and a decision checklist.
This is an engineering summary. Chip and runtime figures come from vendor product pages and documentation, and all sources were read on November 12, 2026. The parts named are examples. They are not recommendations for your design.
What is edge AI?
Edge AI is machine learning inference that runs close to where the data is produced. That can be the device itself, which people also call on-device AI, or a gateway or local computer near it. Cloud AI sends the data to a data center and gets the answer back.
In a connected device, edge AI usually means one of these jobs:
- Listening for a wake word or for speech at all (voice activity detection).
- Classifying motion: steps, sleep, a fall, a gesture.
- Cleaning up audio: noise suppression and echo cancellation.
- Recognizing objects or people in camera frames.
- Spotting anomalies in sensor data before anything is sent.
Training almost always happens in the cloud or on a workstation. Google lists on-device training as not supported in LiteRT for Microcontrollers (LiteRT for Microcontrollers, as of November 2026). The device runs a fixed model, and a new model arrives with a firmware or model update.
What kinds of edge AI devices and chips are there?
Edge AI hardware falls into four tiers. Each step up adds compute and memory, and costs power, board space and price. The rows describe the tiers. They do not describe a specific project.
| Tier | How it runs models | Typical jobs | Example |
|---|---|---|---|
| Microcontroller (TinyML) | Small models run on the CPU of a microcontroller, sometimes with vector instructions. | Wake word, voice activity, gesture and activity detection, simple sensor anomalies. | ESP32-S3: dual-core Xtensa LX7 at 240 MHz with vector instructions for neural network workloads (Espressif). |
| Microcontroller with a microNPU | A small neural processing unit next to a Cortex-M core runs the model layers. | Keyword spotting with more words, small vision models, sensor fusion at lower energy per inference. | Arm Ethos-U55 paired with Cortex-M55 or Cortex-M85 (Arm). |
| Application processor with an NPU | Linux on Cortex-A cores, with an NPU for models and often a Cortex-M core for always-on tasks. | Camera devices, local speech recognition, companions with a screen. | NXP i.MX 93: Cortex-A55 at up to 1.7 GHz, Cortex-M33 at 250 MHz and an Arm Ethos-U65 microNPU (NXP). |
| Edge box or gateway | A computer near the devices, with a dedicated accelerator, serves several sensors or cameras. | Multi-camera vision, local processing for a room or site. | Raspberry Pi AI HAT+ for Raspberry Pi 5: 26 TOPS or 13 TOPS (Raspberry Pi). |
Figures as of November 2026.
Microcontrollers
Google says LiteRT for Microcontrollers needs no operating system support, no standard C or C++ libraries and no dynamic memory allocation. It is written in C++ 17, needs a 32-bit platform, and has been tested on many Arm Cortex-M processors and ported to others, including the ESP32. Its listed limits are a limited set of devices, a low-level C++ API with manual memory management, no on-device training and a subset of TensorFlow operations (LiteRT for Microcontrollers, as of November 2026).
Espressif says the ESP32-S3 adds vector instructions that accelerate neural network and signal processing workloads, used through its ESP-NN and ESP-DSP libraries. It lists 512 KB of internal SRAM, with support for external octal SPI flash and PSRAM (ESP32-S3, as of November 2026). For more on this tier, see TinyML on microcontrollers.
MicroNPUs
Arm describes the Ethos-U55 as a microNPU for ML in deeply embedded systems. Arm states up to a 480 times uplift in ML performance over existing Cortex-M based systems when paired with the Cortex-M55, and up to 90% energy reduction, in an area of about 0.1 mm² (Arm Ethos-U55, as of November 2026). Those are Arm's best-case figures. Measure your own model on the real chip. For how NPUs work and how to read their TOPS figures, see what is an NPU in edge devices.
Application processors and edge boxes
NXP's i.MX 93 pairs one or two Cortex-A55 cores at 1.7 GHz with a Cortex-M33 at 250 MHz and an Arm Ethos-U65 microNPU, and NXP aims it at Linux-based edge applications (NXP i.MX 93, as of November 2026). Above that, accelerator add-ons bring tens of TOPS: Raspberry Pi lists its AI HAT+ for Raspberry Pi 5 with a Hailo-8 at 26 TOPS or a Hailo-8L at 13 TOPS (Raspberry Pi AI HAT+, as of November 2026). That tier suits fixed, powered installations more than wearables.
Why run AI on the device?
- Latency. A wake word or a fall must be detected at once. A cloud round trip adds network time that varies with the connection and fails when there is none.
- Privacy. Audio, images and health signals that never leave the device cannot leak from a server. A device that only sends results, such as "fall detected", collects less personal data.
- Offline operation. Trackers, wearables and companions lose connection in basements, on planes and in rural areas. On-device models keep core features working.
- Data cost and radio power. Sending raw sensor data or audio costs cellular data and radio energy. Sending a result costs little. See cellular data cost for an LLM device.
What does edge AI cost in power, memory and accuracy?
Running the model locally moves cost from the server bill to the device.
- Power. An always-on model runs all day. Its energy per inference, times inferences per day, comes straight out of the battery. See the always-on VAD power budget for a worked example.
- Memory. Model weights need flash, and activations need RAM. Microcontrollers have kilobytes to a few megabytes. That limits model size more than compute does.
- Accuracy. Small devices run small, quantized models, and quantization can cost accuracy.
- Updates. A model in the field needs a signed update path, storage for a second copy and a way to roll back.
How quantization trades size for accuracy
Quantization stores weights and activations as 8-bit integers or 16-bit floats instead of 32-bit floats. Google's LiteRT guide lists these options, as of November 2026:
| Technique | Benefit | Hardware |
|---|---|---|
| Dynamic range quantization | 4x smaller, 2x to 3x speedup | CPU |
| Full integer quantization | 4x smaller, 3x+ speedup | CPU, Edge TPU, microcontrollers |
| Float16 quantization | 2x smaller, GPU acceleration | CPU, GPU |
The same guide says post-training weight quantization can cost accuracy, particularly for smaller networks, and that it is important to check the accuracy of the quantized model. A model with float fallback operations will not run on integer-only devices such as 8-bit microcontrollers (LiteRT post-training quantization, as of November 2026). Our guide to checking INT8 model accuracy on a wearable shows how to run that check.
How do edge and cloud AI work together in one device?
Most voice and AI wearables split the work. The device runs small, always-on models. The cloud runs large models on demand.
- On the device: voice activity detection, wake word, noise suppression, simple commands and anything that must work offline.
- In the cloud: speech recognition for open dictation, large language models, search and anything that needs current data.
The device decides when to wake the radio and send audio. That gate sets both battery life and privacy, because nothing leaves the device until it fires. Its false accept rate matters as much as its accuracy. See wake word false accept and false reject rates.
Small language models on the device are possible on higher tiers. For the trade-offs on a wearable, see cloud LLM vs on-device SLM for a wearable, and for the long-run cost comparison, total cost of ownership: cloud vs edge AI. For how Duobay plans the device, app and cloud split, see device, app and cloud architecture.
Should your device run AI at the edge? A checklist
Work through these before you pick a chip.
- Write down the job: which input, which output, how fast, and what a wrong answer costs.
- Set the latency the user can feel, and check whether a cloud round trip fits inside it on a bad connection.
- Decide which data may leave the device, and get that decision from the privacy owner, in writing.
- Size the model for the target: flash for weights, RAM for activations, and the operations the runtime supports.
- Measure energy per inference on the real chip, then multiply by how often it runs in a day.
- Quantize, then measure accuracy again on a held-out set recorded with the real hardware.
- Plan model updates: where models are stored, how they are signed, and whether the OTA path can replace them.
- Keep a cloud fallback or a hybrid split for tasks the device cannot do well.
Questions about edge AI
What is the difference between edge AI and edge computing?
Edge computing means processing data near where it is produced, on a device or a nearby gateway, instead of in a distant data center. Edge AI is the subset of that work that runs machine learning models, usually inference. A gateway that filters sensor data is edge computing. A wearable that detects a fall with a model is edge AI.
Is on-device AI the same as edge AI?
On-device AI is the narrowest form of edge AI: the model runs on the product itself. Edge AI also covers models that run on a nearby gateway or box. For a wearable, on-device usually means the microcontroller or application processor inside it.
Can a microcontroller run a neural network?
Yes, if the model is small. Google says LiteRT for Microcontrollers is designed to run models on microcontrollers and other devices with only a few kilobytes of memory, and its core runtime fits in 16 KB on an Arm Cortex M3 (LiteRT for Microcontrollers, as of November 2026). The model weights and working memory come on top of that.
Does quantization reduce accuracy?
It can. Google's quantization guide says post-training weight quantization can cost accuracy, particularly for smaller networks, and that it is important to check the accuracy of the quantized model (LiteRT post-training quantization, as of November 2026). Measure on data recorded with your own sensors.
How can Duobay help with an edge AI device?
Duobay develops custom devices with a Shenzhen engineering team that covers electronics, firmware, enclosure, app and test. We plan the split between device, app and cloud, build samples and small batches, and deploy any app and backend in your own cloud account if you ask us to build them. Firmware source and the SBOM are delivered to you, and source files are assigned to you at every paid gate. Initial calls are free. Tell us about your device.
If you are deciding which AI features belong on your device and which belong in the cloud, tell us about your device.
Sources. All pages accessed November 12, 2026. Vendors update product pages and documentation, so confirm current values before you decide.
- Arm, Ethos-U55 product page (link)
- Google AI Edge, LiteRT for Microcontrollers overview (link)
- Google AI Edge, LiteRT post-training quantization (link)
- NXP, i.MX 93 applications processor family (link)
- Espressif, ESP32-S3 product page (link)
- Raspberry Pi, AI HAT+ product page (link)
Planning a device like this? Tell us what it needs to do. A Duobay engineer replies within 1 business day. Send a project brief →