What Is an NPU? Edge AI Chips for Small Devices

Concept render

An NPU, or neural processing unit, is a hardware block that runs the multiply-and-add math of neural networks with less time and energy than a CPU. In small devices it sits beside a microcontroller or application processor core, runs quantized models built with a vendor compiler, and often leaves unsupported operations to the CPU.

Many guides to NPUs are written for laptop buyers. A hardware team has a different question: will an NPU make our model run fast enough, on our battery, on a chip we can actually ship? This guide explains NPUs for small devices, shows what three vendors publish about their parts, and lists the checks that answer that question on your own board.

This is an engineering summary. Vendor figures are quoted as each vendor states them, as of November 2026. Confirm current values on the vendor page before you choose a part.

What is an NPU, and what does it do in a small device?

An NPU is a processor block built for the math inside neural networks. Most of that math is multiplying and adding large arrays of numbers. A CPU does it one small step at a time. An NPU does many of those steps in parallel, on narrow integer data, with less energy per inference.

In small devices the NPU is rarely a separate chip. It sits on the same die as a main core. NXP's i.MX 93 page calls the Arm Ethos-U65 a microNPU. Arm says its Ethos-U55 is designed to accelerate ML inference in area-constrained embedded and IoT devices, and pairs it with Cortex-M55 and Cortex-M85 cores (Arm, as of November 2026).

The NPU does not run your firmware. The main core still handles the radio, sensors, power states and app logic. It hands model layers to the NPU and gets results back. A tool chain decides which layers go where before the firmware is built. For Arm Ethos-U parts that tool is the Vela compiler, which converts networks defined in TOSA or TensorFlow Lite/LiteRT format into Ethos-U command streams (Vela README, as of November 2026).

NPU vs GPU vs CPU: which runs what in edge AI?

Edge AI means running the model on the device itself, close to the sensor, without a round trip to a server. Our guide to what edge AI is covers the latency, privacy and power trade-offs. On a small device the three processor types split that work as below. The rows are general engineering guidance; the vendor sections further down carry the sourced figures.

The practical difference between an NPU and a GPU is flexibility. A GPU will run almost any parallel math you write for it. An NPU runs a fixed menu of operations, and its compiler decides what fits. That menu is the first thing to read when you evaluate one.

Which chips put an NPU on a microcontroller or application processor?

NPUs now appear at two levels. Microcontrollers add a small NPU beside a Cortex-M class core, for sensor and audio models under an RTOS. Application processors that run Linux add a larger one. Which level fits your product is the same choice covered in MCU or Linux SoC for an AI wearable. The table shows what three vendor pages state. It is a sample of three parts.

Notice the spread. One vendor gives operations per second, one gives a multiple over older chips, and one gives no NPU figure at all on its own product page. The two published figures cannot be compared with each other, and the third part has no NPU figure to compare.

Why do TOPS and speed-up figures mislead?

TOPS means trillions of operations per second. GOPS means billions. Both are peak figures. They tell you how much math the NPU could do if every operation in your model ran on it, at the data width the vendor measured, with data always ready in memory. Real models rarely meet all three conditions.

• Different baselines. Arm's 480 times figure compares an Ethos-U55 with a Cortex-M55 against existing Cortex-M based systems. TI's 2.56 GOPS at 80 MHz has no stated baseline or workload on the product page. Both are as of November 2026.
• Different data widths. The TI NPU accepts weights down to 2 bits and data down to 4 bits, and the product page does not say which width its 2.56 GOPS figure assumes (TI, as of November 2026). A peak figure measured at a narrow width does not carry over to an 8-bit model.
• Layers that stay on the CPU. If a third of the model falls back to the main core, that third runs at CPU speed whatever the NPU peak is.
• Memory traffic. Moving weights and activations in and out of memory costs time and energy that the peak figure does not count.

The only figure that predicts your product is your model, compiled with the vendor tool, timed and measured for energy on an evaluation board.

What limits on-device AI on an NPU?

Three limits decide whether on-device AI works on a given NPU: the operators it supports, the memory around it, and the data types it accepts.

Operator support and CPU fallback

Arm publishes a supported-operators report for Vela 5.2.0. For TensorFlow Lite models on Ethos-U55 and U65, an operator that does not meet its constraints is scheduled on the CPU instead, and any operator not listed is left untouched and scheduled on the CPU. For TOSA models on Ethos-U55 and U65, CPU fallback is not supported, and unsupported operators or configurations result in a compiler error (Vela supported operators, as of November 2026).

Constraints are specific. In the same report, ARG_MAX on Ethos-U55 and U65 requires an input feature map depth of 127 or less, and MEAN requires a batch of 1. A model can compile cleanly and still run most of its time on the CPU. Read the compiler output as well as the build status.

Memory

An NPU speeds up math. It does not add memory. The TI MSPM0G5187 pairs its NPU with up to 32 KB of SRAM and up to 128 KB of flash (TI, as of November 2026). The model weights, the working buffers and the rest of your firmware all share that space. On the software side, Google says the LiteRT for Microcontrollers core runtime fits in 16 KB on an Arm Cortex M3 (LiteRT for Microcontrollers, as of November 2026). Budget memory before you compare speed.

Quantized data types

NPUs work on narrow integers. Vela requires quantized activations and quantized weights for a network to be accelerated on Ethos-U, and the Ethos-U55 and U65 TensorFlow Lite path lists INT8, UINT8, INT16, INT32 and INT64 as feature-map data types (Vela README and supported operators, as of November 2026). Quantizing a float model can cost accuracy, so measure it on your own data. Our INT8 quantization accuracy check covers the steps.

Toolchains are a fourth, quieter limit. Each vendor ships its own tools: Vela for Arm Ethos-U, including the U65 in NXP's i.MX 93, and Edge AI Studio for TI. NXP's eIQ Toolkit page lists an Arm Vela extension among its downloads (NXP eIQ Toolkit, as of November 2026). Your build depends on that tool for the life of the product, so keep the exact version with your source files.

When does a device not need an NPU?

Many devices do fine without one. Small models can run on the CPU with a general runtime such as LiteRT for Microcontrollers, which Google designed for devices with only a few kilobytes of memory (Google, as of November 2026). Our guide to TinyML on microcontrollers covers those runtimes and their memory budgets. If the model is a large language model, the real choice is usually cloud or device, covered in cloud LLM vs on-device model for a wearable.

The order matters. Measure the model on the CPU first. Move to an NPU part only when the CPU misses a target you wrote down, and when the compiler report shows that the slow layers will actually run on the NPU.

What should you ask your hardware team?

Bring these questions to the chip selection review, before the schematic starts. Write the answer and the evidence next to each one.

• Which model, in which format, are we planning to run, and has it been through the NPU compiler yet?
• What does the compiler report say: how many layers run on the NPU, and how many fall back to the CPU?
• Which data types does the NPU accept, and has the model been quantized to match?
• What accuracy did the quantized model score against the float model on our own test data?
• How much RAM and flash does the model use, and how much is left for the radio stack and the rest of the firmware?
• What are inference time and energy per inference on an evaluation board, measured with our model?
• Would the same model meet its targets on the CPU alone, and was that measured?
• Is the vendor toolchain free to use, can it run offline, and who keeps a copy of the exact version we built with?

Chip choice sits inside the electronics and firmware work. For how those are organized, see how the engineering gates run.

If you want a chip and model plan checked against your device's battery and latency targets, tell us about your device.

Sources. All pages accessed November 6, 2026. Vendor pages change, so confirm current values before you decide.

• Arm, Ethos-U55 product page (link)
• Arm, Vela compiler project (link), README (link) and supported operators report for Vela 5.2.0 (link)
• Texas Instruments, MSPM0G5187 product page (link)
• NXP, i.MX 93 applications processor product page (link) and eIQ Toolkit page (link)
• Google, LiteRT for Microcontrollers overview (link)