Concept renderCheck INT8 accuracy loss by calibrating on 100 to 500 field-like samples kept separate from your eval set, then comparing float and INT8 on your task metric against a team-set maximum drop. Run per-layer error tools, try a denylist or per-channel quantization on the worst layers, and confirm op coverage on your runtime.
A quantized model can pass a desktop test and still disappoint on the device. The vendor documentation points to the calibration data and to per-layer error. A third check, what the target runtime does with each operation, depends on your chip vendor. This page turns the public LiteRT and ONNX Runtime documentation into a fixed validation routine for a keyword or sensor model.
This is an engineering write-up for products in general, including wearables. It does not cover medical devices, and it is not a guide to medical-device software.
What do the vendor docs actually say?
The table lists the documented facts this routine rests on, as of October 31, 2026. We paraphrase each page, so confirm the exact wording on the live page before you quote it.
| Topic | What the documentation says | Source and date |
|---|---|---|
| Representative dataset for full-integer post-training quantization | LiteRT (TensorFlow Lite) asks for a representative dataset. The docs describe it as a small subset, around 100 to 500 samples, of the training or validation data. | LiteRT post-training quantization, as of October 31, 2026 |
| Accuracy warning | Weights are quantized after training, so accuracy can drop. The docs single out smaller networks. | LiteRT post-training quantization, as of October 31, 2026 |
| Input and output interface | Inputs and outputs stay float by default. The inference_input_type and inference_output_type attributes, supported since TensorFlow 2.3.0, change the interface. | LiteRT post-training quantization, as of October 31, 2026 |
| Per-layer debugging | The Quantization Debugger logs float versus quantized differences per layer. It accepts denylisted_nodes and denylisted_ops to skip quantization for specific layers. | LiteRT Quantization Debugger, as of October 31, 2026 |
| ONNX Runtime static quantization | Quantization parameters are computed offline from calibration data. The docs describe quantization as lossy and list per-channel quantization as a remedy for models with large weight ranges. | ONNX Runtime quantization, as of October 31, 2026 |
| ONNX Runtime weight comparison | The create_weight_matching() function takes a float32 model and its quantized model and compares their weights. | ONNX Runtime quantization, as of October 31, 2026 |
Accuracy-drop thresholds, the effect of calibration mismatch on a specific chip, and the cost of unsupported ops on a specific runtime are not in these pages. These are not in the pages above, so you need to measure them on your board.
What is the go/no-go routine?
The routine has five steps, built only from tools the documentation names. Every threshold in it is a value your team sets.
| Step | What to do | Why it is in the routine |
|---|---|---|
| 1. Build the calibration set | Draw 100 to 500 samples from field-like data (the range the LiteRT docs give). Keep them separate from the set you use to judge the result. | Static quantization computes its parameters from calibration data (ONNX Runtime), and LiteRT asks for a representative dataset, so the data matters. |
| 2. Compare float and INT8 on the task metric | Run the same eval set through both models. Use the metric your product lives on, such as accuracy or false-accept and false-reject rate for a keyword model. | The docs warn that accuracy can drop. They publish no tolerance, so the maximum drop is yours to set. |
| 3. Find the worst layers | Run per-layer error metrics (LiteRT Quantization Debugger) or compare weights (ONNX Runtime create_weight_matching()). Rank layers by error. | The debugger exists to show where float and quantized outputs diverge. |
| 4. Try the remedies the docs name | Skip quantization on the worst layers with a denylist (LiteRT) or switch to per-channel quantization (ONNX Runtime). Re-run step 2 after each change. | Both remedies appear in the vendor documentation. Their effect on your model is unknown until you measure it. |
| 5. Check op coverage on the target runtime | List every op in the quantized graph and check it against the delegate or vendor toolchain documentation for your chip. | The pages we checked do not describe fallback behavior on a specific chip, so this step relies on your vendor. |
The routine ends in a decision with these possible results: ship the INT8 model, change the quantization setup and run it again, or keep a higher-precision model for the layers or the product that need it. You choose the pass line before you look at the results, and you write down why.
What pitfalls follow from the documentation?
These items are engineering reasoning drawn from the pages above. They are not accounts of any company's incident, and we report no measured results.
- Calibration data that looks like the lab and not the field. Static quantization computes its parameters offline from the calibration data (source). If that data differs from what the device sees, the parameters were fitted to the wrong range. The documentation does not quantify this effect, so it is a reason to test with field-like samples.
- Calibrating and judging on the same samples. The result then says little about unseen data. Keep the two sets apart.
- A small network is the documented risk case. LiteRT names smaller networks as more exposed to accuracy loss (source). Keyword and sensor models are often small, so check them first.
- One bad layer can hide inside a good average. The reason per-layer tools exist is to find where float and quantized outputs diverge (source). A single task metric cannot tell you which layer to fix.
- Float interfaces surprise firmware teams. Inputs and outputs stay float by default (source). If your firmware pipeline produces integer samples, decide how they reach the model and test that conversion path.
- Op support is a runtime question. A model that converts cleanly can still contain operations your target runtime handles differently. The pages we checked do not describe this for any chip, so read the vendor documentation for yours.
What should you test on your own board?
Documentation sets the method. Only your hardware and data tell you the numbers. We have not run these on your product, and this page reports no results. The pass line is set by you from your product requirements, and you record the basis for it.
| What to test | How to instrument or capture it | Raw data to record |
|---|---|---|
| Task metric, float versus INT8 | Run the same eval set through both models off the device, then through the INT8 model on the device. | Metric values for each run, model file versions, eval set version, runtime version. |
| Calibration set effect | Rebuild the INT8 model from different calibration sets and compare each on the same eval set. | Where each calibration set came from, its size, and the metric for each resulting model. |
| Per-layer error | Run the LiteRT Quantization Debugger or ONNX Runtime create_weight_matching() and export the per-layer results. | Per-layer error table, the layers you denylisted or changed, and the metric after each change. |
| Op coverage on the target | List every op in the quantized graph and compare it with the vendor's supported-op documentation for your chip. Check the runtime logs for ops it does not run on the accelerator (see what an NPU runs and what falls back to the CPU). | Op list, vendor document version, and any op the runtime reports it handles elsewhere. |
| Latency and memory on the device | Time a full inference on the board with the final build, using a hardware timer or a GPIO toggle read on a logic analyzer. | Raw timings per inference, model size in flash, peak RAM, clock and power settings. |
| Input interface | Feed the model through the same path your firmware will use, including any float to integer conversion. | The input type set at conversion, the firmware conversion code version, and sample inputs with their outputs. |
What should you ask before you commit?
Take this list to your ML team, your chip vendor or any team building the device for you, including us.
- Which runtime and delegate will run the model on the final board, and which version of its documentation did you check?
- Where does the calibration data come from, and how close is it to what the product sees in use?
- Which metric decides pass or fail, who set the maximum drop, and what is the written reason?
- Is the eval set kept separate from the calibration set, and who guards that?
- Which layers carry the most error, and what happens to the metric when you skip them?
- Does every op in the quantized graph have documented support on the target chip?
- What does the firmware feed the model, and does the input type match the converted model?
- What is the plan if the INT8 model misses the pass line late in development?
Questions about INT8 validation
How many calibration samples do I need?
LiteRT describes a representative dataset of around 100 to 500 samples drawn from training or validation data. Treat that as the documented starting range. How the choice of samples changes accuracy on your model is something to measure on your own data. As of October 31, 2026. LiteRT post-training quantization.
What accuracy drop is acceptable?
No documentation we checked gives a number. The LiteRT docs only say accuracy can drop, especially for smaller networks. Set the maximum drop from what your product needs, and write down the reason. LiteRT post-training quantization.
Which tool shows where the error comes from?
LiteRT has the Quantization Debugger, which logs per-layer float versus quantized differences. ONNX Runtime has create_weight_matching(), which compares the weights of a float32 model and its quantized version. LiteRT Quantization Debugger, ONNX Runtime quantization.
Do unsupported ops fall back to the CPU?
The pages we checked do not say. Behavior depends on the delegate and the vendor toolchain, and the vendor's official documentation for your chip is the place to confirm it. The pages we checked give no latency figure, so measure it on your board.
Does the model input stay float after conversion?
By default, yes. LiteRT keeps inputs and outputs as float, and the inference_input_type and inference_output_type attributes (TensorFlow 2.3.0 and later) change that. Decide this early, because it changes what your firmware feeds the model. LiteRT post-training quantization.
Putting a small model on a device and want the validation plan checked against your board and data? Tell us about your device. For how the engineering gates run, see our engineering approach, and for the project stages, see how it works.
Primary sources
All pages below were accessed on 2026-10-31. Vendor documentation changes, so confirm details on the live pages.
- Google, LiteRT post-training quantization (developers.google.com/edge/litert/models/post_training_quantization), accessed 2026-10-31.
- Google, LiteRT Quantization Debugger (developers.google.com/edge/litert/performance/quantization_debugger), accessed 2026-10-31.
- ONNX Runtime, Quantization (onnxruntime.ai/docs/performance/model-optimizations/quantization.html), accessed 2026-10-31.
Planning a device like this? Tell us what it needs to do. A Duobay engineer replies within 1 business day. Send a project brief →