Cloud LLM or on-device small model for a wearable: how to choose

Concept render

Choose by constraint. A cloud LLM needs a radio link for every request, sends audio off the device and bills per audio token. An on-device small model keeps audio local and works offline, but it must fit your memory, power and license limits. A hybrid split is also an option: wake word on the device, heavy reasoning in the cloud.

Founders ask this question early, and the answer changes the bill of materials, the battery, the privacy story and the firmware plan. This guide lays out the constraints that decide it, the published facts you can build on, the formulas to fill with your own numbers, and the measurements to run on your own board. Where the public documentation has no number, we say so and leave a variable for you to fill.

Which constraints decide cloud versus on-device?

Six constraints do most of the deciding. The table below is our engineering analysis. Vendor documentation does not contain it, and nothing in it is a measured result.

What do the vendors publish about small on-device models?

These are the anchors we could verify from primary pages. Every row is as of October 31, 2026. Check the source before you rely on a figure, because vendors revise these pages.

Two cautions on this table. First, Google states that its memory figures "only account for the memory required to load the static model weights." The working memory the model needs while it runs (the KV cache) and the memory your firmware and operating system need come on top of that. Second, Google also says the numbers may change with your inference tool and environment. The Gemma page itself carries a notice that it is deprecated and points to a newer location, so re-check the figures before publishing a spec. We did not verify the Gemma license text for this article.

What does the cloud alternative cost per audio token?

OpenAI publishes per-token prices for its realtime audio models. Prices change often. Treat these rows as a snapshot.

On these listed prices, the mini model's per-token price is 10/32, about 31 percent, of the full model's audio input price. The Realtime API works directly with audio, keeps conversation state and can call tools, over WebRTC in a browser or WebSocket on a server (OpenAI Realtime guide, as of October 31, 2026). A wearable is neither, so the device usually talks to your own backend, which relays to the provider. That relay is yours to build and pay for.

How do you check whether a model fits, and what will the cloud cost?

Two short formulas turn the published facts into a decision. Variables marked as your assumption are not in any source we reviewed. Replace them with values from your datasheet, your measurements or your usage plan.

Memory fit

required RAM = model weights + KV cache + runtime buffers + firmware and OS

• Model weights. From the Gemma docs, as of October 31, 2026: 1.1 GB (E2B mobile) or 2.5 GB (E4B mobile).
• KV cache, runtime buffers, firmware and OS. Your assumption. The Gemma docs do not give them. Measure them on the target.
• Available RAM. Your assumption until you read the datasheet of your candidate chip or module. We did not collect a chip datasheet for this article.

The model fits only if required RAM is below available RAM with margin that you set. Whether you also need an NPU, and how fast the model runs on it, depends on the chip and is not answered by the model docs.

Cloud audio cost

cost per device-hour = (input tokens per hour × input price + output tokens per hour × output price) ÷ 1,000,000

This formula does not include cached-input pricing. When requests hit the cache, the real cost is lower.

• Prices. From the table above, as of October 31, 2026.
• Input and output tokens per hour of audio. Your assumption. We did not find an official conversion from seconds of audio to tokens, so we do not give an hourly figure. Get the conversion from the provider's documentation or from a usage report on a real test session.
• Hours of audio sent per device per day. Your assumption. This is where an on-device wake word and voice activity detection matter, because they decide what you send.

Worked example of the cost formula, with symbols only

Let T_in be your input audio tokens per hour and T_out your output audio tokens per hour. Both are your assumptions. With the prices in the table above, as of October 31, 2026:

• gpt-realtime-2.1: cost per device-hour = (T_in × $32.00 + T_out × $64.00) ÷ 1,000,000
• gpt-realtime-2.1-mini: cost per device-hour = (T_in × $10.00 + T_out × $20.00) ÷ 1,000,000

The result is a dollar figure per device-hour. Multiply it by the hours of audio you send per device per day to get a daily cost per device. We do not put numbers into T_in or T_out because we did not find an official conversion from audio to tokens.

Which design should you pick? A short decision guide

This is engineering inference from the facts above. Work through the rules in order. Rule 1 removes options that cannot be built. Rules 2 to 4 then choose among the options that remain.

1. Check that the weights fit in memory. Compare the published weight sizes, 1.1 GB for Gemma 4 E2B mobile and 2.5 GB for E4B mobile, with the RAM in your datasheet. The KV cache, runtime buffers and firmware come on top of the weights, so a chip with only a little room above the weights counts as not fitting. If even 1.1 GB does not fit with margin, on-device inference of these models is off the table. Go to rule 2. If the weights fit with margin, go to rule 3.
2. Resolve conflicts with your hard requirements. If the weights do not fit and the product must work with no link, or audio must not leave the device, you have a design that cannot be built as specified. Pick one of three fixes: a chip with more memory, a smaller local model, or a relaxed offline or privacy requirement. To decide, price each fix. Compare the bill of materials change of the larger chip, the quality you lose with the smaller model (test it on your own audio), and the business cost of loosening the requirement. Take the cheapest fix that still meets the requirements you cannot drop. If the weights do not fit and no hard requirement applies, use the cloud path and apply the cost check in rule 4.
3. Apply the hard requirements to the options that fit. If the feature must work with no link, a cloud-only design cannot meet it, so pick on-device, or hybrid with an on-device fallback. If audio must not leave the device, pick on-device, because anything you send to a cloud model leaves the device and falls under the provider's terms.
4. Decide the rest on measurements. If the weights fit with margin and no hard requirement applies, the decision comes down to reply speed, cost and battery. Fill T_in and T_out from a real test session, run the cost formula, and compare it with your cost per device target. Run the latency and battery measurements below, then pick the path that wins on the measured figures.

What about a hybrid design?

A hybrid design splits the work. A small model on the device handles the always-on part, such as wake word and voice activity detection, and sends audio to a cloud model only after a trigger. This reduces both radio airtime and audio token spend, and it keeps idle audio on the device. It adds firmware complexity, because you now maintain a local model, a trigger policy and a cloud client, and you have to handle the cases where the link is down. Whether the saving is worth that complexity depends on the measurements below.

What traps can you spot from the published facts?

This section is engineering inference from the sources above. It describes risks that follow from the published facts.

• Sizing RAM from the weights table alone. Google says the table covers static weights only. A chip that fits the weights with no room left will fail once the KV cache and your firmware are loaded.
• Copying a mobile phone number to a wearable. The mobile footprints are published for mobile deployment. A wearable-class chip can have a different memory budget. Check the datasheet.
• Assuming a quantized figure is permanent. Google notes the numbers may change with the inference tool and environment. Pin your runtime version and re-measure after any change.
• Reading license terms late. Llama 3.2 requires a visible attribution in a product's interface or documentation, which is a design and packaging task. The 700 million monthly active user clause is a business condition to track. Phi-3-mini's MIT license is simpler, but read it too. Check the Gemma license text before you choose Gemma.
• Budgeting the cloud from the price table without a token conversion. The price per million tokens is public. Tokens per hour of audio is the missing input. Without it the monthly cost is unknown.
• Treating a price as stable. Provider prices and model names change. Recheck the pricing page when you finalize a business case.
• Ignoring the no-link case for a cloud design. If the product must work offline, a cloud-only design cannot meet that requirement.

What should you measure on your own board?

The vendor documentation has no numbers for latency or battery. These measurements fill that gap on your hardware. Run them on a development board that matches your target chip, with your real model and your real microphone path. We have not run them for you, and no results are published here.

Our works-like prototype gate exists for these measurements: proven modules on a custom carrier board, so your model runs on the target silicon. See how the prototype gates work.

What should you ask a supplier, and yourself, before you commit?

• What is the available RAM on this chip or module, and how much of it does the firmware use before any model loads?
• Which inference runtime does the vendor support on this chip, and which model formats does it load?
• Does the NPU, if there is one, support the operators your model uses, or does work fall back to the CPU?
• Which license covers the model I plan to ship, and what does it require in the product's interface or documentation?
• What leaves the device in each design, and does my privacy statement match it?
• What happens when there is no link: does the feature degrade, queue or stop?
• How will I update the model and the runtime on a deployed device, and how will I roll back?
• What is my token estimate per hour of audio, and where did the number come from?
• When do I recheck provider prices, and what unit cost would make me change the design?

Primary sources

All pages accessed October 31, 2026.

• https://ai.google.dev/gemma/docs/core (accessed 2026-10-31). The page states it is deprecated and points to gemma.ai. Re-check these figures at the new location before publishing.
• https://huggingface.co/microsoft/Phi-3-mini-4k-instruct (accessed 2026-10-31)
• https://dev.meta.ai/llama/llama3_2/license (accessed 2026-10-31)
• https://developers.openai.com/api/docs/guides/realtime (accessed 2026-10-31)
• https://developers.openai.com/api/docs/pricing (accessed 2026-10-31)

Building a wearable with a voice or AI feature? See our AI recording wearable development work, then tell us about your device and we will propose where to start.