API rate limits when thousands of devices share one LLM account

Illustration

Rate limits belong to your account or project, so every device shares one pool. On Anthropic's published Claude Sonnet 5.5 limits as of October 31, 2026, 5,000 devices split to between 0.8 and 2.6 requests per device per minute, depending on tier. Bursts can trip limits earlier, so stagger wake-ups, back off with jitter and queue centrally.

A device fleet looks like many users, but a model API sees one customer. Provider limits attach to an organization or a project, not to a device, so 5,000 units on one key draw from one pool. This guide turns the published limits into per-device arithmetic you can rerun with your own fleet size, then lists the launch-day failure patterns that follow from the documentation and the tests worth running on your own system.

All numbers below come from the providers' public documentation as of October 31, 2026. Limits change often and differ by model, so check the linked pages before you rely on any figure.

How do the three big providers scope their limits?

The scope decides whether a second key, a second user or a second device adds any capacity. The mechanics below are paraphrased from each provider's documentation, as of October 31, 2026.

Per-model request and token numbers for OpenAI and Google are not in this table. We did not capture them from the official pages, and Google shows its per-model limits only inside AI Studio. Read them from your own account before you plan capacity.

What do the published numbers look like for one model?

Anthropic publishes a table per tier and model. This one shows Claude Sonnet 5.5, taken from the official rate limits page, as of October 31, 2026. The page states that these are maximums, not guaranteed minimums.

The same page gives the Message Batches API its own limits, shared across all models. On the Build tier that is 2,000 requests per minute and 300,000 batch requests in the processing queue (source, as of October 31, 2026).

How many requests does each device actually get?

The first calculation is a ceiling. Divide the account limit by the number of devices. The table uses 5,000 devices, and the formula works for any fleet size.

Formula: per-device allowance per minute = account limit ÷ N, where N is the number of devices sharing the account. Monthly budget per device = monthly spend cap ÷ N.

Inputs: the Anthropic Sonnet 5.5 row above (source, as of October 31, 2026) and N = 5,000. The result is a ceiling. The provider may enforce each limit over shorter intervals than a minute, so a fleet that spends its whole minute allowance in one second can still be rejected.

Check your own load against the ceiling

The second calculation is your demand. Put your own numbers in the variables below. We do not supply them, because they depend on your product.

• Requests per minute = N × a × r, where a is the share of devices active in the same minute and r is requests per active device per minute. a and r are assumptions you replace.
• Uncached input tokens per minute = N × a × r × t × (1 − h), where t is input tokens per request and h is the share of that input served from cache. t and h are assumptions you replace. Anthropic documents that for most Claude models only uncached input tokens count toward the input limit, so a higher h raises effective throughput (source).
• Output tokens per minute = N × a × r × o, where o is output tokens per request. o is an assumption you replace.
• Monthly spend = your measured cost per request × requests per month. Compare it with the cap for your tier.

If any of the first three results exceeds the tier limit, or monthly spend exceeds the cap, that tier cannot carry the fleet. The a variable is the one that surprises teams, because it is set by your wake-up design. An always-on assistant and a button-press recorder produce very different values for the same N.

What breaks on launch day?

These items are engineering reasoning drawn from the documentation above. They are not accounts of any specific company's incident, and we have no measured results to report.

• One pool for the whole fleet. Because limits attach to the organization or project, a firmware bug that makes a few thousand devices chatty draws down the same pool as everyone else. Google also states that adding API keys inside one project adds no capacity (source).
• The per-minute number is not a safe burst rate. Anthropic says a limit of 60 requests per minute might be enforced as 1 request per second, and that short bursts can trigger errors (source). A synchronized wake-up at the top of the minute is a burst by construction.
• Retries can multiply the load. OpenAI states that unsuccessful requests contribute to the per-minute limit (source). If thousands of devices retry on the same schedule, every retry wave spends quota while earning nothing. OpenAI's guidance is to honor Retry-After, and otherwise to use exponential backoff with jitter.
• A spend-cap rejection looks like a rate limit but behaves differently. Anthropic documents a monthly spend cap that returns a 429 with no retry-after header, and automatic SDK retries fail until access resumes (source). A fleet that treats every 429 the same way will retry against a wall.
• A sharp ramp can be rejected on its own. Anthropic describes acceleration limits that can return 429 when usage rises sharply, and advises increasing traffic gradually (source). A same-day firmware rollout to the whole fleet is a sharp ramp.
• Tier upgrades take time. Google's tiers need cumulative spend plus elapsed days, and it makes no guarantee about increases (source). A new account cannot move to a high tier the week before launch. OpenAI moves tiers automatically as credit purchases reach thresholds (source).
• Cache design changes your capacity. When cache reads do not count toward the input limit, a long shared system prompt that caches well costs less quota than the same prompt sent fresh. That makes prompt layout a capacity decision as well as a latency one.

What should you test on your own system?

The published numbers set the ceiling. Only your own fleet and prompts tell you where you sit under it. Run each test against a staging account with its own limits so a failure does not touch production. We have not run these on your product, and this page reports no results.

What design choices follow from this?

• Put your own backend between devices and the model API. It gives you one place for a queue, a shared rate limiter and retry policy, and it keeps provider keys off the devices.
• Spread wake-ups. Add random delay to scheduled or push-triggered calls so the fleet does not act on the same second.
• Separate the fleet from your other products. Anthropic workspaces can carry lower limits than the organization, which keeps a runaway device cohort from consuming the whole organization pool. Organization-wide limits still apply (source).
• Move non-urgent work to batch endpoints where the provider offers them, as with Anthropic's Message Batches API and its separate limits.
• Decide what the device does when the model is unreachable. A defined offline behavior is cheaper than a spinner that retries forever.

What should you ask before you commit?

Take this list to your backend team, your cloud vendor or any team building the integration for you, including us.

• Which provider, model and tier will the fleet run on, and what are that model's published limits today?
• What are the assumptions for a, r, t, h and o, and where did each one come from?
• Who owns the provider account and the keys, and are they in your company's name?
• Is there a backend queue and a central limiter, or do devices call the API directly?
• How do devices behave on a 429 with Retry-After, a 429 without it, and a timeout?
• What alert fires before the monthly spend cap, and who receives it?
• How is the launch ramped, and what is the rollback if 429 rates climb?
• Which workloads can run as batch jobs outside the real-time path?
• What is the fallback if the provider changes limits or retires the model?

Planning a device that talks to an LLM and want the load math checked against your real wake-up pattern? Tell us about your device. For how we scope the app and cloud side, see App, cloud and AI in our engineering approach.

Primary sources

All pages below were accessed on 2026-10-31. Limits change, so confirm figures on the live pages.

• Anthropic, Rate limits (platform.claude.com/docs/en/api/rate-limits), accessed 2026-10-31.
• OpenAI, Rate limits guide (developers.openai.com/api/docs/guides/rate-limits), accessed 2026-10-31.
• Google, Gemini API rate limits (ai.google.dev/gemini-api/docs/rate-limits), accessed 2026-10-31.