IllustrationThe circuit breaker pattern stops a client from calling a failing service once failures pass a threshold. In the closed state calls pass and failures are counted, in the open state calls fail fast until a timer expires, and in the half-open state a few trial calls test recovery. Devices pair it with backoff, jitter and local fallback.
A connected device calls the cloud all day: an MQTT broker for telemetry, an LLM API for voice replies, a file server for firmware updates. When one of those endpoints fails, every device in the fleet notices at the same time. This guide explains the circuit breaker pattern and its three states, shows how fleet retries turn into a retry storm, and gives a decision table, a device-side state machine and a fault-injection checklist for battery-powered products.
This is an engineering summary. Vendor defaults and quotas change, so check the linked pages before you set values in firmware. All sources were read on November 14, 2026.
What is the circuit breaker pattern?
A circuit breaker is a proxy that sits in front of a call that might fail. It watches recent failures and decides whether to let the next call through or to fail it at once. Microsoft's Azure Architecture Center describes it as a state machine with three states named after an electrical breaker (Azure circuit breaker, as of November 2026):
- Closed. Calls go through. The proxy counts recent failures, and when they pass a threshold within a time period it moves to open and starts a time-out timer.
- Open. Calls fail immediately and return an error to the application.
- Half-open. A limited number of calls pass through. If they succeed, the breaker closes and resets the failure counter. If any fails, it goes back to open and restarts the timer.
Azure explains that the half-open state keeps a recovering service from being flooded with requests, because a service in recovery may handle a limited volume before it is fully back. The failure counter in the closed state is time based and resets at periodic intervals, so occasional failures do not trip the breaker (Azure circuit breaker, as of November 2026).
The pattern was written for cloud services calling other services. A device fleet fits it well, because thousands of identical clients share one dependency. For how devices, brokers and backends fit together, see IoT architecture for connected devices.
How do thousands of devices turn retries into a retry storm?
Retries multiply. The AWS Builders' Library gives an example of a call that passes through a five-deep stack of services. If each layer retries three times independently, the load on the database increases 243x, which makes it unlikely to recover (AWS Builders' Library, as of November 2026). A device that retries, behind a gateway that retries, in front of a backend that retries, builds the same stack.
Timing makes it worse. AWS notes that clients often send requests on a regular interval, such as once per minute, and that many of them can line up and send at the same time. It also says retries can be ineffective when errors come from load and all clients retry together (AWS Builders' Library, as of November 2026). A fleet that lost power in one building, or that all dropped off when the broker restarted, starts in lockstep.
Microsoft warns IoT Hub customers that a high number of connection attempts per second can look like a distributed denial-of-service attack, that the effect can reach beyond the tenant to the whole scale unit, and that the result can be higher cost or a service interruption (Azure IoT reconnections, as of November 2026). Broker quotas set a hard ceiling. AWS IoT Core lists a default of 3000 MQTT CONNECT requests per second per account, or 100 in select Regions, and limits the same client ID to 1 MQTT CONNECT operation per second (AWS IoT Core quotas, as of November 2026). A large fleet that reconnects at once can hit the account limit, and the rejected connects then trigger more retries. This is engineering reasoning.
Defaults in device libraries matter here. The Espressif esp-mqtt client reconnects to the broker after a fixed interval, default 10000 ms, unless auto reconnect is disabled (esp-mqtt guide, as of November 2026). A fixed interval with no jitter keeps devices that dropped together in step on every attempt. Change that default before shipping. For the protocol choice behind these connections, see MQTT vs HTTP for IoT devices.
Retry, backoff, breaker or local fallback: which fits each failure?
These tactics stack. A device usually retries once, backs off with jitter, trips a breaker if failures continue, and falls back to local behavior while the breaker is open. The table shows when each one fits. Sourced cells link to the page they came from, as of November 2026.
| Tactic | Use it when | Watch out for |
|---|---|---|
| Retry now | A single call failed on a fault that usually clears fast, such as a dropped socket. Azure describes transient faults as ones that typically correct themselves after a short period of time (Azure circuit breaker, as of November 2026). | AWS calls retries selfish, because a retrying client spends more of the server's time to raise its own chance of success. It also says APIs with side effects are not safe to retry unless they provide idempotency (AWS Builders' Library, as of November 2026). |
| Capped exponential backoff with jitter | Any reconnect to an MQTT broker or HTTPS endpoint. The AWS IoT Lens tells device makers to use exponential backoff with a randomization factor so retries from many devices do not happen at the same time (AWS IoT Lens, as of November 2026). | Backoff keeps each device calling a dead endpoint forever, only less often. The AWS IoT Device SDK for JavaScript sets its maximum reconnection back off to 128 seconds by default (AWS IoT Device SDK JS v2, as of November 2026). |
| Circuit breaker | Failures keep arriving inside a time window, or the server signals it is out of capacity. Azure describes a breaker that trips after failures reach a threshold and fails fast while open (Azure circuit breaker, as of November 2026). | The open state is a separate mode that rarely runs in normal use. AWS warns that circuit breakers add modal behavior that can be hard to test and can add time to recovery (AWS Builders' Library, as of November 2026). |
| Local fallback | The device can still do useful work offline: keep a reading, play a stored prompt, queue an event. AWS says devices should store important messages durably offline and send them once reconnected (AWS IoT Lens, as of November 2026). | Storage fills up. AWS suggests capping the space used and considering a FIFO overwrite strategy (AWS IoT Lens, as of November 2026). |
| No retry | Data that is worthless once late, such as a live status ping. Azure lists a no retry policy that is typically used when bandwidth or cost is a concern (Azure IoT reconnections, as of November 2026). | Azure notes that with this policy, messages that fail to send are lost and can't be recovered (Azure IoT reconnections, as of November 2026). |
Jitter is the cheapest fix in the table. AWS describes jitter as randomness added to the backoff to spread retries around in time, and capped exponential backoff as backoff with a maximum wait (AWS Builders' Library, as of November 2026). AWS also limits retries locally with a token bucket, so retries are allowed while tokens remain and then fall to a fixed rate (AWS Builders' Library, as of November 2026). A token bucket fits in a few bytes of firmware state and caps total retries during a long outage. This is engineering reasoning.
What does a device-side breaker state machine look like?
Below is one design for a device that talks to several cloud endpoints. Each step follows the states Azure describes (Azure circuit breaker, as of November 2026). The firmware details are engineering reasoning.
- Closed. Send the call with a timeout. On failure, add one to a counter for that endpoint. The counter covers a rolling window and clears when the window ends. Trip to open when the count passes the threshold inside the window.
- Open. Fail every call to that endpoint at once and switch to the local fallback. Start a timer. Do not power up the radio for this endpoint while the timer runs.
- Half-open. When the timer expires, allow one trial call, or a small fixed number. Close after the set number of consecutive successes. On any failure, reopen and restart the timer with a longer value.
Setting thresholds
Azure suggests the open timer can grow: a few seconds at first, then a few minutes if the failure persists. It also says a breaker can treat error types differently, for example needing more time-outs to trip than failures from an unavailable service (Azure circuit breaker, as of November 2026). Some vendor guidance gives fixed reference points. For IoT Hub with the Device Provisioning Service, Microsoft says to reprovision the device when a retryable error has failed reconnection 10 consecutive times, and to reprovision on HTTP responses 401, 403 or 404 (Azure IoT reconnections, as of November 2026).
Timeouts feed the counter, so set them with care. AWS picks an acceptable rate of false timeouts, such as 0.1%, and sets the timeout from the matching latency percentile of the downstream service, p99.9 in that example (AWS Builders' Library, as of November 2026). Measure that percentile on your own network path, since cellular and Wi-Fi latency differ. For latency budgets on voice devices, see the realtime voice LLM latency budget.
One breaker per endpoint, saved across sleep
Azure cautions against one breaker for a resource that has several independent providers, because a failure in one would block calls to the others (Azure circuit breaker, as of November 2026). On a device, keep separate breakers for the broker, the LLM API and the update server. A battery device also sleeps. If breaker state sits in RAM that deep sleep clears, every wake starts closed and the device retries the dead endpoint again. Store the state, the open-until time and the current timer step in retained memory or flash. This is engineering reasoning.
How should LLM, MQTT and OTA endpoints be treated?
LLM APIs
LLM APIs say why they reject a call. Anthropic documents 429 for rate limits, 500 for internal errors, and 529 when the API is temporarily overloaded. A tier spend-cap 429 has no retry-after header and keeps failing until access resumes. Its SDK retries transient failures with exponential backoff twice by default and honors retry-after when present (Claude API errors, as of November 2026). Azure calls this accelerated circuit breaking, where the error response itself carries enough information to trip and stay tripped (Azure circuit breaker, as of November 2026). For how a shared account limits the whole fleet, see API rate limits for a fleet of devices.
MQTT brokers
Brokers expect clients to back off. AWS IoT Device Advisor has tests that withhold CONNACK and check that the device uses jitter backoff for at least five reconnects. In the server-disconnect and unstable-connection tests, a device with linear or constant backoff fails (AWS Device Advisor MQTT tests, as of November 2026). AWS also notes that publish requests over the per-connection quota of 100 per second are discarded (AWS IoT Core quotas, as of November 2026). Store-and-forward queues that flush all at once after a reconnect need their own rate limit.
OTA update servers
Firmware downloads are large, so a failed one costs more energy than a failed telemetry message. AWS IoT Jobs lets the cloud pace the fleet: in AWS's example, a rollout starts at a base rate of 50 per minute and grows by an increment factor of 2. An abort configuration cancels queued executions when a threshold percentage of devices fail, and a single device can have up to 10 retries across failure types (AWS IoT Jobs configurations, as of November 2026). The device side still needs its own breaker so a broken download stops early and the old image keeps running. See OTA update failures that brick devices and tracker OTA over LTE-M and NB-IoT.
How do you fault-inject retries on a battery-powered device?
Every connect attempt powers the radio, so a retry policy costs battery charge as well as server capacity. This is engineering reasoning. Azure lists failed operations testing among its considerations and notes the breaker can ping a health endpoint instead of relying only on a timer (Azure circuit breaker, as of November 2026). Run these tests on real hardware with a stub server you control and a current meter on the battery line.
- Block the broker so it never sends CONNACK. Does the device back off with jitter, and how many radio wake-ups does it spend per hour while blocked?
- Drop the connection right after each successful connect. Does the backoff keep growing, or does each short success reset it to the minimum?
- Return 429 with a retry-after header from a stub LLM endpoint. Does the device wait at least that long before the next call?
- Return 429 without retry-after, then 529, then 500. Does each error type move the breaker the way the design says?
- Hold the endpoint in a half-open test: let the first trial succeed and fail the second. Does the breaker reopen and restart its timer?
- Put the device into deep sleep while the breaker is open. Does it wake with the breaker still open, or does it forget and retry at once?
- Power-cycle a rack of devices at the same moment. Do their first connects spread out in time, or do they arrive in one burst?
- Fill the offline queue to its cap. Which messages are dropped, and is the drop logged for later upload?
- Fail an OTA download halfway, several times. Does the device report FAILURE, stop after its retry budget and keep running the old image?
- Measure battery current during a full outage of every cloud endpoint, and compare it with normal operation over the same period.
Log every state change with a timestamp and upload the log once the device reconnects. AWS recommends sending device logs to the cloud as events and alerting on devices that have not connected for an extended period (AWS IoT Lens, as of November 2026). For how to turn the current measurements into an honest battery life figure, see how to test and state battery life. For how the firmware work is organized and what you receive at handover, see firmware development.
Questions about circuit breakers on devices
Is a circuit breaker the same as exponential backoff?
No. Azure says the Retry pattern expects an operation to succeed eventually, while the Circuit Breaker pattern stops calls that are likely to fail. You can combine them, but the retry logic should stop when the breaker reports that the fault is not transient (Azure circuit breaker, as of November 2026).
Should the breaker live on the device or in the cloud?
Often both. Azure notes that service meshes can provide circuit breaking without changes to application code (Azure circuit breaker, as of November 2026). A cloud-side breaker protects your backend, but each device still spends radio time on calls the cloud will reject. A breaker in firmware saves that energy. This is engineering reasoning.
What should a device do while the breaker is open?
Azure lists options: degrade functionality, call an alternative operation, return a default value, or record requests in a journal to replay once the service is back (Azure circuit breaker, as of November 2026). On a device that might mean a cached answer, a short spoken message, or a queued event.
How can Duobay help design connection and retry logic for a device?
Duobay develops custom devices with a Shenzhen engineering team that covers electronics, firmware, enclosure, app and test, and builds samples and small batches. Firmware source and the SBOM are delivered to you, and source files are assigned to you at every paid gate. Initial calls are free, and we do not publish prices. Tell us about your device.
If you want the retry, breaker and fallback logic designed and tested for a specific device, tell us about your device.
Sources. All pages accessed November 14, 2026. SDK defaults, quotas and error codes change, so confirm current values before you set them in firmware.
- Microsoft, Azure Architecture Center, Circuit Breaker pattern (link)
- Amazon Web Services, Amazon Builders' Library, Timeouts, retries, and backoff with jitter (link)
- Microsoft, Azure IoT Hub, Manage device reconnections to create resilient applications (link)
- Amazon Web Services, AWS Well-Architected IoT Lens, Failure management (link)
- Amazon Web Services, AWS IoT Core Device Advisor, MQTT test cases (link)
- Amazon Web Services, AWS IoT Device SDK for JavaScript v2, DEFAULT_RECONNECT_MAX_SEC (link)
- Amazon Web Services, AWS IoT Core endpoints and quotas (link)
- Amazon Web Services, AWS IoT Jobs, How job configurations work (link)
- Anthropic, Claude API errors (link)
- Espressif, ESP-MQTT programming guide (link)
Planning a device like this? Tell us what it needs to do. A Duobay engineer replies within 1 business day. Send a project brief →