Concept renderEvery request carries the system prompt, tool definitions, any history you resend, the new input and the reply budget, and all of it counts toward the window. Providers handle overflow differently, and recall degrades as context grows. Plan memory in layers: keep recent turns, summarize older ones, drop used tool output, and store durable facts as notes.
A device that talks with a person for weeks will outgrow any context window if it resends everything. Even before it does, a long prompt costs more, answers slower and recalls worse. This guide shows what three major providers document about limits, how that reads for a device, and a layered memory design for firmware and backend teams. It contains no measured result.
What do the providers document about limits and overflow?
Each row comes from the provider's own documentation page, read on October 31, 2026. Window sizes and behavior change, so re-read the page before you build.
| Provider | What counts toward the window | What happens at or near the limit | Source |
|---|---|---|---|
| Anthropic (Claude API) | The documentation states that a 1M window applies on current models. The system prompt, every message (including tool results, images and documents) and your tool definitions all count. | If the input alone exceeds the window, the API returns a 400 invalid_request_error ("prompt is too long") on every model. On newer models, input plus max_tokens over the window ends with stop_reason model_context_window_exceeded instead of an error. | Anthropic context windows page, as of October 31, 2026 |
| OpenAI (Responses API) | The page says the maximum number includes input, output and reasoning tokens. The window size per model is on the model pages, which this article does not quote. | Tokens generated in excess of the context window limit may be truncated. With previous_response_id, all earlier input tokens in the chain are still billed as input tokens. | OpenAI conversation state guide, as of October 31, 2026 |
| Google (Gemini API) | On Gemini 4, one 1M window is shared across input, thinking and output. On Gemini 3 and 2.5, the page describes a separate 1M input budget and a 64K output budget. | The page says longer input means higher latency and that multi-needle retrieval is less accurate. It names context caching as the primary optimization for long context. | Google long context guide, as of October 31, 2026 |
Prices are not in this table. Per-token prices change and sit on each provider's pricing page, which this article does not quote. The Gemini 4 model name and window wording should be re-checked against the page before you rely on them.
What does this mean for a device?
Each turn of a conversation adds to the history. If the device resends that history on every request, the input for each request keeps growing, and the total across a long conversation grows faster than the number of turns. Chaining does not change what you pay for. OpenAI states that previous_response_id still bills all earlier input tokens in the chain as input tokens. Context caching, which the Gemini page calls its primary optimization for long context, targets that repeated part.
The fixed part matters too. The system prompt and tool definitions ride along on every request, so a long tool list costs room on every turn. Measure your own token counts on a prototype before you decide anything. The measurement list below shows what to record.
What should a device keep, summarize, drop or store?
The Anthropic engineering guide on context for agents (source, as of October 31, 2026) describes compaction, tool result clearing and structured note-taking. The tiers below apply those ideas to a device. The mapping to companion and recorder products is our engineering judgment.
| Tier | What goes there | Why |
|---|---|---|
| Always in the window | System prompt, tool definitions, the current exchange, the last few turns verbatim. | These count on every request. Every token here is resent every turn, so keep it short. |
| Summarize | Older turns that still matter for the current topic. | The engineering guide says the art of compaction lies in choosing what to keep and what to discard, and that overly aggressive compaction can lose subtle but critical context. |
| Drop | Raw tool output that has already been used, repeated boilerplate, duplicate transcript text. | The same guide calls tool result clearing one of the safest, lightest forms of compaction. |
| Store outside the window | User preferences, stable facts about the user, open tasks. Keep them as short structured notes on the device or in the client cloud account and load only the relevant ones. | The guide describes structured note-taking as persistence outside the context window. |
The Anthropic page also lists server-side compaction (beta, on 4.6 and later models) and context editing, which clears old tool results and thinking blocks. Whether earlier thinking blocks stay in context depends on the model. If you use a provider feature instead of writing your own, read the page for your exact model and test the result.
For a companion, the notes store can carry the person's name, preferences and open tasks. For a recorder, it can carry the structure of the recording session: topics, timestamps and open tasks. Raw transcript retention follows your written retention rule. In both cases, tell the user what is kept and for how long.
What are the common traps?
These follow from the documentation quoted above. They are engineering reasoning about how the limits behave. None describes an incident at any company.
- Treating the window as free space. Everything in the request counts: system prompt, all messages, tool results, images, documents and tool definitions. A device that attaches a large tool list or long documents to every call shrinks the room for conversation before the user says a word.
- Filling the window because it is large. The Anthropic page says accuracy and recall degrade as token count grows (context rot) and that curating context matters as much as the space available. The Gemini page says multi-needle retrieval is less accurate and latency rises with input size. Treat the 1M window as the most you can send, and send only what the current turn needs.
- Assuming the platform trims history for you. The Anthropic page does not describe silent trimming of conversation history. That is an inference, not a quote. Plan as if your client owns what stays and what goes, and confirm the behavior for the exact endpoint you call.
- One overflow handler for three providers. Claude returns a 400 error for an over-long prompt and a stop_reason when input plus max_tokens passes the window on newer models. OpenAI may truncate tokens generated beyond the limit. Gemini 4 shares one window across input, thinking and output, while Gemini 3 and 2.5 split input and output budgets. Code that catches only one of these behaviors will mishandle the others.
- Paying for history again and again. OpenAI states that with previous_response_id all earlier input tokens in the chain are billed as input tokens. Chaining saves you from resending text. It does not remove the cost of reading it.
- Over-aggressive summaries. The Anthropic engineering guide warns that overly aggressive compaction can lose subtle but critical context. A summary that drops a name, a date or a stated preference can break a companion conversation in a way the user notices later. Keep a structured notes store for facts that must survive.
- Keeping everything a recorder hears. For recording products, what the device keeps in memory is also a privacy decision. Keep a written retention rule, tell the user what is kept and for how long, and never design for covert capture. This is engineering guidance; ask counsel to confirm the retention terms for your market.
What should you measure on your own board and backend?
The documentation does not tell you your token counts, your summary quality or your latency. This article reports no measurements. The list below says what to measure, how to capture it and which raw data to keep. Set the pass line yourself from your product requirements, and record what you based it on.
| What to measure | How to instrument or collect it | Raw data to record |
|---|---|---|
| Tokens per turn | Log each prompt and reply from prototype sessions, using the provider token counting endpoint or tokenizer. | System prompt and tool definition token count, user input tokens, reply tokens (including reasoning tokens where the model produces them), history tokens resent, number of turns, which conversation each row came from. |
| Window headroom | Replay your longest logged conversation and count the tokens in every request before sending it. | Tokens in each request, the window size from the provider page and the date you read it, the max_tokens you reserved, the turn where compaction fired. |
| Overflow behavior per provider | Send an over-long request to each provider you support, from a test account. | HTTP status, error type, stop reason, whether output was truncated, what the device showed the user, whether the user turn was kept. |
| Memory strategy comparison | Write one fixed scripted conversation. Run it with full history, with a rolling summary and with structured notes, on the same models. | The script, model names and dates, tokens per turn for each strategy, the scoring rubric written before scoring, who scored, and every answer returned. |
| Summary fact retention | Seed the script with a short list of facts, such as a name, a preference and an open task. After summarizing, ask questions that need each fact. | The seeded fact list, the summary text produced, which facts the later answers used correctly, and which they missed. |
| Latency by input size | Time the first token and the full reply from the device on its real network path, at several input sizes you choose. | Input token count, time to first token, time to full reply, network type, timestamp, and the provider region if known. |
| Connection loss mid-conversation | Cut the connection during a turn on a prototype and bring it back. | What the device did with the pending turn, what the user saw, whether any text was lost or sent twice. |
Write each pass line before you run the test, so the result cannot move it.
What should you answer before you commit?
These are yes or no questions. A no marks a gap to close before you fix the memory design.
- Do we know the exact model and endpoint we call, its window on the provider page, and the date we read it?
- Do we know how many tokens our system prompt and tool definitions use on every call?
- Does our code handle what each supported provider does when a request is too long?
- Is there a written rule for what is summarized and what is dropped?
- Have we listed the facts that must survive across sessions, and decided where each is stored?
- Do we know what we resend each turn, and whether chaining responses still bills the full history?
- For recorder and companion products, have we written down how long raw audio or transcript is kept, and does the user see it?
- Have we looked at context caching or an equivalent on the provider pages?
- Do we know what the device does when the connection drops mid-conversation?
- Can we say which numbers in our memory design were measured on a prototype and which are still guesses?
Questions about context windows and device memory
If the window is 1M tokens, do I still need to manage memory?
Yes. The Anthropic page says accuracy and recall degrade as token count grows and that curating context matters as much as available space (source, as of October 31, 2026). The Gemini page also says latency rises with input size.
What happens when a conversation passes the window?
It depends on the provider. Claude returns a 400 error when the input alone is too long, and newer models stop with a stop reason when input plus max_tokens is too long. OpenAI says tokens generated beyond the limit may be truncated. Test each provider you plan to use.
Does chaining responses make long conversations cheaper?
The OpenAI page says no on billing. With previous_response_id all earlier input tokens in the chain are still billed as input tokens (source, as of October 31, 2026). The Gemini page names context caching as its primary optimization for long context.
What belongs in device memory versus the prompt?
Keep the current exchange and recent turns in the prompt. Keep stable facts and preferences as short structured notes, and load only the relevant ones into each request. The tier table above lays this out.
Can Duobay tell me which memory design fits my product?
A Gate 0 Feasibility study delivers a system architecture document for your device. Tell us about your device and we will explain how it works.
If you want to talk through the memory design of your device, tell us about your device. You can also read how Gate 0 Feasibility works, or our AI recording wearable work.
Sources
All pages accessed October 31, 2026. Window sizes and behavior change, so re-read them before you decide.
- Anthropic, Context windows: https://platform.claude.com/docs/en/build-with-claude/context-windows (accessed 2026-10-31)
- OpenAI, Conversation state: https://developers.openai.com/api/docs/guides/conversation-state (accessed 2026-10-31)
- Google, Gemini API long context: https://ai.google.dev/gemini-api/docs/long-context (accessed 2026-10-31)
- Anthropic, Effective context engineering for AI agents: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents (accessed 2026-10-31)
Planning an AI recording wearable? Tell us what it needs to do. A Duobay engineer replies within 1 business day. Send a project brief →