Skip to content
Back to posts

Why Giving AI More Context Doesn’t Always Help

Notes by SDG ·

Why Giving AI More Context Doesn’t Always Help

Recently, while debugging a multi-file refactor with an AI assistant, its performance degraded noticeably as the session dragged on.

At first, keeping the full history seemed like an advantage. The assistant already had the codebase and my constraints in view, so there was less explaining to do. Instead, it began ignoring rules I had set twenty minutes earlier and repeating discarded approaches.

The slowdown alone couldn’t tell me what was happening inside the model. It did, however, send me looking into context management—and a question that becomes critical as AI takes on longer, multi-step work:

How do we decide what an AI system needs to know right now, what it should remember for later, and what it should stop carrying around?

What counts as context?

Context is the information available to the model when it produces a response. Your latest message is only part of it.

Depending on the application, that information can include earlier conversation, system instructions, documents, code, search results, tool outputs, user preferences, and details of the task in progress.

The context window is the model’s capacity to hold that information. Context management is the discipline of deciding what goes into it, how it gets there, whether it is still valid, and what happens to it afterward.

That is broader than writing a good prompt. A carefully worded instruction cannot make an outdated payment record current or recover a policy exception that the retrieval system never found.

It also helps to separate four distinct layers that often get lumped together as “memory”:

  • Conversation history: A log of what was said in the current session.
  • Persistent memory: Facts or preferences stored across sessions.
  • Organizational knowledge: Authoritative policies, product catalogs, and documentation.
  • Workflow state: Transactional records showing that an approval is pending or a payment has already been submitted.

These serve entirely different purposes. An agent remembering that someone said “the refund is done” is very different from the payment ledger confirming that the funds cleared.

A bigger window helps, but it doesn’t settle the problem

Larger context windows are useful. They make it possible to analyze larger codebases, compare lengthy contracts, or maintain continuity across complex tasks.

The catch is that fitting information into a window does not guarantee the model will reason over it reliably.

Early research like Lost in the Middle showed that models struggled to retrieve facts buried in the center of long inputs (aclanthology.org). While frontier models have largely solved basic needle-in-a-haystack retrieval, the failure mode has shifted toward reasoning degradation and distraction.

For example, the NoLiMa benchmark (ICML 2025) tested how models handle long contexts when the question and the evidence do not share obvious keyword overlap—requiring multi-hop reasoning rather than simple text matching (proceedings.mlr.press). As context grew, model accuracy dropped sharply.

In real-world agentic workflows, two common problems accelerate this degradation:

  • Tool-output bloat: Agents frequently call APIs that return massive, raw JSON payloads—such as full ERP records or verbose logs. Leaving tens of thousands of tokens of raw tool output in the history crowds out instructions and dilutes the model’s attention.
  • Contradictory and superseded state: A long session often contains abandoned approaches alongside the final decision, outdated drafts, or conflicting notes.

All of that fits comfortably inside a million-token window. Yet for an enterprise, having the information available and having the right, uncorrupted information for a decision are separate requirements.

What managing context actually involves

Effective context engineering relies on five core mechanisms, each solving a different bottleneck:

  • Retrieve information on demand. Retrieval-augmented generation (RAG) searches external sources and injects relevant material into the prompt. Hybrid retrieval combines keyword search (essential for exact SKUs or error codes) with semantic search (which captures conceptual similarity), followed by a reranker to prioritize the best matches. However, retrieval still fails if the decisive policy never makes the initial shortlist.
  • Compress and prune older material. Long interactions can be summarized into key decisions and open questions, while verbose, completed tool outputs are stripped out once their result is recorded (anthropic.com). The risk is over-compression: dropping a single exception, condition, or “not” can silently invert a business rule.
  • Externalize persistent memory. Keeping user preferences and historical facts outside the active prompt prevents bloating every turn. However, this introduces data-governance overhead: who corrects a mistaken memory, when does it expire, and should context learned in one workflow bleed into another?
  • Structure prompts for prefix caching. Prompt caching reduces the compute cost and latency of processing repeated, unchanged context (developers.openai.com). Because caching relies on exact prefix matching, changing a single dynamic variable near the top of a prompt invalidates the cache downstream. Engineers must place static instructions and reference documents first, keeping dynamic state at the very end—and even then, caching optimizes compute, not whether the cached material is still business-valid.
  • Decouple workflow state from the LLM. An agent operating over several days needs a durable record of completed actions, pending approvals, and transaction IDs that survives outside the chat window (docs.temporal.io). A conversational summary is a weak substitute for a transactional state machine.

Not every application needs every mechanism. A small, static knowledge base works well in a single cached prompt. A multi-step workflow across live enterprise systems requires strict state separation.

How the vendor ecosystem maps to the stack

Vendors address context management at different layers of the architecture. Enterprises rarely choose between a cloud provider and an orchestration framework; they combine them.

Sources: Microsoft, OpenAI, Anthropic, Google, AWS, LangGraph, LlamaIndex, Letta, Mem0, Zep.

None of these tools decide whether a customer support note should overrule a payment ledger, whether an old approval still applies, or whether one employee is authorized to see data retrieved for another. Those remain business-logic decisions.

The economics: tokens, latency, and failure recovery

Carrying large context increases token spend and time-to-first-token (TTFT) latency. Conversely, adding retrieval pipelines, rerankers, and compression steps introduces infrastructure costs and extra network hops.

To see the trade-off clearly, consider a workload of 10,000 customer cases per month, averaging 5 model calls per case (50,000 total calls). Assume each call generates 1,000 output tokens, priced at $3.00 per million uncached input tokens, $0.60 per million cached input tokens, and $15.00 per million output tokens.

Note: Output token spend is constant at $750/month across all four rows (50,000 calls × 1,000 tokens × $15/1M). Approach 4 evaluates 20% of calls at $0.003/call ($30) and includes $50/month for background summarization.

The numbers reveal an important nuance: if your context is static and reusable across turns, simple prefix caching ($2,490) beats a complex RAG pipeline ($3,050) on pure infrastructure cost.

However, pure token math misses two critical enterprise realities:

  1. Latency: Every retrieval query, reranker pass, and secondary LLM evaluation adds hundreds of milliseconds to seconds of delay. In synchronous voice or live-chat workflows, latency directly degrades user experience.
  2. Cost of failure: If a human escalation costs $20 to resolve, reducing escalations by just 1 percentage point across 10,000 cases saves $2,000—wiping out the entire cost difference between the cheapest and most expensive architectures.

A cheap, compressed prompt that drops a critical policy exception and triggers a wrong refund costs far more than carrying extra tokens. The right metric is cost per accurately completed business outcome, factoring in latency, retries, and human escalations.

A concrete example: tracking a missing refund

Consider a customer who buys an item online, returns it in-store, and contacts support three days later because the refund hasn’t appeared.

To handle this reliably, an AI agent needs access to the order record, the in-store return receipt, current payment gateway status, the return policy, and prior support notes. It does not need the customer’s five-year browsing history or unrelated marketing preferences.

Now trace what happens when the workflow spans multiple systems and hours:

  • Turn 1 (Investigation): The agent retrieves the order and return records. It sees the return was accepted, checks the payment ledger, and finds no refund transaction. It flags the issue for approval.
  • Overnight pause: While the ticket sits open, a billing specialist manually processes the refund in the ERP system.
  • Turn 2 (Next morning): The customer replies asking for an update. If the agent relies solely on its conversation summary (“Refund missing; awaiting approval”), it will give the wrong answer—or worse, trigger a duplicate refund.

Before taking or confirming any action, the system must query the authoritative payment ledger rather than trust conversational memory. Similarly, if an API call to issue a refund times out, the agent cannot rely on a remembered note saying “refund requested”—it must check the transaction status via an idempotency key before retrying.

Finally, access controls must govern every layer. If a supervisor asks the agent about the ticket, the agent can surface internal fraud flags; if the customer asks, those flags must remain strictly out of scope. A shared memory store must never become a backdoor around role-based permissions.

Should another AI decide what context matters?

Using a dedicated "context evaluator" model to inspect retrieved chunks, flag contradictions, and curate what the primary agent sees is an increasingly common pattern.

It works well for ambiguous, unstructured tasks—such as synthesizing messy customer call transcripts to see if a complaint matches a known product defect.

However, adding a second model introduces new failure modes:

  • Compound error: The evaluator can hallucinate a contradiction or discard a vital edge-case clause before the main model ever sees it.
  • Latency tax: Running a synchronous LLM judge on every turn doubles the reasoning overhead and slows down response times.
  • Judge bias: Research shows LLM evaluators exhibit measurable biases, such as favoring verbose text or specific formatting over factual precision (arxiv.org).

A better architecture divides the labor: use deterministic code and access rules for permissions, required fields, and live transaction checks; use hybrid retrieval to narrow down unstructured documents; and reserve model-based evaluation only for steps where semantic interpretation genuinely adds value.

Where this leaves the enterprise

My refactor session didn't fail because the model's context window was too small. It failed because discarded attempts, raw logs, and active constraints were all mixed together in the same flat stream with equal weight.

Enterprise AI systems face the exact same problem at scale. Context management is ultimately a systems-engineering and data-governance problem, not a prompt-engineering one.

Building reliable agents requires separating conversational fluency from operational truth: keeping static rules cached, pruning noisy tool outputs, enforcing permissions at the retrieval layer, and grounding every action in live transactional state rather than what the model merely remembers being said.