Context Rot: What Actually Happens to a Coding Agent

TL;DR. Context rot is the measured drop in an LLM's output quality as its input grows, well before the context window is full. The 2026 study everyone quotes is narrower than the headlines suggest: its famous 8/10 to 3/10 collapse is not statistically significant, and the one result in the same abstract that is has nothing to do with token counts. It's about handing the agent an external checklist. Meanwhile the largest measured gain I found in this literature didn't come from a compaction setting at all. It was moving the long context out of the window and onto the file system.

Every long-running agent I've built has hit the same wall. It starts sharp, works for an hour, and somewhere in the middle it quietly stops honoring a constraint I set at the beginning. Nothing errors. The output still looks like output.

I went and read the research recently, because I was about to change how my own pipelines work and wanted to know whether the folklore was true. Some of it is. A lot of what gets repeated is a misreading of a paper that says so itself.

What context rot actually is

Context rot is the degradation in an LLM's output quality as input length increases, independent of whether the context window is anywhere near full.

That last clause is the whole thing, so it's worth separating two failures that get conflated. Overflow is when you exceed the model's token limit: you get a hard error or truncation, and you know it happened. Rot produces no error at all. The model accepts the input, generates a response, and the response is worse in a way that looks like ordinary variance.

Anthropic's own cookbook on context engineering puts it plainly: "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases," and so "even before the hard context limit is reached, the agent may be getting less out of each token."

The term comes from a Chroma technical report published in July 2025 by Kelly Hong, Anton Troynikov and Jeff Huber. Their framing of the assumption being tested is the clearest sentence anyone has written on this: models are "typically presumed to process context uniformly, that is, the model should handle the 10,000th token just as reliably as the 100th. However, in practice, this assumption does not hold." Across their experiments, performance degraded as input length grew "often in surprising and non-uniform ways." The replication toolkit is MIT-licensed, which is more than most research in this space offers.

Why needle-in-a-haystack gave everyone a false all-clear

For a couple of years the standard long-context test was Needle in a Haystack: hide a sentence in a long document, ask the model to find it. Models started scoring near-perfectly, and a lot of people concluded that long context was solved.

Chroma's objection is that the test is easier than it looks. NIAH "typically assesses direct lexical matching, which may not be representative of flexible, semantically oriented tasks." Finding a string you can pattern-match is not the same as reasoning over a document while holding twelve constraints in mind. So they varied it: needles that match semantically rather than literally, haystacks whose content shifts, a conversational benchmark, and a task where the model just replicates a sequence of repeated words. Performance still fell off. Their own caution is that real work is harder than any of these, so "the influence of input length may be even more pronounced in practice."

There's an older result underneath this one too. Liu et al.'s Lost in the Middle (Stanford, TACL 2024) found accuracy tracks a U-shape against where the relevant fact sits: reliable at the start, reliable at the end, weakest in the middle. Position isn't free.

What the 2026 white-box study actually found

The paper driving most of this year's discussion is Yue Xue's How Agent Skills Fail under Long Contexts, posted 2026-07-20. It takes a real code-audit workflow, holds the task and 24 artifact checks fixed, varies the surrounding context, and classifies where failures first show up: lost requirements, editing drift, failed checking, and failures caused by the evaluator or runtime rather than the agent.

Here's the number that travels: "Codex with gpt-5.4-mini passes 8/10 runs in a 10,991-character clean context but only 3/10 in both a 299,140-character relevant context and an equal-length irrelevant context."

Two things sit in that same abstract and almost never travel with it.

The famous number is not statistically significant

The paper says so itself: "This 50-percentage-point difference is large but remains trend-level under two-sided Fisher tests (p = 0.0698)." Ten runs per condition. One model, one workflow. The author goes further and states that "a second task passes all clean and long runs, so the evidence does not support a universal context-length threshold," and explicitly disclaims the broader reading: "We do not introduce context rot, a new monitoring language, or a general theory of agent failure."

I don't think the effect is fake. I think a result quoted as proof of a general law is doing work its own author declined to do.

The result that is significant is about a checklist

Same abstract, different sentence: "A detailed external checklist passes 10/10 runs, compared with 5/10 for a generic self-check (p = 0.0325)."

That one clears significance, and I have seen it quoted almost nowhere. The intervention isn't a token budget. It's writing down what "done" means somewhere outside the model's head and checking against it.

Two more details worth holding onto. The 299,140-character relevant context hurt exactly as much as the equal-length irrelevant one, so "just keep your context clean" doesn't cover it; length itself was the variable. And requirement coverage stayed high while runs failed anyway: "98.8% in clean and above 92% in both long conditions." An agent can satisfy 92% of your requirements and still hand you a broken artifact. Averages hide the omission that matters.

What compaction actually costs

Compaction is the standard mitigation: summarize the conversation, replace it with the summary, keep going. In August, Rogerio Chaves at LangWatch published the most instrumented look at it I've seen, covering 287,748 API calls across 2,451 agents and threads over 162 consecutive days, plus 873 compactions and 201 hand-audited steps. It's one engineer's telemetry, and the paper is careful to call itself "a case study on a single subject with deep instrumentation, not a controlled experiment and not a multi-subject survey." Read it as a very good case study, not an industry benchmark.

The reframe I took from it: context "behaves like a recurring charge whose rate rises monotonically over the life of the session." Not a bucket you fill. A meter that runs faster the longer you go. Each doubling of context costs roughly 6x more, because 25 tool calls at a 500k window "is not one cached request to the LLM at 500k tokens, but 25 x 500k, and increasing."

The finding that should change behavior is about utility, not cost: "the part of the window the agent actually uses barely grows at all: sixteen times the window carries about the same amount of useful context." Sixteen times the window. Same useful content.

Compaction isn't free either. After one fires, "for dozens of steps after a compaction the agent makes visibly worse decisions." The measured recommendation lands between 250,000 and 450,000 tokens by task type, and in the write-up the author "could find no justifiable benefit in letting an agent run anywhere near 1M."

Where the tokens actually went

The origin story is the useful part. Chasing 2.25 billion tokens burned in a single day, 82% of it from subagents, he found 1.14 billion of it came from two subagents out of thirty-four. One ran 10.7 hours and made 1,301 API calls. Both were forks, and a fork "starts at the parent's context size," so they began at 523K and 304K tokens instead of the roughly 30,000 a fresh subagent gets. One peaked at 997,634 tokens, within 0.3% of the model's ceiling, and ran 1,112 bash commands from there. At its average context, a single pnpm test cost about 568,000 tokens before the tests did anything.

Nothing there is a bug. Every step is a reasonable decision that compounds.

The tools don't agree on when to compact

If there were a known-correct threshold, the tools would have converged on it. Across the seven surveyed below, they haven't. The best prior art here is wasnotwas's source-level survey of seven coding agents, each pinned to a version:

Agent	Version pinned	Compaction fires at
Gemini CLI	29b3aa8	50% of the window
Codex	2322e49	90%, configurable down, never up
opencode	22a4c5a	96%
Claude Code	v2.1.68	extract pattern
Roo Code	3e237e6	extract pattern
Pi	f430dce	extract pattern
OpenHands	bf769d1	different architecture entirely

The three thresholds above are the ones that survey calls out by number; the rest it groups by mechanism. Six of the seven do the same basic thing: a separate call summarizes the history, and the summary replaces it. The spread between 50% and 96% is, as it puts it, "the difference between compacting five times per long session versus once." Codex hard-caps at (w * 9) / 10 and, on OpenAI-hosted models, skips local summarization for a POST /v1/responses/compact endpoint that returns an opaque compressed representation.

Practitioner advice is scattered the same way: compact at 20%, at 50%, at 60%, once you cross 256K. Those posts can't all be right, and none of them measured anything.

The result nobody quotes: put the horizon on disk

Here's the paper I think deserves the attention the other one got. In March, a Duke group (Weili Cao, Xunjian Yin, Bhuwan Dhingra, Shuyan Zhou) published Coding Agents are Effective Long-Context Processors. The question they asked is whether long-context processing "can be externalized from latent attention into explicit, executable interactions, by allowing coding agents to organize text in file systems and manipulate it using its native tools."

Across five benchmarks spanning 188K to three trillion tokens, off-the-shelf coding agents "outperform published state-of-the-art by 17.3% on average." They attribute it to two things: native tool proficiency, meaning the agent runs code and terminal commands instead of issuing passive semantic queries, and file system familiarity, meaning it treats a huge corpus as a directory tree. In one example on a 385K-token transcript, the agent wrote a Python script, found its own domain-specific edge cases through failure analysis, and refined the script until it worked.

Read those two papers together and the compaction debate looks different. Tuning when to summarize is optimizing the accumulation. The Duke result says the accumulation was the choice.

How I structure long-horizon agent work

I want to be exact about my footing here: I have no benchmark of my own on this. What follows is a design I run, not a measured effect, and I'd rather say that than imply I proved something. I've since made that refusal the whole subject of a post on why nobody has a trustworthy number for AI productivity, where the most-quoted figure in the argument turns out to carry an out-of-date banner written by its own authors.

The systems I build for long work aren't one long conversation. They're pipelines of discrete steps, each defined in a markdown file, and the state between steps lives in files rather than in the window. A run has a working directory. Step 01 writes sources.md, step 02 reads it and writes brief.md, and so on to step 08. Each step file carries the same four headings: Inputs, Actions, Outputs, Exit criteria. A status.md records progress after every step, so the next step can pick up from what the last one left behind rather than from a conversation.

I arrived at that shape because long sessions kept drifting, not because I'd read a paper. Reading these two, the reasons it holds up are less mysterious than I assumed. Files as the handoff medium is the externalization the Duke paper measured. That per-step Inputs/Actions/Outputs/Exit criteria block is an external checklist, which is the intervention that actually cleared significance in the white-box study. Each step starting fresh means step 07 never inherits step 02's exploration, so there's no accumulation to compact.

This post is a worked example: it was produced by the engine that wrote this post, running that pipeline. The same instinct shows up in what I learned keeping LLM extraction honest, where the failure mode was also silent and also invisible to the check I'd have thought to write.

What I'd actually do

1. Cut what enters the window before tuning what leaves it. Tool results and file dumps are where the volume is, and clearing old ones is cheap when they can be fetched again on demand.
2. Give every unit of work an explicit external checklist. Not "review your work." A written list of what must be true at the end. It's the cheapest thing here and the only item with a statistically significant result behind it, though that result comes from one small single-model study rather than settled general guidance.
3. Don't fork a subagent from a fat parent. A fork inherits the parent's context, so a 500K parent spawns a 500K child that then grows. Start workers clean and pass them a file path.
4. Treat a compaction threshold as the last dial, not the first. If you need one, 250k to 450k is the only measured range I could find, and it comes from a single-subject study.
5. Assume the failure is silent. Coverage above 92% still shipped broken artifacts. Whatever verifies your agent's output should be a program, not the agent's own opinion of its work.

What I'd like to see next is somebody running the LangWatch analysis across a dozen engineers instead of one. The single-subject numbers are good enough to act on and not good enough to settle anything, and the gap between those two states is where most of the confident advice on this topic lives.

If you're running agents over long horizons and your experience contradicts any of this, I'd genuinely like to hear it. I'm @AliFoundry, and the other things I've been building are mostly agent-shaped these days.

FAQ

What is context rot, in one sentence? Context rot is the measurable decline in an LLM's output quality as its input grows longer, which happens whether or not the context window is close to full.

Is context rot the same as running out of context window? No. Running out of window is a hard failure: you get truncation or an error, and you know. Context rot is silent, and the output just gets worse in ways that resemble normal model variance.

When should I compact my coding agent? The only measured recommendation I could find is 250,000 to 450,000 tokens, from LangWatch's 162-day case study of one engineer's telemetry. Real agents ship with thresholds anywhere from 50% to 96% of the window, so there's no consensus to follow.

Does a bigger context window fix it? No. In LangWatch's data the useful portion of the window barely grew: sixteen times the window carried about the same amount of useful context, while each doubling cost roughly 6x more.

Does keeping the context relevant avoid the problem? Not on its own. In the 2026 white-box study, 299,140 characters of relevant context produced the same 3/10 pass rate as the same length of irrelevant context, so length mattered independently of relevance.