What is context collapse?
Context collapse is the drop in answer quality when a language model is given a long input whose parts depend on each other. A bigger context window lets you load more text, but it does not make the model read that text evenly. Answers that need several far-apart sections at once are the first to drift.
What are the four failure patterns?
- Lost intent. The model answers the literal question and misses what the question was for.
- Middle-window blindness. Material in the middle of a long input gets less weight than material at the start or end. Liu and colleagues documented this in “Lost in the Middle” (2023).
- Hallucinated synthesis. The model produces a confident conclusion that nothing in the document supports.
- Instruction dilution. Rules set at the top of the prompt lose force as the context grows.
Why doesn't a bigger context window fix it?
A model attends over tokens, and that attention is not uniform. It is shaped by position, distance and noise. Filling the window is not the same as reading the document. When the input is long and internally cross-referenced, the result gets less reliable, and a wrong answer drawn from a larger document sounds more authoritative.
How can I test for it in five minutes?
- Load a fifty-page contract into the model's context.
- Ask a question whose answer sits in one section, such as section 34. The model will probably get it.
- Ask a question that needs sections 2, 17 and 34 at once, for example whether an indemnity clause survives a termination right that is limited by a definition near the front.
- Check each answer against the document. Watch for the four patterns above.
What is the difference between RAG and long context?
Retrieval-augmented generation (RAG) looks up passages that seem relevant to a question and puts only those in the prompt. Long context loads everything and relies on the model's attention to find what matters. Each has trade-offs: retrieval can miss passages that only matter in combination, and long context can drift as the input grows. Both can be checked against the source.
What can I do about it?
Keep each question focused on the evidence it needs, check answers against the source, and for sources too large to prompt, use source-grounded context prepared once and reused. The full argument is in the essay The Competence Gap.
Frequently asked questions
What is context collapse?
The drop in answer quality when a language model is given a long input whose parts depend on each other.
Does a bigger context window fix context collapse?
No. A larger window lets you load more text, but attention across the input is still uneven, so answers that need far-apart sections can still drift.
What is "lost in the middle"?
A finding by Liu and colleagues (2023) that material in the middle of a long input is used less reliably than material at the start or end.
How do I test my model for context collapse?
Ask one question answered by a single section, then a question that needs several far-apart sections at once, and compare both answers against the source.
Build on it
Open the studio, compare plans, or request enterprise access.