Context length is quoted as a single number, which encourages a false model of how it behaves: a bucket that either holds your document or does not. What actually happens is that a fact placed at the start or the end of the context is found reliably, and the same fact placed in the middle of a long context often is not.
The shape is stable enough to be worth internalising. The opening of the context is attended to strongly. Training data puts instructions and framing there, and positional encodings treat early positions differently, and it is not settled how much each contributes. The most recent tokens dominate because that is exactly what an autoregressive objective trains for. Between those two anchors, every additional token competes for a fixed attention budget, and the middle sags.
Worth separating this from attention sinks, which are a different thing despite also involving the first token. A sink is where a head parks weight it has nothing useful to do with, and it attaches to the position rather than to whatever token occupies it. Mass sitting on a position for that reason does not help you retrieve the content there.
Extend the context and the sag deepens. That is the part that matters for design, because it means the failure is not "the document did not fit". The document fit. The model read it. The fact in the middle was simply not used.
Three consequences follow directly. Put the instruction you actually care about at the end of the prompt, not buried after a long document. Retrieval and reranking are not obsoleted by long context, because putting fewer, better chunks in a short context beats putting everything in a long one. And when you evaluate a long-context model, sweep the position of the fact rather than testing it once, or you will measure the best case and ship the average.
Two caveats on magnitude. The published measurements are milder than the extremes reachable here: the original multi-document QA work reports something closer to a twenty-point swing rather than the near-total collapse the weakest profile draws, so treat the shape as the lesson and the depth as a dial. And the effect is strongly model-dependent. Simple single-needle retrieval is close to saturated on current frontier models, and what persists is degradation on tasks that need several facts from different places at once, which is exactly the case a single-needle test fails to measure.
The curve here is a parametric fit to the published needle-in-a-haystack shape rather than a measurement of any particular model. The shape generalises across models and is what the diagram is for. The exact percentages do not.
The maths
- The parametric shape
Position p runs from 0 (start) to 1 (end). Two exponentials model primacy and recency; whichever anchor is closer dominates, and the floor f is what remains in the middle. The recency constant is the smaller of the two, so the bump at the end of the context is narrower than the one at the start and the worst position sits past the halfway mark rather than at it.
- Dilution with length
The floor drops with the log of context length relative to a comfortable N0 of 4k tokens, scaled by the model’s robustness rho. This is why the same fact at the same relative position is found less often in a longer context.
Related terms
- Context windowThe context window is the maximum amount of text, measured in tokens, that a model can consider for a single request. It is a hard ceiling, and it is the main resource you manage when working with an agent.
- Lost in the middleLost in the middle is the well-known tendency for models to attend best to the start and end of a long context and to miss information buried in the middle.
- Attention degradationAttention degradation is the quality drop a model shows as its context grows: recall weakens and it misses or confuses buried details, often well below the hard token limit. It is also called context rot.
- ContextContext is all the text a model can see for a single request: the system prompt, your message, the conversation so far, and any files or tool output the agent has pulled in. It is the only thing the model knows about your specific situation.