How order gets into a mechanism that has none, why a longer window is a model change rather than a setting, and what 128k actually costs in four places at once.
Every query against every key. Nothing in it knows which token came first.
vs “The man bit the dog”. Very different sentences.
Attention computes exactly the same thing for both.
Position must be injected deliberately — and how decides how far you can stretch later.
Each card carries a word and everything the model knows about it. Attention is a researcher who can compare any card to any other, in any order, as often as they like — and the pile is genuinely unordered. Somebody has to write a position on each card before it goes in.
Used by essentially every modern open model. It does not add anything — it rotates the query and key vectors by an angle proportional to the token’s position.
Treat each pair of dimensions as a dial.
Turn a lot per position — they distinguish neighbours precisely.
Barely move per position — they separate tokens thousands apart.
A query rotated by m meeting a key rotated by n depends only on m − n.
No bookkeeping, no learned parameters. Applied to queries and keys only, never values — position decides what to attend to, not what gets handed over. And inside every layer, at the point Q and K are formed, not once at the bottom.
One: the rotation is applied to a key before that key goes into the cache, so what is stored is already position-baked — which is why compressing the cache collides with RoPE and why MLA needed a split key.
Two: the scheme decides whether the window can be extended later — a capacity and roadmap question.
Three: the multi-rate design is what makes extension possible at all.
“Positions 1 and 3 are 90° apart. So are 5 and 7.” True only for the one frequency band where the per-position angle happens to be 45°. There are 64 bands in a 128-dimensional head, all rotating at different rates.
The model has seen rotation angles only in that range — and calibrated every relationship against them.
At position 25,000 the rotations reach angles it has never encountered.
It collapses — and not gracefully.
RoPE gives relative positions for free but does not extrapolate.
Setting --max-model-len 131072 on a model that declares a 131,072 window is fine — it was trained or adapted for it. Setting it on a model trained to 8k is not, and the server will happily let you.
The differences between them are exactly about the frequency bands — which is why the multi-rate design had to come first.
Scale all positions down to fit the trained range.
Raise the base so slow bands stretch further while fast ones are left alone.
Band-by-band treatment plus an attention-scaling correction.
Once a band’s wavelength is shorter than the context, it has wrapped round and can no longer distinguish absolute distance — two tokens 60,000 apart look the same as two tokens 6,000 apart on that dial. With base 500,000, a 128k window sits comfortably inside the range where even the slowest band is still monotonic.
Raising the base is the cheap part. Llama 3.1 also ships a frequency-banded scaling config and was trained for the longer window.
Its job is to distinguish adjacent tokens, and for that a wavelength of about six positions is exactly right. It is supposed to wrap. The band that must not wrap is the slowest one, because that is what carries long-range absolute distance.
1 GiB → 16 GiB on the 8B. 40 GiB on the 70B.
Worst case on one H100: 53 → 3.
Deep in the quadratic regime — the 8B crosses at 30,633.
At 131,072 tokens the cache is 52% of every step.
Cuts per-token cache by the head ratio — 4× on the 8B, 8× on the 70B.
Halves it again.
The S×S score matrix is never materialised.
Nobody pays for tokens they have not sent.
A long window is a capacity claim, not a quality claim. Retrieval from the middle of a very long context is a known weak spot across models, and an advertised 128k does not guarantee the model uses the middle of it well.
What is RoPE, in one breath?
Why can I not just set a bigger context length?
Why did Llama 3 change rope_theta from 10,000 to 500,000?
Product wants 128k context. Walk me through it.
How does RoPE interact with the KV cache?
QIs RoPE applied to values too?
No — queries and keys only. Position decides which tokens to attend to; the content handed over when a token is selected does not need rotating. And it is applied inside every layer.
QDoes a longer window mean the model uses it well?
Not necessarily, and this is the most important caveat here. Retrieval from the middle of a very long context is a known weak spot. Treat an advertised window as a capacity claim and test retrieval across its full range before relying on it.
QAre “positions 1 and 3 are 90° apart”-style claims right?
Only for the one frequency band where the per-position angle happens to be 45°. There are 64 bands in a 128-dimensional head. Stated as a general fact it makes RoPE look like a single rotation — and then band-aware extension makes no sense.
QWhat is the serving cost of long context, in one line?
Linear in tokens for the cache, quadratic in tokens for prefill. A 70B with GQA costs 320 KiB per token, so 128k is about 40 GiB — for one user. Which is why long context in practice means GQA or MLA, plus a quantised cache, plus prefix reuse, all together.
QCan I convert a short-context model myself?
Technically yes — the methods are published and fine-tuning is far cheaper than pre-training, especially with YaRN. Practically it is a training project with an evaluation burden, and unless long context is core to your product you are better off selecting a model that already declares the window.
QWhy does the fastest band wrap almost immediately?
Because its job is to distinguish adjacent tokens, and a wavelength of about six positions is exactly right for that. It is supposed to wrap. The band that must not wrap is the slowest, because that carries long-range absolute distance.
QDo other position schemes exist?
Yes — learned absolute embeddings in older models, relative position biases in the T5 family, ALiBi which adds a distance-proportional penalty. RoPE won for decoder-only models because it gives relative positions for free, composes cleanly with the KV cache, and extends reasonably well with the right rescaling.
QDoes sliding-window attention solve the position problem?
No — a different axis. A window bounds how far back a token can attend, which bounds the cache; RoPE encodes where a token sits. They compose: Mistral 7B v0.1 used RoPE and a 4,096-token window. Bounding attention does sidestep extrapolation — at the cost of not attending to those tokens at all.
QWhy does the cache complicate context extension?
Keys are rotated before being cached, so the stored tensor already carries its position. Any rescaling has to be consistent with what is already in the cache, and any attempt to compress it runs into content and rotation being entangled.
QWhat single thing should I check before trusting a window?
max_position_embeddings, then rope_theta, then any rope_scaling block. If the scaling block declares an original_max_position_embeddings well below the headline, the window was reached by rescaling plus fine-tuning — normal, and the signal to run the retrieval test.
A longer window is a model change, not a setting — and a long window is a capacity claim, not a quality claim. Positions past the training length produce rotation angles the model has never seen, so quality collapses rather than degrading. And once the model can do it, 128k still costs sixteen times the cache per user, takes concurrency from 53 to three, and puts prefill deep into the quadratic regime. Test retrieval at 10, 25, 50, 75 and 90% of the range before you advertise the number.
What window do your models declare, and what were they actually trained for?
Has anyone tested retrieval across the full window, or is the number from the card?
What fraction of real requests uses more than a quarter of the limit?
Is long-context traffic on the same deployment as everything else?
What is rope_theta, and is there a scaling block in the config?