Document 07 · Quick learn Query heads ÷ KV heads · Convex curve · Inherited, not tuned

One dialfour settings

The biggest term in the cache formula, and the one lever you cannot turn at serving time — so it becomes a model-selection decision worth a factor of four.

“Thirty-two researchers, and the only question is how many share a shelf.”

4×

the saving on Llama 3.1 8B

75%

of the cache goes at the first step, 32→8

93.3%

MLA’s reduction on DeepSeek-V2

0 flags

that change it at serving time

One dial, three settingsnot three techniques

Document 06 showed that kv_heads is the biggest lever in the cache formula. This document is about that one term. The dial is: how many key-value heads exist for the query heads to share?

MHA · 2017

32 query heads, 32 KV heads. Every head has its own.

512 KiB / token

GQA · 2023

32 query heads, 8 KV heads. Groups of four share one.

128 KiB / token · the knee

MQA · 2019

32 query heads, 1 KV head. All thirty-two share it.

16 KiB / token · and a real quality drop

The arithmetic trap that catches everyone

8 KV heads on a 4,096‑wide model gives 8 × 128 = 1024 columns. It is not 4096 ÷ 8 = 512. Head dimension is fixed at 128; you multiply it by the head count.

Per head, a projection matrix is always d_model × 128, and only the number of heads changes. Get this wrong on a whiteboard and the rest of your answer collapses

The analogy — a reading room with a shared reference shelf

Thirty-two researchers each have their own question — those are the query heads, and they stay thirty-two in every version. MHA gives each a private copy of the whole reference collection: no contention, enormous storage. MQA gives one copy for all thirty-two: minimal storage, but some questions are badly served.

GQA gives one shelf per four researchers — the middle setting, and it turns out to be where almost all the benefit already is

What changes, and what does notthe two that stay still are the whole point

W_Q · unchanged

d × d, always 4096×4096.

All 32 query heads still compute their own scores

W_O · unchanged

d × d, always 4096×4096.

Identical across MHA, MQA and GQA

W_K · shrinks

d × (kv_heads × 128).

4096×4096 → 4096×1024

W_V · shrinks

d × (kv_heads × 128).

Same shape, same saving

Workedsaving = query heads ÷ KV heads · 32÷8 = 4× on the 8B · 64÷8 = 8× on the 70Bderive it, never quote a ratio

Why the quality cost is so small

Keys and values are redundant across heads

Many heads end up looking at similar material, so sharing the redundant part costs little. Queries stay genuinely diverse, and they are untouched — every one of the 32 query heads still computes its own scores and its own output.

A model trained with GQA adapts around it

The query heads never learn to depend on 32 distinct key spaces in the first place.

This is different from retrofitting, which does cost something to recover — next section

The curve is convex, which is why 8 won

32 → 8 removes 75% of the cache

8 → 1 removes 22 more

Almost all the benefit arrives early… and almost all the quality cost arrives lateGQA sits at the knee

Retrofitting GQAthe recipe, and the two wrong answers

You have an MHA checkpoint and you want GQA without paying for pre-training again. The GQA paper’s contribution is a recipe for exactly this — and it is a favourite follow-up because it separates people who read the paper from people who read a summary.

1 · Start

The existing key and value projections — one head per query head.

2 · Group

Into the size the target implies: fours, to go 32 → 8.

3 · Mean-pool

Each group down to one head. It beat both alternatives tried.

4 · Uptrain

Continue pre-training on ~5% of the original compute.

“Just delete the extra heads”

Wrong, and it sounds it. You would be throwing away three quarters of the learned key and value information rather than combining it.

Mean pooling was tested against exactly this — selecting one head per group — and beat it, and beat reinitialising from scratch

“It is basically free”

Five per cent of a frontier pre-training run is still a serious job with a serious bill.

And note the framing: GQA is normally an architecture decision made before training. Uptraining is the exception for checkpoints that already exist, not the standard path

MLAcompress instead of shrink

GQA reduces how many things you store. Multi-head latent attention keeps all of them and reduces how big each one is. Same goal, opposite direction.

GQA — store fewer things

Keep 8 full-size KV heads instead of 32.

The cache holds the tensors attention needs directly
No extra arithmetic at read time
4× on the 8B, 8× on the 70B
Slightly behind plain MHA on quality

Same goal.
Opposite direction.

MLA — store smaller things

Keep all 32 heads; project them jointly into one small latent and cache that.

Reconstructs K and V by projecting the latent back up
Extra arithmetic every step — favourable, because decode is bandwidth-bound
DeepSeek-V2: 93.3% KV reduction, 5.76× max generation throughput
Their ablation had MLA beating plain MHA on quality

The RoPE complication — the detail that shows you read the paper

Keys are rotated for position before caching, and a rotated key cannot be cleanly squeezed and unsqueezed. DeepSeek split the key into a compressed content part and a small separate part carrying the rotation.

Be measured about it, and say why

MLA is an architectural decision made before training, exactly like GQA. You cannot switch a GQA model to MLA with a flag. Support is also newer.

Check your serving stack implements the MLA kernels properly rather than falling back to a generic path — a naive implementation reconstructs K and V into something the same size as MHA, and you lose the entire benefit

The full ladderfive rungs, and the whole attention-memory story

MHA · 2017every query head has its own KV head · original Transformer, GPT-2, Llama 2 7B/13B

512 KiB

GQA · 2023query heads share in groups — the middle setting · all of Llama 3, Mistral, nearly everything since

128 KiB

MQA · 2019all query heads share one KV head · PaLM

16 KiB

SWA · 2023bound the cache at a fixed window instead of shrinking entries · Mistral 7B v0.1, Gemma 2

flat

MLA · 2024keep all heads, cache a compressed latent, reconstruct on the fly · DeepSeek-V2

~7%

And what each one costs you

MHA

Nothing in quality; everything in memory.

GQA

A fraction of a percent of quality.

MQA

A real, measurable quality drop — and training instability.

SWA

Genuinely lossy beyond the window; awkward with prefix caching.

MLA

Extra arithmetic per step; newer, less universally supported.

What to say when asked to choose

“For serving, I do not choose this — I inherit it by choosing the model, because it is baked into the weights. So it becomes a model-selection criterion: I would read num_key_value_heads from the config before anything else, because on an otherwise equal comparison a model with 8 KV heads serves four times the users of one with 32. If long context is central I would look seriously at an MLA model; if exact long-range recall is not critical, a sliding-window or hybrid design for the bounded cache.”

What I would not do is treat it as a tuning knob

Answer it out loudthe ninety-second version, then seven questions

“Multi-head, multi-query and grouped-query attention are one dial at three settings: how many key-value heads the query heads share. Only the KV side shrinks — the query projection and the output projection stay full width in all three, which is the whole mechanism, because queries are where the diversity lives and keys and values are highly redundant across heads. The saving is query heads divided by KV heads, so four times on Llama 3.1 8B and eight on the 70B. GQA won because the curve is convex: going from 32 heads to 8 removes 75% of the cache, and going the rest of the way to 1 removes only 22 more while costing most of the quality.”

Seven questions, and the spine of each answer

Architect

What problem does GQA actually solve?

KV cache memory. 32 query heads over 8 KV heads → 128 KiB per token instead of 512
That memory caps batch size, and decode throughput is a function of batch size
So it translates directly into users: 53 on an H100 instead of 13

One thing not to say: that GQA speeds up attention. Every query head still computes its own scores. The speed benefit is downstream — decode is bandwidth-bound, so a smaller cache moves fewer bytes. Saying it in that order reads as understanding.

Architect

How does GQA differ from MQA, and why did GQA win?

Same dial, different settings. MQA collapses to a single shared KV head
Maximum saving, but a measurable quality drop — particularly on reasoning — plus training instability
The curve is convex: 32→8 removes 75 points, 8→1 removes 22 more and most of the quality

Worth adding: GQA is presented as a generalisation of MQA, with MHA and MQA as the two endpoints of one continuum. Framing it that way is what a systems person sounds like.

Architect

Can you convert an existing MHA model to GQA?

Yes — mean-pool W_K and W_V within each group, then continue pre-training on ~5% of original compute
Demonstrated on public T5 checkpoints: quality close to the original, decode speed comparable to MQA
Mean pooling beat picking one head per group and beat reinitialising from scratch

Two things not to say: “just delete the extra heads” (explicitly tested against), and “it is free” (5% of a frontier run is a serious bill).

Architect

How is MLA different from GQA, and why did DeepSeek choose it?

GQA reduces the number of KV heads cached. MLA keeps them all and caches a small joint latent
Extra arithmetic per step for a much smaller cache — favourable, since decode is bandwidth-bound
DeepSeek-V2: 93.3% KV reduction, 5.76× max generation throughput

They chose it because their ablation had MLA beating plain MHA on quality where GQA sits slightly below — a much smaller cache without the concession. The RoPE split is the detail that shows you read the paper.

Eng manager

Two models look equivalent on benchmarks. How do you choose for serving?

Open both config files before looking at another benchmark
num_key_value_heads first — 8 versus 32 is 4× the users at the same quality score
Then layers (also a multiplier), then MLA or sliding window, then vocabulary size

Turn it into the number the business understands: users per GPU, and cost per million tokens. Two models that tie on a leaderboard can differ by 3–4×, and it is a half-hour calculation.

Architect

Why is 8 KV heads so common?

It sits at a good point on the quality-memory curve — most of the saving, almost none of the cost
And it maps cleanly onto eight-way tensor parallelism: one KV head per card on a standard node

The corollary, raised unprompted: if the parallelism degree exceeds the KV head count, heads must be replicated and the aggregate cache goes up. So the head count quietly caps how far you can usefully shard.

Architect

Does GQA hurt long-context quality more than short?

A fair worry — fewer distinct key spaces could plausibly mean coarser retrieval
What the evidence supports: quality close to MHA overall, and every frontier long-context open model uses GQA
They would not, if it broke retrieval at length

If someone wants a stronger claim, say you would want to see the evaluation — a needle-in-a-haystack test across the full window against an MHA model of the same family. Measurable, so measure it rather than have an opinion.

Fast answersthe eight that come up most

QDoes GQA reduce model parameters?

Slightly. On the 8B, attention parameters per layer fall from about 67M to about 42M because W_K and W_V shrink — roughly 4% of the whole model. Real, but an order of magnitude less interesting than the cache saving. Mention it second or third, never first.

QCan I change a served model’s KV head count with a flag?

No. It is baked into the weights — the shape of W_K and W_V. What is a runtime flag is the cache dtype: fp8 KV halves the same number and stacks multiplicatively with whatever head ratio the model already has.

QDoes GQA interact with tensor parallelism?

Yes, and it is a good detail to raise. With 8 KV heads and 8-way TP, each GPU holds exactly one — clean. If the degree exceeds the head count you must replicate heads across devices, which costs the memory back.

QIs “GQA gives an 8× saving” correct?

Only for models where it happens to be true. It gives query heads ÷ KV heads — 4× on Llama 3.1 8B and 8× on the 70B, within the same family. Quoting a single ratio as a general fact is wrong roughly half the time.

QWhat is sliding-window attention doing differently?

It bounds the cache instead of shrinking each entry. Each token attends only to the last W tokens, so the cost becomes flat in context length rather than linear. Genuinely lossy: a token outside the window is not attended to at all. Gemma 2’s alternating design is the interesting compromise.

QCould a model use different KV head counts per layer?

Nothing forbids it, and hybrid designs are an active area — Gemma 2’s alternation of local and global layers is this idea applied to the attention pattern. Mainstream open models keep it uniform, which is why the cache formula has a single layers multiplier.

QIs MLA going to replace GQA?

Unknown, and the honest answer is that it depends on whether the quality result replicates across labs and on tooling support catching up. What is clear is that it is a pre-training decision either way — track it as a model-selection input, not a migration plan.

QWhat about MoE — does it help the cache?

Not at all. Mixture of experts replaces the feed-forward block; attention — and therefore the KV cache — is completely untouched. It cuts operations per token and saves no memory whatsoever. MoE and GQA solve orthogonal problems and are frequently used together.

You do not choose this. You inherit it by choosing the model. The KV head count is baked into the shape of W_K and W_V and no flag changes it — which makes num_key_value_heads a model-selection criterion worth a factor of four in users per GPU. Two models that tie on a leaderboard can differ by 3–4× on cost per million tokens, and it is a half-hour calculation you can do before anyone commits.

Where this goes next

The four projection matrices, and where they sit 02 · AnatomyThe cache read this halves or quarters 05 · PhasesThe formula this term sits in 06 · KV cacheThe RoPE rotation that complicates MLA 08 · PositionThe fp8 cache that stacks on top of this 10 · Cache opsWhy the head count caps the tensor-parallel degree 11 · Many GPUsReading the config before you buy anything 14 · Planning

Questions to ask them

What is num_key_value_heads on the models you serve?

Was the KV head count part of how the model was chosen, or discovered afterwards?

If you use tensor parallelism, is the degree above or below the KV head count?

Have you evaluated any MLA or sliding-window models, and does your stack support them well?

For long context, has anyone tested retrieval across the full window?