What the bits in a float actually buy, why the nominal bit width is never the real one, and which of five methods to reach for on which hardware.
Quantisation is usually explained without explaining what a float is, which turns the whole topic into memorisation. A floating-point number spends its bits on two things.
How big or small the value can be.
How finely it can be distinguished from its neighbours.
fp16 has five exponent bits, which caps the largest representable value at about 65,500. During training, activations and gradients routinely exceed that, so fp16 needs loss scaling and careful handling. bf16 spends the same sixteen bits differently: eight exponent bits, the same as fp32, so the same enormous range — and only seven mantissa bits.
One scheme uses most of the boxes for decimal places — brilliant for anything between 0.001 and 100, and unable to write down a million at all. The other reserves boxes for an exponent — it can write anything from a trillionth to a trillion, but only to three significant figures.
Everybody in a normal group rounds to the same value. Useless.
All between 1.5 and 1.9 m — fit the same 16 marks across just that band.
Typically 32 to 128 weights that sit together.
Stored alongside — and that is the catch.
4-bit formats store a scale — usually 16-bit — and often a zero-point, per block of 32 to 128 weights. And mixed-precision recipes keep sensitive tensors at higher precision.
Smaller blocks track the local range more tightly, so quality is better, and cost proportionally more overhead — a group of 32 has four times the scale cost of a group of 128.
Round-to-nearest has three weaknesses. Every named method is an attempt to fix one of them — and being able to say which is a far better answer than listing acronyms.
Per-block scale from the largest magnitude; round to the nearest level.
Quantise weight by weight, adjusting the not-yet-quantised weights to compensate for each rounding error.
Find the ~1% of channels carrying large activations and scale those weights up before rounding.
Mixed precision inside a layer — attention and output tensors keep more bits than the FFN.
Cast to native 8-bit float; Hopper and later multiply it directly, with no unpacking.
It nudges the not-yet-quantised weights to compensate for each rounding error it introduces — like shifting people in a group photo when one has to move.
It watches the activations, finds the roughly one per cent of channels carrying unusually large signals, and scales those weights up before rounding so they land where they should.
A one-off job producing a smaller file. Same architecture, same tokeniser.
In VRAM, for the whole session. Never expanded back to full size in memory.
To 16-bit inside on-chip memory, used, and discarded.
Multiplied natively on Hopper-class hardware — no unpacking at all.
Learned during training. Never change. One copy serves everyone.
Calculated from your input. Change every single request. Yours alone.
Activations that you chose to keep.
Activations were 16-bit all along. Getting this direction right is a genuine signal.
fp8. No calibration data, minutes to produce, near-lossless, and multiplied natively.
AWQ at group size 128.
GGUF Q4_K_M.
The honest answer to “does quantisation make the model dumber” is: a little, it depends on the method and the bit width, and you measure it on your own workload rather than trusting a general claim.
It degrades unevenly. Aggregate scores can hold up while one capability falls off a cliff — frequently long-context retrieval, multilingual output, or exact formatting. Test the slices separately, not just the mean.
Below about 4 effective bits, ask a different question.
What does quantisation do, and what does it cost?
GPTQ or AWQ?
Does a 4-bit model run four times faster?
Can we halve our GPU bill by quantising?
A 70B at 3-bit, or an 8B at bf16?
QWhy is bf16 the format most weights ship in?
Same sixteen bits as fp16, split differently: eight exponent bits instead of five, so fp32’s dynamic range with less precision. Networks tolerate imprecision far better than overflow, so that trade is much more forgiving during training.
QDoes quantisation make the model dumber?
A little, and how much depends on method and bit width. fp8 is nearly indistinguishable; good 4-bit loses a small measurable amount; very low widths degrade noticeably. The honest answer is that it is a trade you measure on your own workload.
QCan I quantise a model myself?
Yes. Data-free methods — GGUF conversion, an fp8 cast — run on an ordinary machine in minutes to hours. Calibration methods (GPTQ, AWQ) need a GPU and can take many hours, because they run the model forward on sample text repeatedly.
QIs quantisation the same as pruning or distillation?
No. Quantisation keeps every weight but stores each with fewer bits. Pruning removes weights. Distillation trains a smaller model to imitate a bigger one. Quantisation is the only one that leaves the architecture untouched — which is why it is an afternoon rather than a project.
QWhat does group size control?
How many weights share one scale factor. Smaller groups track the local range more tightly and cost proportionally more overhead — a group of 32 has four times the scale cost of 128. 128 is the usual default.
QWhy does loading a model take so long?
Most of it is reading tens of gigabytes off disk. Which is a real argument for quantisation that nobody makes: a 4-bit model is a quarter of the bytes, so cold start is roughly four times faster — and cold start is what makes autoscaling an LLM hard.
QCan I mix fp8 weights and an fp8 cache?
Yes, and think of them as independent levers that multiply. fp8 weights free memory that becomes KV budget; an fp8 cache halves what each user needs from that budget. Evaluate them separately, because their error modes differ.
QWhat is SmoothQuant, and where does it fit?
It addresses the same activation-outlier problem AWQ does, but for activation quantisation — migrating the difficulty from activations into weights by rescaling both. Weight-only is far more common in serving because it captures most of the memory benefit.
QWhy is my quantised model slower than expected?
Most likely the kernel. A quantised format only gets its speedup if the server has a fused kernel reading the packed weights directly; a generic path that dequantises into a full-size buffer gives you the memory saving and none of the speed. Be suspicious if throughput barely changed while memory did.
QOne sentence on when to quantise?
When memory is the binding constraint. If cache utilisation is pinned and requests are queueing for blocks, quantising the weights directly buys concurrency. If the card is half idle, quantisation is solving a problem you do not have.
The nominal bit width is never the real cost. A 4-bit format stores a scale — usually 16-bit — and often a zero-point, per block of 32 to 128 weights, and mixed recipes keep sensitive tensors higher. AWQ at group size 128 is 4.16 effective bits; GGUF Q4_K_M is 4.90. Size from the published file, never from the number in the name — the 20% gap is exactly what turns “it fits” into an out-of-memory failure on the first request.
What precision are you serving, and was it chosen for memory or for speed?
Has anyone measured the quality drop on your own traffic, per slice rather than on average?
Do you size models from the published file, or from nominal bit width?
Is the engine using a fused quantised kernel, or dequantising into a full-size buffer?
If you are on Hopper or later, is there a reason you are not on fp8?
Has anyone compared a smaller model at higher precision against the large one at 4-bit?