Why LLMs are an unusually good — and unusually risky — quantization target
INT8, INT4, and sub-byte formats substitute for the FP16 or BF16 representations that typically parameterize large language models. The point is not to make the model “smaller” in an abstract sense. It is to reduce the number of bytes the accelerator must move from memory for every token generated. LLM inference is overwhelmingly memory-bandwidth-bound rather than compute-bound, which makes any technique that reduces bytes-per-parameter act almost directly as a throughput multiplier.
That memory-bandwidth fact is what makes LLMs an unusually attractive quantization target. It is also what makes the accuracy story unusually easy to misread: a benchmark that reports “INT4 quantization with negligible accuracy loss” can be entirely correct on the workload it measured, and entirely misleading for the workload you intend to deploy. The temptation to read a single number as a deployment guarantee is exactly the trap that the framing of quantization as controlled approximation rather than model damage is meant to defuse.
Why does memory bandwidth dominate LLM inference?
Token generation demands a full traversal of model weights from accelerator memory during the forward pass. For a 70B-parameter model in FP16, that is roughly 140 GB of weight data the accelerator must fetch through its memory subsystem before it can emit one token. Even on accelerators with multiple terabytes per second of memory bandwidth — per NVIDIA’s published HBM specifications for H100-class parts — the memory access cost dominates the per-token wall-clock budget for the autoregressive generation phase. This is an observed pattern across the inference stacks we work with: TensorRT-LLM, vLLM, and the various PyTorch-native serving paths all converge on the same bottleneck shape once batch size drops into the latency-sensitive regime.
Reducing each weight from 16 bits to 8 bits halves the bytes that must be moved. Reducing to 4 bits divides them by four. Because the bottleneck is bandwidth and not arithmetic, the throughput improvement tracks the byte reduction more closely than it would for a compute-bound workload such as image-model training, where the arithmetic dominates and lower precision does less to lift the wall-clock cost.
This is why quantization is a first-order economic lever specifically for LLM inference, not because lower precision is intrinsically virtuous. The improvement comes from the fact that LLM inference happens to be exactly the workload shape where bandwidth reduction translates directly into latency and concurrency. The arithmetic units on the accelerator spend a meaningful fraction of every token cycle waiting on HBM; lowering the bytes-per-weight cuts that wait roughly in proportion. The same wait shape shows up clearly when GPU utilization is contrasted with sustained throughput, where high utilization can coexist with low token rates because the SMs are stalled on memory rather than compute.
Where the accuracy story breaks
A single aggregate metric cannot describe how quantization affects accuracy across diverse tasks. Quantization accuracy is task-conditional and metric-conditional in ways that aggregate scores conceal.
The same quantization scheme that preserves performance on multiple-choice benchmarks — where the model only needs to produce a correct token in a constrained position — can degrade long-form generation, code generation, or behaviour on out-of-distribution prompts substantially. Multiple-choice tasks are forgiving: they ask the model to pick the highest-probability completion among a small set of candidates. Long-form generation compounds errors token by token, and small per-token probability shifts at one position can cascade into qualitatively different outputs many tokens later. Code generation is even less forgiving, because syntactic correctness is binary at every token boundary, and a single mis-quantized weight that nudges a brace token below threshold can break an entire output.
The implication is that a quantized LLM evaluated on a benchmark suite oriented toward multiple-choice reasoning may report less than one percentage point of accuracy loss while showing materially worse behaviour on the workloads that actually matter for a production deployment. The quantization scheme has not failed in any absolute sense — the evaluation has failed to measure the workload that will be deployed. In our experience, this is the most common mode of disappointment when teams adopt a published INT4 recipe and find that customer-facing generation quality drifts in ways the benchmark line never predicted.
Comparing quantization-evaluation regimes
| Evaluation regime | What it measures | What it misses |
|---|---|---|
| Multiple-choice benchmark accuracy | Per-position token correctness in constrained outputs | Compounding error in long generations, code, OOD behaviour |
| Aggregate perplexity on held-out text | Average per-token likelihood across a reference corpus | Tail behaviour on rare or workload-specific input distributions |
| Long-form generation quality (human or LLM-judged) | Coherence and correctness across many-token outputs | Latency cost, throughput characteristics under load |
| Workload-matched evaluation | Behaviour on a sample of the actual deployment workload | Higher cost to set up; less directly comparable across reports |
Realistic workload evaluation provides deployment-ready data, yet remains uncommon in published results. It is expensive to construct, it is rarely portable across organisations, and it does not produce a single headline number — which is exactly why we treat its absence as the diagnostic signal it is.
What this means for benchmarking quantized LLMs
Treating quantization as a production choice rather than a leaderboard entry requires transparent reporting of methods and tradeoffs. It has to specify which weights and activations were quantized (weight-only, weight-and-activation, or with a separately-quantized KV cache), which calibration data was used, which calibration method (GPTQ, AWQ, SmoothQuant, simple min-max, percentile clipping), and which evaluation workload — including whether the evaluation included long-form outputs or only constrained tasks. The calibration choice is not cosmetic; it is one of the dominant drivers of the per-layer error distribution, and an evaluation that fails to disclose its calibration and harness cannot be mapped onto any specific deployment.
There is a procedural half to this that gets less attention than the numerical half. Approximation discipline starts before any number is produced: the models are prepared once, ahead of knowing which device will run them, and the identical prepared artefacts are then handed to every vendor’s hardware. Where two vendors’ kernels disagree about what an approximation is permitted to do, the stricter constraint is applied to all of them rather than negotiated per vendor. Correctness criteria are enforced per test type and per precision, so an approximation that has gone too far is recorded as a zero rather than reported as a fast result. None of that validates a reader’s own quantization scheme — it is a demonstration that the bookkeeping around an approximation can be written down instead of assumed.
Benchmark reproducibility for quantized LLMs is therefore not a matter of citing a precision format. It is a matter of disclosing the full quantization tool chain and evaluation workload alongside the throughput and accuracy numbers. Two reports of “INT4 quantization with X% accuracy loss” that differ in tool chain and evaluation workload are not comparable, and the difference between them is not noise — it is methodology.
The benchmark question for a quantized LLM is therefore not “what accuracy did it lose?” but “on which workload, with which quantization scheme, with which calibration, and with which evaluation harness, did this accuracy number arise?”
The framing that actually helps
Each token’s generation cost balances bytes transferred against the fidelity of numerical computation. The trade is favourable for many production workloads because LLM inference is bandwidth-bound, but the favourability is conditional on the deployment workload tolerating the specific accuracy regression that the chosen quantization scheme produces.
The same principle holds here as elsewhere: quantization is a calibrated, bounded trade, not a one-way degradation, with a deployment cost that is measurable rather than mythological. The LLM-specific point is that the bandwidth bottleneck makes the throughput payoff unusually large, which makes the temptation to under-evaluate the accuracy side unusually strong. The character of the per-weight numerical error itself can be reasoned about analytically; what cannot be reasoned about analytically is whether the deployment workload tolerates the cumulative effect.
A quantization claim for an LLM that does not name its evaluation workload is not wrong because the number is wrong. It is incomplete because the number, on its own, does not describe the deployment behaviour the workload owner needs to know. Before an “INT4 with negligible loss” claim becomes a deployment plan, are the quantization scheme, calibration corpus, and evaluation harness that bounded achievable quality at this precision disclosed beside the throughput — or is the bandwidth-driven payoff being quoted against an accuracy figure the production prompt distribution will not honour?
Frequently Asked Questions
Why does INT4 give a larger throughput gain on LLMs than on image-model training?
LLM inference is memory-bandwidth-bound: the accelerator must read every weight from HBM at least once per token, so cutting bytes-per-weight cuts the dominant wait roughly in proportion. Image-model training is compute-bound, where the arithmetic dominates the wall-clock cost, so lower precision does far less to lift it. That difference in workload shape is why INT4 acts almost like a direct throughput multiplier for token generation but does not behave the same way for a training workload.
Does a quantization scheme that passes multiple-choice benchmarks mean code generation is safe?
No, and treating it that way is the most common source of disappointment. Multiple-choice tasks only require the highest-probability token in a constrained position and are forgiving of small per-token shifts. Code generation is binary-correct at every token boundary, so a single mis-quantized weight that nudges a brace token below threshold can break an entire output — a regime the multiple-choice line never measured.
What should I check first when a published INT4 recipe degrades my production quality?
Start by confirming which weights and activations the recipe quantized and whether your deployment workload was represented in its evaluation, because the published benchmark may have measured constrained tasks rather than your long-form or code outputs. Then check the calibration method and corpus, since those are among the dominant drivers of the per-layer error distribution. A workload-matched evaluation on a sample of your actual traffic is what produces deployment-grade information.
Why are two “INT4 with X% loss” reports often not comparable?
Because the precision format alone does not describe the measurement. Reports can differ in which weights and activations were quantized, the calibration method (GPTQ, AWQ, SmoothQuant, min-max, percentile clipping) and corpus, and the evaluation workload — including whether long-form outputs were tested at all. Those differences are methodology, not noise, so two reports citing the same bit width and loss number can describe materially different deployment behaviour.
Does quantizing the KV cache carry the same risk profile as quantizing weights?
Not quite. Weight quantization error is fixed once the model is prepared, while KV-cache quantization error accumulates across the generated sequence, so its effect grows with context length rather than staying constant. That makes long-context and multi-turn workloads the place to look first when a weight-only recipe was clean but end-to-end quality still drifted.
Balancing speed, memory, and model fidelity
Reduced precision shrinks memory requirements and raises throughput, though accuracy penalties differ by network depth and task type. So the question to carry forward is this: do you know the executor, the rules, and the release behind the number in front of you — and if not, what would it take to find out?
