A different memory pressure than weight quantization addresses
Conflating KV-cache quantization with weight quantization obscures fundamental differences in how, when, and why each technique compresses data. It is not. It addresses a different memory pressure, has a different accuracy risk profile, and answers a different deployment question. Treating the two as interchangeable produces evaluation mistakes that show up only in production — usually after the deployment has been sized on numbers that turn out to be the wrong numbers.
Weight quantization reduces the static memory footprint of an LLM — the bytes that hold the model’s parameters and that are loaded once per inference server. KV-cache quantization reduces the per-request key/value tensors that the model accumulates during generation, and that grow linearly with context length and request concurrency. The two compress different things, and the operational lever each gives the serving engineer is different.
In our experience reviewing inference deployments where memory was the binding constraint, the team almost always knew which of the two pressures was hurting them. They just hadn’t separated the two quantization decisions cleanly, and the published accuracy numbers they were citing didn’t separate them either.
What does KV-cache quantization actually unlock?
Each previously processed token contributes a key and value projection stored in the cache, indexed by attention head and layer, throughout autoregressive generation. This is the KV cache. Its size scales as roughly 2 × layers × heads × head_dim × bytes_per_value × context_tokens per concurrent request. For a long context — tens or hundreds of thousands of tokens — and several concurrent requests, the KV cache becomes the dominant consumer of accelerator memory, exceeding the weight footprint of the model itself.
When KV-cache memory becomes the binding constraint, the practical consequences are immediate: the maximum supported context window shrinks, the maximum number of concurrent requests collapses, or both. KV-cache quantization addresses this directly. Storing keys and values in INT8, INT4, or FP8 instead of FP16 reduces the per-token cache footprint by roughly 2× to 4× (observed pattern across typical inference stacks; the exact ratio depends on the format and on whether scales are stored per-head or per-token). That translates into proportional increases in either supported context length or supported concurrency on a fixed-memory accelerator.
This is a different optimization axis than weight quantization. Weight quantization makes the model fit on a smaller accelerator, or makes per-token inference faster by reducing weight-load bandwidth. KV-cache quantization makes longer contexts or higher concurrency fit on the same accelerator. They are complementary, not substitutes — and in real deployments we typically see both applied, but evaluated separately.
Why the accuracy risk profile is distinct
Weight quantization and KV-cache quantization face entirely separate accuracy trade-offs. It is structurally different.
Weights are static. Once a model is trained, its weight distributions are fixed. A calibration procedure for weight quantization observes those distributions on a representative input set and chooses scale factors that bound the quantization error tightly across known weight values. The quantization error introduced into the model is fully determined at calibration time and is constant across all inferences thereafter. This is the standard frame in which quantization is controlled approximation, not model damage: a one-time, bounded, calibratable perturbation.
KV-cache values are activations, not weights. They are produced at runtime, conditional on the input prompt and on every previously generated token. Their distributions are workload-dependent, and they can exhibit large outliers — single attention positions whose key or value norms are many standard deviations above the typical range. Low-precision integer formats represent outliers poorly, because the format’s representable range must be set wide enough to cover them, which leaves the typical values represented at coarser-than-necessary granularity. Modern inference stacks — vLLM, TensorRT-LLM, FasterTransformer — all expose KV-cache quantization knobs, and all of them ultimately push that representable-range decision back onto a calibration choice the operator has to make.
The implication is that KV-cache quantization can produce one accuracy profile on a benchmark whose prompt distribution rarely produces large outliers, and a substantially worse profile on a deployment workload whose prompts routinely do. The gap is not noise — it is a real difference in the input distribution the cache observes. This is also the point at which calibration data, not calibration method, becomes the dominant factor in the result.
Where the two also differ is in when the approximation is chosen. Weight quantization can be pushed back into training — a quantization-aware training run makes the network adapt to the coarser grid before the weights are frozen. There is no equivalent move for a runtime cache whose contents depend on prompts nobody has written yet. The KV-cache decision is post-training by construction, which means calibration is the only control surface available, and the only place discipline can be applied.
Comparing weight quantization and KV-cache quantization
| Dimension | Weight quantization | KV-cache quantization |
|---|---|---|
| What is quantized | Model parameters (static) | Per-request key/value tensors (dynamic) |
| Memory pressure addressed | Static model footprint | Per-request cache growing with context length |
| Operational lever | Fits model on smaller accelerator; reduces per-token bandwidth cost | Increases max context length or max concurrency on same accelerator |
| Distribution stability | Fixed once trained; calibrated once | Workload-dependent; varies per prompt and per token |
| Outlier behavior | Bounded by training-time weight distribution | Includes runtime activation outliers that low-precision formats represent poorly |
| Where error can be controlled | Calibration, or training-time (QAT) adaptation | Calibration only; no training-time equivalent exists |
| Calibration validity | Calibration data only needs to span weight value ranges | Calibration must span the deployment workload’s activation distribution |
| Evidence class for accuracy claims | benchmark when measured on a named test set |
observed-pattern unless measured on the actual deployment workload |
Without separate disclosure of KV cache quantization scheme, calibration method, and bit-width, any reported LLM accuracy figure remains fundamentally incomplete. An “INT4 quantized” label on its own does not tell you which tensors are at INT4 and which distributional assumptions the calibration made.
What does an honest KV-cache evaluation need to expose?
Extrapolating weight-quantization accuracy to predict KV-cache behavior on the same model is methodologically invalid. The two operate on different tensors, with different distributional properties, and their accuracy regressions are not additive in any clean way. Evaluating KV-cache quantization requires running the deployment workload — including its long-context and high-concurrency regimes — and measuring output behavior under those conditions, not extrapolating from short-context standard benchmarks.
The evaluation question for KV-cache quantization is also operational rather than purely accuracy-driven: what context length, at what concurrency, does each cache precision support before the accelerator runs out of memory? That number is a property of the deployment configuration, not of the model alone, and it is the number that determines whether KV-cache quantization is the right intervention for a particular memory-pressure problem.
A KV-cache evaluation that we would consider trustworthy discloses, at minimum: which tensors were quantized (keys only, values only, both); the per-tensor format and granularity (per-head, per-token, group-wise); the calibration corpus and how it relates to the deployment workload; the long-context regime tested; and the concurrency regime tested. Without those, “we quantized the KV cache to INT4” is a configuration string, not a result.
There is a procedural version of the same discipline worth borrowing. In the released LynxBenchAI catalogue, the models are prepared once — before any target device is known — and the identical prepared artefacts are then run on every vendor’s hardware; where two vendors’ kernels disagree about what an approximation may do, the stricter constraint is applied to all of them rather than settled per vendor. Correctness criteria are enforced per test type and per precision, so an approximation that has gone too far is recorded as a zero rather than reported as a fast result. None of that measures a reader’s quantization scheme. It illustrates that the bookkeeping around an approximation can be written down instead of assumed, and a KV-cache report can adopt the same posture: state what was prepared, under whose rules, and what threshold the result had to clear.
The framing that actually helps
Unlike static weight compression, KV-cache quantization compresses activation tensors that scale dynamically with both context length and concurrent requests. It addresses a memory-pressure regime that weight quantization cannot, and its accuracy risk profile is determined by activation distributions that calibration must observe in deployment-shaped workloads, not in standard-benchmark prompts.
The general principle that quantization is controlled approximation rather than model damage holds for KV-cache quantization as it does for weight quantization. The KV-cache-specific point is that the activation-distribution dependency makes the calibration step strictly more workload-coupled than the weight-quantization equivalent — and that calibration step is what determines whether the bounded approximation stays bounded under the prompts the deployment will actually see. LynxBenchAI treats KV-cache precision as a separately reported regime from weight precision, with the calibration workload disclosed, because conflating them in a single “INT4 quantized” label hides exactly the distributional dependency that determines whether the deployment will hold up. For any “INT4 quantized” claim you are about to act on, does the report separately disclose which tensors hold which precision — and which calibration workload bounded achievable quality at this precision under the context length and concurrency regime the production deployment will actually inhabit?
Frequently Asked Questions
Does quantizing the KV cache change latency, or only memory capacity?
Its primary effect is capacity: a smaller per-token cache means more context or more concurrent requests fit in the same accelerator memory. Latency effects are indirect and stack-dependent — reading fewer cache bytes per attention step can help bandwidth-bound decoding, while dequantization work on the attention path can offset that. Treat throughput and latency as separate measurements rather than assuming they follow the memory saving.
Should keys and values be quantized to the same precision?
Not necessarily. Keys and values enter attention differently, and in practice key tensors are often the more outlier-prone of the two, so a configuration that keeps keys at a higher precision than values is a reasonable thing to test rather than a compromise. The point is that “KV cache at INT4” is under-specified: keys-only, values-only, and both-at-once are three different configurations with three different accuracy profiles.
How should a KV-cache calibration corpus be chosen?
It should span the activation distribution the deployment will actually produce, which means prompts of the length, domain, and structure the service receives — not a short-context standard benchmark. Calibration corpora that under-represent long prompts systematically under-represent the outliers that low-precision formats handle worst, and the resulting scale factors look safe in evaluation and fail in production.
Can a weight-quantization result be used to predict the KV-cache result on the same model?
No. The two act on different tensors with different distributional properties, and their accuracy regressions do not compose in any clean additive way. A model that tolerates INT4 weights well may tolerate an INT4 cache badly, or the reverse, so each has to be measured under the long-context and concurrency regime it will run in.
Choosing between cache and weight quantization
Long-context workloads that hit memory bandwidth limits benefit most from KV cache quantization; weight quantization improves performance more broadly. If any of those differ, are you still looking at a comparison, or two unrelated observations?
