A throughput number without context is not a throughput number
Vendors publish accelerator specifications: requests per second, tokens per second, images per second. The number is large. It seems to characterize the device. It does not. The same accelerator on the same model produces different throughput numbers under different batch configurations, different precision regimes, and different operating points on the latency curve — and a throughput figure that names none of these is reporting a peak observation, not a comparable measurement.
For AI inference, throughput has a precise definition that ties it inseparably to the batch policy and the latency budget under which it was measured. Reporting throughput without those dimensions is the inference equivalent of reporting peak silicon TFLOPS as application performance: technically a real number, operationally uninformative.
In our experience reviewing inference benchmarks across customer engagements, the throughput-only report is the single most common failure mode — and it is the one that survives furthest into procurement decisions before someone notices the deployment doesn’t reproduce the cited number.
What is throughput in AI inference, precisely?
Throughput in AI inference measures completed inference operations — requests or generated tokens — per second of wall-clock time, always qualified by batch size, concurrency, latency ceiling, and the specific AI Executor configuration.
Each component of that definition is non-optional.
Rate of completed work, not initiated work. Throughput counts what came out the other side. Started-but-not-completed requests do not contribute. This matters under heavy load: a server that accepts more requests than it can complete inflates a “requests per second” measurement that doesn’t survive into the response stream.
Per unit wall-clock time. The denominator is real time, not CPU time or GPU-active time. A measurement that excludes queue time or framework overhead is reporting kernel throughput, not inference throughput.
Under a declared batch size. The same model on the same accelerator can produce dramatically different throughput at batch=1 vs batch=8 vs batch=64. The throughput number is a function of the batch policy, not a property of the device alone. This holds whether the runtime is TensorRT-LLM with continuous batching, Triton with dynamic batching, or a hand-rolled PyTorch server with static batches.
At a declared latency budget. Throughput can almost always be raised by increasing batch size — at the cost of per-request latency. A throughput number untethered from a latency budget can be optimized arbitrarily by accepting arbitrarily bad latency, which is why the latency budget is part of the throughput report, not a separate concern.
On a fully-specified AI Executor. Accelerator hardware, driver, runtime (CUDA version, cuDNN version), framework (PyTorch, JAX), inference runtime (TensorRT, vLLM, TGI), and precision regime (FP16, BF16, FP8, INT8) all enter the throughput a measurement will produce.
Why batch size is inseparable from throughput
Batch processing amortizes per-request overhead across multiple items, which is why AI accelerators achieve peak efficiency under batch workloads. Per-batch overhead distributes cost across every item in the batch. Because AI accelerators amortize fixed per-batch costs across every item in that batch. Larger batches generally raise throughput (more items per unit kernel time) and raise per-request latency (each item waits for the batch to form and complete). The relationship is not linear in either direction, and it has a saturation point: beyond some batch size, throughput stops growing because some other resource — memory, scheduler, kernel-occupancy — becomes the bottleneck.
The practical consequence is that “throughput at batch X” and “throughput at batch Y” are different numbers describing different operating points of the same system. Comparing the throughput of one accelerator at its optimal batch to the throughput of another accelerator at a different batch is not a hardware comparison; it is a comparison between two operating points of two different systems.
A throughput number that doesn’t name its batch is comparable to nothing. A throughput number that names its batch is comparable only to throughput numbers measured at the same batch on the same workload — and even then, only if the latency at that batch is acceptable for the deployment scenario.
This is also the reason a benchmark harness has to declare which side of the trade-off it is instrumented for. A LynxBenchAI run raises the batch size until throughput stops improving inside a defined noise band, then counts completed iterations inside a continuous timed window after a discarded warm-up. Model architecture and numerical precision are held constant within a release; only the batch dimension adapts. That makes the resulting figure a saturated-throughput measurement by construction — a useful thing to have, and not a latency instrument.
Throughput and latency are coupled, not independent
Plotting throughput against latency reveals the fundamental trade-off surface between these two dimensions. For a given AI Executor and a given workload, every batch and concurrency configuration produces both a throughput and a latency distribution. The curve traced by sweeping configurations is the system’s operating envelope. We treat this curve, not any single point on it, as the object a benchmark report should characterize.
Three points on that curve illustrate why throughput in isolation is uninformative:
| Operating point | Throughput | p99 latency | Useful for |
|---|---|---|---|
| Single request, no batching | Low | Low | Latency-critical small workloads |
| Optimal batch for throughput | High | High (often unacceptably so) | Offline / batch-mode workloads |
| Highest batch under latency SLO | Moderate-to-high | Bounded by SLO | Online inference services |
This is an observed pattern across the inference deployments we’ve worked through, not a benchmarked rate from a single test — the actual numbers shift with model, hardware, and runtime, but the shape of the table holds.
The vendor-quoted “peak throughput” is typically the middle row. The number a deployment team needs is the third row. They are different points on the same curve, and they describe different operational realities.
The framing that follows from this is to report throughput at a declared latency budget — for example, “X requests per second at p99 ≤ 100 ms” — rather than throughput in isolation. This bounds the trade-off explicitly and produces a number a deployment team can apply. The full structural argument for why this coupling matters lives in our companion piece on picking the wrong optimisation target between throughput and latency.
Why averages lie about latency in throughput reports
Reporting throughput with mean latency adds little diagnostic value over throughput alone. Inference latency distributions are routinely heavy-tailed: the mean sits well below the tail, and the tail is where SLO violations live. A p99 of 280 ms with a mean of 60 ms describes a system that meets a 100 ms SLA on average and breaches it for one request in a hundred — which, at a thousand requests per second, is ten breaches per second.
The throughput report that names “throughput at p99 ≤ X ms” is doing two things at once: bounding the operating point, and forcing the latency disclosure to be the percentile that matters. A report that names “throughput at mean latency ≤ X ms” is hiding the tail behind an average and should be read with the same skepticism as any other averaged claim about a heavy-tailed distribution.
Throughput vs bandwidth: a related distinction
Bandwidth and throughput are frequently conflated despite measuring fundamentally different system properties. Bandwidth measures the rate at which data can be moved through a channel (memory bus, NVLink, PCIe, network link) and is typically reported in bytes per second. Throughput in inference measures the rate of completed work and is reported in requests, tokens, or items per second.
Bandwidth is an upper bound on the work throughput a memory-bound workload can sustain — but it is not the same number. A workload bottlenecked by HBM bandwidth (LLM decode, for instance) will exhibit throughput proportional to the bandwidth available to its access pattern; a workload bottlenecked by compute (prefill on small batches, dense vision models at high batch) will exhibit throughput unrelated to nominal bandwidth. Reporting bandwidth and calling it throughput conflates the upper-bound resource with the work-rate measurement, and a benchmark report should keep them lexically and methodologically separate.
What disclosure makes a throughput number useful
A throughput number for AI inference becomes interpretable when the report names:
- The model and its size.
- The precision regime of the inference (FP16, BF16, FP8, INT8, mixed).
- The AI Executor — accelerator, driver, CUDA/runtime versions, framework, inference runtime versions.
- The batch policy (static N, dynamic with timeout, continuous batching) and the batch size at which the throughput was measured.
- The concurrency level under which the measurement was sustained.
- The latency budget under which the throughput was achieved (e.g. p99 ≤ X ms), with percentile named explicitly.
- The duration of the measurement window, with warm-up excluded and the window declared rather than inferred.
A throughput report that satisfies this list characterizes an operating point on the system’s throughput-vs-latency curve. A throughput report that names a single number is a peak observation under unspecified conditions, and the inference into deployment is the reader’s problem.
The framing that helps
AI inference throughput is work completed per wall-clock second, always qualified by batch policy, concurrency, latency ceiling, and executor specification. It is coupled to latency by the trade-off curve the system traces; it is bounded by — but not equal to — the bandwidth of the resources it depends on; and it is uninformative as a single number divorced from the operating-point disclosure that ties it to a deployment scenario.
LynxBenchAI treats throughput as a function of batch and concurrency at a declared latency budget on a fully-specified AI Executor — because the throughput-vs-latency trade-off is operationally meaningful only when both axes are scoped to the same disclosed operating point. For any throughput claim you intend to act on: was the number measured at the latency budget the SLO actually requires — the binding operating point for throughput under your concurrency — or at a more permissive operating point the deployment will never reproduce?
Frequently Asked Questions
Why can’t a throughput figure be reported without naming its batch size?
The same model on the same accelerator produces dramatically different throughput at batch=1 versus batch=8 versus batch=64, so the number is a function of the batch policy rather than a property of the device. A throughput figure that names no batch is comparable to nothing, and one that does name its batch is comparable only to figures measured at the same batch on the same workload. Without the batch dimension you are reporting a peak observation, not a measurement.
What is the difference between throughput and bandwidth in an inference benchmark?
Bandwidth measures the rate at which data can be moved through a channel — memory bus, NVLink, PCIe, network — in bytes per second, while throughput measures the rate of completed inference work in requests, tokens, or items per second. Bandwidth is an upper bound on the throughput a memory-bound workload can sustain, but it is not the same number: a compute-bound workload exhibits throughput unrelated to nominal bandwidth. A report that calls bandwidth “throughput” conflates the resource limit with the work-rate measurement.
What disclosures turn a raw throughput number into a usable one?
A throughput number becomes interpretable only when the report names the model and its size, the precision regime, the full AI Executor (accelerator, driver, CUDA/runtime, framework, inference runtime), the batch policy and batch size, the concurrency level, the latency budget with its percentile named, and the measurement window after warm-up. With those fields the number characterizes a specific operating point on the throughput-vs-latency curve. Without them it is a peak observation under unspecified conditions.
If a benchmark reports saturated throughput, what extra measurement gives a tail-latency answer?
A saturated-throughput run deliberately pushes the batch size until the throughput curve flattens, so its operating point is the one where per-request latency is worst — the run cannot be re-read to yield p95 or p99 behaviour it never recorded. To get a tail-latency answer you have to run a separate sweep on the same hardware at your production concurrency, holding batch at or below the level your SLO tolerates, and record the full latency distribution rather than a rate. Two measurements, two questions.
Requests per second versus tokens per second
Request-per-second, token-per-second, and compute-utilization metrics each capture a distinct facet of throughput performance. So the question to carry forward is this: do you know the executor, the rules, and the release behind the number in front of you — and if not, what would it take to find out?
