Throughput Definition for AI Inference: Why Batch Size Is Part of the Number

A throughput number without context is not a throughput number

A vendor cites a throughput figure for an AI accelerator: requests per second, tokens per second, images per second. The number is large. It seems to characterize the device. It does not. The same accelerator on the same model produces different throughput numbers under different batch configurations, different precision regimes, and different operating points on the latency curve — and a throughput figure that names none of these is reporting a peak observation, not a comparable measurement.

For AI inference, throughput has a precise definition that ties it inseparably to the batch policy and the latency budget under which it was measured. Reporting throughput without those dimensions is the inference equivalent of reporting peak silicon TFLOPS as application performance: technically a real number, operationally uninformative.

In our experience reviewing inference benchmarks across customer engagements, the throughput-only report is the single most common failure mode — and it is the one that survives furthest into procurement decisions before someone notices the deployment doesn’t reproduce the cited number.

What is throughput in AI inference, precisely?

Throughput in AI inference is the rate of completed inference requests (or, for token-generative models, generated tokens) per unit wall-clock time, measured under a declared batch size, concurrency level, and latency budget on a fully-specified AI Executor.

Each component of that definition is non-optional.

Rate of completed work, not initiated work. Throughput counts what came out the other side. Started-but-not-completed requests do not contribute. This matters under heavy load: a server that accepts more requests than it can complete inflates a “requests per second” measurement that doesn’t survive into the response stream.

Per unit wall-clock time. The denominator is real time, not CPU time or GPU-active time. A measurement that excludes queue time or framework overhead is reporting kernel throughput, not inference throughput.

Under a declared batch size. The same model on the same accelerator can produce dramatically different throughput at batch=1 vs batch=8 vs batch=64. The throughput number is a function of the batch policy, not a property of the device alone. This holds whether the runtime is TensorRT-LLM with continuous batching, Triton with dynamic batching, or a hand-rolled PyTorch server with static batches.

At a declared latency budget. Throughput can almost always be raised by increasing batch size — at the cost of per-request latency. A throughput number untethered from a latency budget can be optimized arbitrarily by accepting arbitrarily bad latency, which is why the latency budget is part of the throughput report, not a separate concern.

On a fully-specified AI Executor. Accelerator hardware, driver, runtime (CUDA version, cuDNN version), framework (PyTorch, JAX), inference runtime (TensorRT, vLLM, TGI), and precision regime (FP16, BF16, FP8, INT8) all enter the throughput a measurement will produce.

Why batch size is inseparable from throughput

Modern AI accelerators are designed to process work in batches because per-batch overhead amortizes across the batch’s items. Larger batches generally raise throughput (more items per unit kernel time) and raise per-request latency (each item waits for the batch to form and complete). The relationship is not linear in either direction, and it has a saturation point: beyond some batch size, throughput stops growing because some other resource — memory, scheduler, kernel-occupancy — becomes the bottleneck.

The practical consequence is that “throughput at batch X” and “throughput at batch Y” are different numbers describing different operating points of the same system. Comparing the throughput of one accelerator at its optimal batch to the throughput of another accelerator at a different batch is not a hardware comparison; it is a comparison between two operating points of two different systems.

A throughput number that doesn’t name its batch is comparable to nothing. A throughput number that names its batch is comparable only to throughput numbers measured at the same batch on the same workload — and even then, only if the latency at that batch is acceptable for the deployment scenario.

Throughput and latency are coupled, not independent

The throughput-vs-latency curve is the structure that links the two metrics. For a given AI Executor and a given workload, every batch and concurrency configuration produces both a throughput and a latency distribution. The curve traced by sweeping configurations is the system’s operating envelope. We treat this curve, not any single point on it, as the object a benchmark report should characterize.

Three points on that curve illustrate why throughput in isolation is uninformative:

Operating point	Throughput	p99 latency	Useful for
Single request, no batching	Low	Low	Latency-critical small workloads
Optimal batch for throughput	High	High (often unacceptably so)	Offline / batch-mode workloads
Highest batch under latency SLO	Moderate-to-high	Bounded by SLO	Online inference services

This is an observed pattern across the inference deployments we’ve worked through, not a benchmarked rate from a single test — the actual numbers shift with model, hardware, and runtime, but the shape of the table holds.

The vendor-quoted “peak throughput” is typically the middle row. The number a deployment team needs is the third row. They are different points on the same curve, and they describe different operational realities.

The framing that follows from this is to report throughput at a declared latency budget — for example, “X requests per second at p99 ≤ 100 ms” — rather than throughput in isolation. This bounds the trade-off explicitly and produces a number a deployment team can apply. The full structural argument for why this coupling matters lives in our companion piece on picking the wrong optimisation target between throughput and latency.

Why averages lie about latency in throughput reports

A throughput number paired with average latency is barely better than a throughput number alone. Inference latency distributions are routinely heavy-tailed: the mean sits well below the tail, and the tail is where SLO violations live. A p99 of 280 ms with a mean of 60 ms describes a system that meets a 100 ms SLA on average and breaches it for one request in a hundred — which, at a thousand requests per second, is ten breaches per second.

The throughput report that names “throughput at p99 ≤ X ms” is doing two things at once: bounding the operating point, and forcing the latency disclosure to be the percentile that matters. A report that names “throughput at mean latency ≤ X ms” is hiding the tail behind an average and should be read with the same skepticism as any other averaged claim about a heavy-tailed distribution.

A common adjacent confusion is between throughput and bandwidth. Bandwidth measures the rate at which data can be moved through a channel (memory bus, NVLink, PCIe, network link) and is typically reported in bytes per second. Throughput in inference measures the rate of completed work and is reported in requests, tokens, or items per second.

Bandwidth is an upper bound on the work throughput a memory-bound workload can sustain — but it is not the same number. A workload bottlenecked by HBM bandwidth (LLM decode, for instance) will exhibit throughput proportional to the bandwidth available to its access pattern; a workload bottlenecked by compute (prefill on small batches, dense vision models at high batch) will exhibit throughput unrelated to nominal bandwidth. Reporting bandwidth and calling it throughput conflates the upper-bound resource with the work-rate measurement, and a benchmark report should keep them lexically and methodologically separate.

What disclosure makes a throughput number useful

A throughput number for AI inference becomes interpretable when the report names:

The model and its size.
The precision regime of the inference (FP16, BF16, FP8, INT8, mixed).
The AI Executor — accelerator, driver, CUDA/runtime versions, framework, inference runtime versions.
The batch policy (static N, dynamic with timeout, continuous batching) and the batch size at which the throughput was measured.
The concurrency level under which the measurement was sustained.
The latency budget under which the throughput was achieved (e.g. p99 ≤ X ms), with percentile named explicitly.
The duration of the measurement window — long enough for thermal equilibrium, with warm-up excluded.

A throughput report that satisfies this list characterizes an operating point on the system’s throughput-vs-latency curve. A throughput report that names a single number is a peak observation under unspecified conditions, and the inference into deployment is the reader’s problem. ## The framing that helps

Throughput for AI inference is the rate of completed work per unit wall-clock time under a declared batch policy, concurrency, latency budget, and AI Executor. It is coupled to latency by the trade-off curve the system traces; it is bounded by — but not equal to — the bandwidth of the resources it depends on; and it is uninformative as a single number divorced from the operating-point disclosure that ties it to a deployment scenario.

LynxBench AI treats throughput as a function of batch and concurrency at a declared latency budget on a fully-specified AI Executor — because the throughput-vs-latency trade-off is operationally meaningful only when both axes are scoped to the same disclosed operating point. For any throughput claim you intend to act on: was the number measured at the latency budget the SLO actually requires — the binding operating point for throughput under your concurrency — or at a more permissive operating point the deployment will never reproduce?