Latency Testing for AI Inference: A Methodology Beyond Best-Case Numbers

The single-request best-case trap

Running requests serially—one at a time, waiting for each response—produces a latency number that tells you almost nothing about how the service will perform under real traffic. The number that falls out is the AI Executor’s best-case per-request latency on a quiescent system. It is a real number — and it is a number the production deployment will essentially never see, because production serves concurrent traffic against a continuously-loaded inference server, and the latency distribution under those conditions does not resemble the single-request quiescent case at all.

We see this pattern regularly in vendor benchmarks and internal performance reports: a model card cites a “12 ms latency on an H100” headline, and six months later the team discovers that p99 under realistic concurrency sits closer to 400 ms. Nothing about the headline number was false. It was just measured in a regime no production service operates in.

Latency testing for AI inference has to be designed around the conditions production will actually impose, not the conditions easiest to instrument. That requires explicitly varying the axes that govern latency — batch size, concurrency, arrival pattern — and reporting the tail percentiles that latency-sensitive systems care about, not the averages that mask them.

Which three axes have to be declared?

Holding a single variable constant leaves too many degrees of freedom unspecified to make the measurement reproducible or actionable. The three axes any meaningful latency test must declare are independent, and they interact in ways that change the conclusion.

Batch size. Whether the inference server processes requests one at a time, in fixed-size batches, in dynamic batches with a timeout, or via continuous batching (the pattern vLLM and TensorRT-LLM use for autoregressive models) determines how queue time accumulates and how kernel execution amortizes across the batch. The same model on the same accelerator produces very different latency distributions under each policy. A static batch of 32 may double single-request throughput while tripling p99 latency; a continuous-batching scheduler may keep p50 nearly flat while letting p99.9 wander because of preemption and prefill collisions.

Concurrency level. The number of simultaneous in-flight requests the test sustains determines queue depth, how often batches form at their target size, and how close the system runs to its saturation point. Low concurrency exposes per-request execution time; high concurrency exposes queue-and-saturation behavior; the relationship between them is the system’s load profile. A test that runs at a single concurrency level reports one point on a curve and presents it as if it described the whole curve.

Request arrival distribution. A closed-loop test (each finished request triggers the next) measures throughput-bounded behavior. An open-loop test (requests arrive on a fixed schedule regardless of completion) measures response under independent load. Production traffic is closer to open-loop with bursty arrivals than to closed-loop, and in our experience a closed-loop test systematically understates queue-induced tail latency — sometimes by an order of magnitude at p99 under realistic concurrency. This is the classic coordinated-omission problem Gil Tene has described for years; it is no less present in AI inference benchmarks than it was in JVM service benchmarks a decade ago.

A test that fixes one of these axes and varies the other two communicates how latency depends on the varied axes under a stated condition for the third. A test that varies all three, or fixes all three without disclosure, produces a number that cannot be interpreted operationally.

Tail percentiles, not averages

Systems built for latency sensitivity must characterize their tail behavior, not their central tendency. A system whose mean latency is 50 ms and whose p99 is 2 seconds delivers a categorically different experience than a system whose mean is 80 ms and whose p99 is 150 ms. Average latency does not distinguish them. p99 does.

The averaging operation is the failure. Latency distributions for batched and queued inference are heavy-tailed by construction: most requests get serviced in the typical regime, and a long thin tail accumulates queue waits, preemption delays, memory-allocation stalls, and prefill collisions in continuous batching. A mean smears those tails into the bulk and reports a number close to the median. The user experiencing the tail is not averaged out of existence; they are simply not represented in the headline figure.

The percentiles that latency tests should report at minimum:

p50 (median) — the typical request’s experience.
p90 — the shoulder of the distribution, useful for seeing where queueing begins to bite before the tail dominates.
p95 — the experience of one request in twenty.
p99 — the experience of one request in a hundred. For high-traffic systems this is a non-negligible portion of total user-facing requests; at 1,000 requests per second, p99 represents ten users per second hitting the tail.
p99.9 — the experience of one request in a thousand. For services with strict SLOs, this is often the controlling number.

Reporting the maximum latency observed during the test can be informative but is sample-size-dependent and noisy; the percentiles above are more stable across runs. A benchmark that reports only mean or median latency systematically hides the operational risk that latency-sensitive systems exist to manage, and a deployment decision made from such a benchmark is uninformed about the regime that actually drives the service-level objective.

Token-level metrics change what “latency” means

Autoregressive LLM serving involves two structurally distinct phases within each request, making token-level granularity more informative than request-level summaries. Three metrics carry the distinction:

Metric	What it measures	Which trade-off it exposes
TTFT (time-to-first-token)	Wall-clock from request arrival to the first emitted token — queueing plus batch assembly plus the prefill forward pass	Batch-assembly and admission policy: larger batches and longer prefill queues raise TTFT while lifting aggregate throughput
TPOT (time-per-output-token)	Mean decode time per generated token across the response	Steady-state decode efficiency: how much of the accelerator’s throughput advantage survives when the batch is shared across many concurrent decodes
ITL (inter-token latency)	The distribution of gaps between consecutive tokens, not their mean	Scheduler jitter: preemption, prefill collisions with in-flight decodes, and eviction show up here and nowhere else

TTFT and TPOT belong to different phases and respond to different levers, so a report that collapses them into one “latency” figure cannot be acted on. ITL is the metric most often omitted and the one that governs perceived smoothness in streaming interfaces — a service with an acceptable TPOT mean can still stutter visibly if its ITL tail is wide. Each of these is a percentile distribution in its own right; a mean TTFT is subject to exactly the same tail-hiding failure described above.

A latency-testing methodology checklist

Useful latency tests meet these criteria: Treat any missing item as a bound on how far the result generalizes:

A test that satisfies this list produces a result the reader can apply to their own deployment decision. A test that satisfies a subset produces a result whose generalization is bounded by what is missing — which is fine, provided the missing items are disclosed rather than implied.

How latency testing relates to throughput testing

Throughput and latency are not independent: they describe complementary views of the same capacity surface. Every (batch, concurrency, arrival) configuration produces both a latency distribution and an aggregate throughput, and the trade-off between them is the curve the system can traverse. This is the operational expression of the underlying throughput vs latency trade-off the rest of this K-space addresses.

A complete latency-test report sweeps the configuration space and produces a curve, not a point: throughput on one axis, p99 latency on the other, and the curve traced by varying batch and concurrency. The deployment decision then becomes “where on this curve should we operate?” instead of “is this system fast enough?” — which is the question latency benchmarks should be designed to answer in the first place. There is also a distinction between model-only latency and end-to-end system latency the curve should make explicit; the former is the time the accelerator spends in the forward pass, while the latter includes ingress, queueing, batch assembly, detokenization, and egress. Collapsing the two is a common reason that a benchmark’s p99 and a production service’s p99 disagree, and any latency report that does not disclose which one is being measured leaves the reader unable to interpret the number.

A methodology that measures one axis without bounding the other is producing a number divorced from the trade-off it sits in. That is the structural problem the checklist above is trying to prevent.

Why a throughput run cannot be re-read as a latency answer

Consider the inverse approach and why it fails. A saturated-throughput measurement — raise the batch size until throughput stops improving inside a defined noise band, then count completed iterations inside a declared timed window after a discarded warm-up — is a throughput instrument by construction. That is what a LynxBenchAI run produces, deliberately: model architecture and numerical precision are held constant within a release, only the batch size adapts, and the result is an aggregate rate at the saturation point.

No arithmetic recovers percentiles from that. The run does not retain per-request completion timestamps under an independent arrival schedule, it does not run at a concurrency level chosen to represent production traffic, and it says nothing about the machine’s thermal or clock condition inside the window — it only declares that the window exists after the warm-up was discarded. Dividing the window by the iteration count yields a mean per-iteration time, which is precisely the averaged figure the tail-percentile argument rejects, and no tail behaviour, TTFT, or ITL is present in the data at all.

What the throughput number does give a reader is a starting coordinate. Knowing where a device saturates tells you which batch size the latency sweep should bracket, and how much headroom you are trading away when you move down the curve to defend an SLO. Then run the open-loop, percentile-reporting test described above on the same hardware. Two instruments, one curve.

The framing that helps

Declare batch policy, concurrency, and arrival patterns; report tail percentiles over averages; generate an operating curve rather than isolated point estimates—these are the non-negotiable requirements for actionable AI inference latency testing. A best-case quiescent number is real but operationally unrepresentative; a percentile distribution under sustained, declared load is the minimum useful unit for a deployment decision.

The question to put to any latency claim before relying on it is whether the number is a tail percentile under realistic load, or a best-case quiescent average that production conditions will not reproduce. Is the tail percentile in front of you the right metric for this workload at the operating point the SLO will actually defend — measured on the production AI Executor under realistic batch, concurrency, and arrival load — or a quiescent average produced under conditions the deployment cannot rebuild?

Frequently Asked Questions

What do TTFT, TPOT, and ITL each measure, and which throughput-vs-latency trade-off does each expose?

TTFT covers arrival through queueing, batch assembly, and the prefill forward pass to the first emitted token, so it exposes admission and batching policy directly. TPOT is the mean decode time per generated token, which reflects how much of the accelerator’s throughput advantage survives when a batch is shared across concurrent decodes. ITL is the distribution of gaps between consecutive tokens, and it is where scheduler jitter — preemption, prefill collisions, eviction — becomes visible; a service with an acceptable TPOT mean can still stutter if its ITL tail is wide.

How should percentiles be reported so tail behaviour stays visible?

Report p50, p90, p95, and p99 as a set, and add p99.9 wherever a strict SLO governs the service — for those contexts p99.9 is often the controlling number. p50 alone describes the typical request, p90 shows where queueing starts to bite, and p99 and beyond describe the regime the SLO exists to defend; at 1,000 requests per second, p99 is ten users per second hitting the tail. The observed maximum is informative but sample-size-dependent and noisy, so it should never stand in for a stable percentile.

If a run reports a saturated-throughput number, what extra measurement gives a tail-latency answer?

You need a separate open-loop test on the same hardware: a declared arrival schedule at a production-representative concurrency level, per-request timestamps retained, and TTFT/TPOT/ITL reported as percentile distributions. The throughput run cannot be re-read to supply this because it holds no per-request completion data under an independent arrival schedule and its only latency-shaped derivative is a mean per-iteration time — the exact averaged figure that hides tails. The throughput result is still useful as the coordinate that tells you which batch size the latency sweep should bracket.

What is the coordinated-omission problem, and why does a closed-loop test understate tail latency?

Coordinated omission happens when a closed-loop test waits for each request to finish before issuing the next, so a slow request suppresses the very requests that would have piled up behind it in production. The result is a latency distribution that hides queue-induced delay, sometimes by an order of magnitude at p99 under realistic concurrency. An open-loop test that issues requests on a fixed schedule regardless of completion preserves that queueing, which is why production traffic is closer to open-loop than closed-loop.

Percentile distributions matter more than medians

Prefill and decode each deserve separate P95 and P99 tracking; median speed offers no warning before tail latencies violate service-level commitments. If any of those differ, are you still looking at a comparison, or two unrelated observations?