The single-request best-case trap The easiest latency test to run on an AI inference service is also the worst latency test for predicting production behavior: send one request at a time, wait for the response, record the time, repeat. The number that falls out is the AI Executor’s best-case per-request latency on a quiescent system. It is a real number — and it is a number the production deployment will essentially never see, because production serves concurrent traffic against a continuously-loaded inference server, and the latency distribution under those conditions does not resemble the single-request quiescent case at all. We see this pattern regularly in vendor benchmarks and internal performance reports: a model card cites a “12 ms latency on an H100” headline, and six months later the team discovers that p99 under realistic concurrency sits closer to 400 ms. Nothing about the headline number was false. It was just measured in a regime no production service operates in. Latency testing for AI inference has to be designed around the conditions production will actually impose, not the conditions easiest to instrument. That requires explicitly varying the axes that govern latency — batch size, concurrency, arrival pattern — and reporting the tail percentiles that latency-sensitive systems care about, not the averages that mask them. Which three axes have to be declared? A latency test that fixes only one variable under-specifies the result. The three axes any meaningful latency test must declare are independent, and they interact in ways that change the conclusion. Batch size. Whether the inference server processes requests one at a time, in fixed-size batches, in dynamic batches with a timeout, or via continuous batching (the pattern vLLM and TensorRT-LLM use for autoregressive models) determines how queue time accumulates and how kernel execution amortizes across the batch. The same model on the same accelerator produces very different latency distributions under each policy. A static batch of 32 may double single-request throughput while tripling p99 latency; a continuous-batching scheduler may keep p50 nearly flat while letting p99.9 wander because of preemption and prefill collisions. Concurrency level. The number of simultaneous in-flight requests the test sustains determines queue depth, how often batches form at their target size, and how close the system runs to its saturation point. Low concurrency exposes per-request execution time; high concurrency exposes queue-and-saturation behavior; the relationship between them is the system’s load profile. A test that runs at a single concurrency level reports one point on a curve and presents it as if it described the whole curve. Request arrival distribution. A closed-loop test (each finished request triggers the next) measures throughput-bounded behavior. An open-loop test (requests arrive on a fixed schedule regardless of completion) measures response under independent load. Production traffic is closer to open-loop with bursty arrivals than to closed-loop, and in our experience a closed-loop test systematically understates queue-induced tail latency — sometimes by an order of magnitude at p99 under realistic concurrency. This is the classic coordinated-omission problem Gil Tene has described for years; it is no less present in AI inference benchmarks than it was in JVM service benchmarks a decade ago. A test that fixes one of these axes and varies the other two communicates how latency depends on the varied axes under a stated condition for the third. A test that varies all three, or fixes all three without disclosure, produces a number that cannot be interpreted operationally. Tail percentiles, not averages Latency-sensitive systems exist to bound the worst-case latency users experience, not the average. A system whose mean latency is 50 ms and whose p99 is 2 seconds delivers a categorically different experience than a system whose mean is 80 ms and whose p99 is 150 ms. Average latency does not distinguish them. p99 does. Why is average latency a misleading metric in latency-sensitive AI systems? The averaging operation is the failure. Latency distributions for batched and queued inference are heavy-tailed by construction: most requests get serviced in the typical regime, and a long thin tail accumulates queue waits, preemption delays, garbage-collection or memory-allocation stalls, and prefill collisions in continuous batching. A mean smears those tails into the bulk and reports a number close to the median. The user experiencing the tail is not averaged out of existence; they are simply not represented in the headline figure. The percentiles that latency tests should report at minimum: p50 (median) — the typical request’s experience. p95 — the experience of one request in twenty. p99 — the experience of one request in a hundred. For high-traffic systems this is a non-negligible portion of total user-facing requests; at 1,000 requests per second, p99 represents ten users per second hitting the tail. p99.9 — the experience of one request in a thousand. For services with strict SLOs, this is often the controlling number. Reporting the maximum latency observed during the test can be informative but is sample-size-dependent and noisy; the percentiles above are more stable across runs. A benchmark that reports only mean or median latency systematically hides the operational risk that latency-sensitive systems exist to manage, and a deployment decision made from such a benchmark is uninformed about the regime that actually drives the service-level objective. A latency-testing methodology checklist A latency test that produces a result a deployment team can use should satisfy the following. Treat any missing item as a bound on how far the result generalizes: Workload definition stated: model, model size, precision regime (FP16, BF16, FP8, INT8, INT4), input shape distribution. For LLMs, prompt-length and output-length distributions, not just averages. AI Executor stated: accelerator + driver + runtime + framework + inference runtime versions. “H100 + TensorRT-LLM” is not enough; the driver version and TensorRT-LLM commit hash matter for reproducibility. Batch policy stated: static batch size N, dynamic batch with timeout T, or continuous batching with policy parameters (max batch tokens, scheduling discipline). Concurrency level stated: number of simultaneous in-flight requests sustained during the test. Arrival distribution stated: open-loop (with arrival rate λ) or closed-loop, with any bursty/Poisson/uniform parameters. Warm-up window excluded: the first N seconds — long enough for thermal equilibrium and any one-time framework initialization, CUDA graph capture, or kernel autotuning — discarded from measurement. Measurement window long enough for thermal equilibrium: typically minutes, not seconds, for sustained-load representativeness. GPUs throttle, and short windows miss it. Percentiles reported: at minimum p50, p95, p99; for strict-SLO contexts, p99.9. Throughput reported alongside: so the latency numbers are scoped to a specific operating point on the throughput-vs-latency curve. Number of trials and inter-trial variance reported: to distinguish stable measurements from noisy ones. Co-tenant load disclosed: whether the host was otherwise quiet or under realistic background load. Shared NVLink or PCIe topology with other tenants changes the tail. A test that satisfies this list produces a result the reader can apply to their own deployment decision. A test that satisfies a subset produces a result whose generalization is bounded by what is missing — which is fine, provided the missing items are disclosed rather than implied. How latency testing relates to throughput testing Latency testing and throughput testing are not separate concerns; they are the two axes of the same operating curve. Every (batch, concurrency, arrival) configuration produces both a latency distribution and an aggregate throughput, and the trade-off between them is the curve the system can traverse. This is the operational expression of the underlying throughput vs latency trade-off the rest of this K-space addresses. A complete latency-test report sweeps the configuration space and produces a curve, not a point: throughput on one axis, p99 latency on the other, and the curve traced by varying batch and concurrency. The deployment decision then becomes “where on this curve should we operate?” instead of “is this system fast enough?” — which is the question latency benchmarks should be designed to answer in the first place. There is also a distinction between model-only latency and end-to-end system latency the curve should make explicit; the former is the time the accelerator spends in the forward pass, while the latter includes ingress, queueing, batch assembly, detokenization, and egress. Collapsing the two is a common reason that a benchmark’s p99 and a production service’s p99 disagree, and any latency report that does not disclose which one is being measured leaves the reader unable to interpret the number. A methodology that measures one axis without bounding the other is producing a number divorced from the trade-off it sits in. That is the structural problem the checklist above is trying to prevent. ## The framing that helps Latency testing for AI inference must declare batch policy, concurrency level, and arrival distribution; it must report tail percentiles, not averages; and it should produce a curve across the operating space, not a single point estimate. A best-case quiescent number is real but operationally unrepresentative; a percentile distribution under sustained, declared load is the minimum useful unit for a deployment decision. LynxBench AI treats latency as a distribution under declared batch and concurrency configurations against a fully-specified AI Executor, and treats tail percentiles — not the average — as required disclosure, because the latency-sensitive deployment decisions the methodology exists to inform are governed by tails, not means. The question to put to any latency claim before relying on it is whether the number is a tail percentile under realistic load, or a best-case quiescent average that production conditions will not reproduce. Is the tail percentile in front of you the right metric for this workload at the operating point the SLO will actually defend — measured on the production AI Executor under realistic batch, concurrency, and arrival load — or a quiescent average produced under conditions the deployment cannot rebuild?
