“Latency” without a domain isn’t a measurement
Performance reports across networking, storage, databases, web services, and AI inference all invoke “latency,” yet the term shifts meaning between domains in ways that routinely derail cross-functional discussions. In each domain, latency is the elapsed time between two events — but which two events, and what the workload that produces them looks like, differs enough that the numbers are not comparable across domains and not interchangeable within a single benchmark report.
For AI inference, latency has a specific operational meaning. Pinning it down — and distinguishing it from the latency definitions used in adjacent domains — is the prerequisite for reading or producing useful inference benchmark results. In our experience, this is also where most cross-functional disagreements about “performance” originate: two teams use the same word for two different physical quantities and reach different conclusions from the same number.
What is latency in AI inference, precisely?
Latency in AI inference is the elapsed wall-clock time from the arrival of an input request at the inference service to the completion of the corresponding output, measured per request, under a declared batch size, concurrency level, and request arrival pattern.
Three things are notable about that definition.
It is per-request. A single inference latency number describes one request. A batch of requests has many latencies, not one. Reporting “the latency” of a batched system without specifying which request in the batch (or which percentile across requests) is under-specified.
It includes everything between the two events. Queue time, model-load time (if not amortized), framework dispatch through PyTorch or TensorRT, kernel execution on the accelerator, post-processing, and serialization back to the client are all inside the latency envelope. Reports that name only the kernel-execution component as “latency” are reporting a model-execution time, not an inference latency. The two routinely differ by an order of magnitude in production.
It depends on conditions, not just the model and hardware. Batch size, concurrency, and request arrival distribution change the latency the same model produces on the same accelerator. Changing any of these without re-stating the configuration changes the number being reported. This is an observed pattern across the inference deployments we look at: the model and the GPU stay constant, the batch policy moves, and the “latency” number triples or halves with no other change.
How AI inference latency differs from latency in other domains
Milliseconds serve as the common unit across all these contexts. The physical quantities those units describe are not. The table below names the two events that bound the elapsed-time measurement in each domain, and the conditions that govern it.
| Domain | What latency is | What it depends on |
|---|---|---|
| Networking | Round-trip transit time of a packet between endpoints | Distance, link bandwidth, queueing, protocol overhead |
| Storage | Time from I/O request to I/O completion | Queue depth, service time at the storage device, caching layer |
| Database query | Time from query submission to result return | Query plan, index hit/miss, lock contention, IO subsystem |
| Web service | Time from HTTP request to response received | Application processing + downstream calls + network legs |
| AI inference | Time from request arrival to inference output completion | Batch size, concurrency, model size, precision, executor saturation |
A networking latency of 5 ms (illustrative) and an inference latency of 5 ms (illustrative) are not comparable as “system performance”; they are reporting on different operations against different infrastructure with different governing dynamics. The networking number describes packet transit; the inference number describes computation that includes everything from queue admission through CUDA kernel launches on the accelerator to result serialization.
A benchmark report that mixes these without scoping each — for example, claiming an “end-to-end latency” of N ms without separating the network leg from the inference leg — is folding incommensurable quantities into a single number that no reader can decompose. The reader cannot tell whether tightening the network would move the headline number by 10% or by nothing at all.
This is also where the distinction between model latency and end-to-end system latency matters operationally. Model latency is the time a forward pass takes once execution has begun on the accelerator; end-to-end system latency is the per-request number defined above. A model-latency improvement of 2 ms (illustrative) is invisible to a user whose end-to-end latency is dominated by queueing under load. Conflating the two is one of the more common reasons inference benchmark results fail to predict production behaviour.
Why a single average latency under-specifies AI inference
Request-to-request variation under load in AI inference grows large enough that relying on a single mean or median becomes operationally misleading. The reasons are mechanical, not statistical sophistication for its own sake.
- Batch effects. When the inference server batches requests, the latency a request experiences depends on where in the batch window it arrived. The first request in a forming batch waits for the batch to fill or the timeout to fire; the last request in a forming batch experiences near-zero queue time but the same kernel execution time.
- Concurrency effects. Under sustained concurrent load, queue depth fluctuates, and request latencies spread accordingly. Average latency under a load pattern hides the worst-case behaviour the system is actually exposed to.
- Saturation effects. As load approaches the AI Executor’s saturation point, latency distributions become heavy-tailed: a small fraction of requests experience much larger latencies than the median while the median moves only slightly. The mean drifts up; the tail explodes.
The minimum useful reporting unit for AI inference latency is therefore a percentile distribution under declared load conditions: p50, p95, p99 — and frequently p99.9 for latency-sensitive systems — at a stated batch size, concurrency, and arrival distribution. A single average number under load is structurally incapable of expressing what a latency-sensitive deployment needs to know about the system. We treat this as an observed pattern across the latency-sensitive deployments we audit, not as a benchmarked rate from a single named test.
The strategic argument — when latency is the right target at all, versus when throughput should be — lives in throughput vs latency trade-offs. Operationally, the trade-off between the two metrics is governed by the latency distribution, not by a point estimate of latency, and benchmarks that report point estimates leave the trade-off un-evaluable.
Token-level latency in LLM serving: TTFT, TPOT, ITL
Per-request timing envelopes in autoregressive generation contain multiple phases that any aggregate metric will collapse into obscurity. Time-to-first-token (TTFT) is the elapsed time from request arrival to the first output token, and it carries the prefill computation plus whatever queue time the request accumulated. Time-per-output-token (TPOT) is the average interval across the decode phase, and inter-token latency (ITL) is the per-step interval whose distribution reveals scheduling jitter that TPOT’s average smooths away. A request can have an excellent TPOT and a user-hostile TTFT, or a stable average with visibly stuttering ITL — three different failure shapes, one aggregate.
They also expose different halves of the trade-off. TTFT degrades first when batching is aggressive, because prefill for an incoming request waits behind the current batch window. TPOT and ITL degrade when the decode phase shares an accelerator with more concurrent sequences than its memory bandwidth comfortably serves. A serving stack tuned for tokens-per-second across all sequences can be simultaneously optimal on throughput and unacceptable on TTFT, which is why the token-level metrics have to be reported alongside the aggregate rather than derived from it.
What disclosure makes an AI latency number meaningful?
A latency number for AI inference becomes interpretable when the report names:
- The model and its size.
- The precision regime of the inference (FP32 / FP16 / BF16 / INT8 / FP8 / quantization scheme).
- The AI Executor — accelerator, driver, runtime, framework (PyTorch, TensorRT, ONNX Runtime), and inference server.
- The batch size policy: static, dynamic with timeout, or continuous batching.
- The concurrency level under which latency was measured.
- The request arrival distribution: closed-loop, open-loop, or a specific load shape.
- Which percentiles are reported (mean alone is insufficient), and for generative workloads, TTFT, TPOT, and ITL separately.
- Whether warm-up was excluded and how long the measurement window was.
A latency report that satisfies this list is informative. A latency report that names a number without these dimensions is reporting on an unspecified executor under unspecified conditions, and any reader who tries to compare it to their own deployment is comparing a known thing against an unknown thing. The numerical answer is the same; the epistemic status is incommensurable.
Where a throughput benchmark stops, and what has to be measured instead
State this consequence explicitly: the scope of the definition determines which questions any measurement tool can credibly address. A LynxBenchAI run raises the batch size until throughput stops improving inside a defined noise band, then counts completed iterations inside a continuous timed window after a discarded warm-up. That is a saturated-throughput measurement by construction. It does not report percentiles, tail behaviour, TTFT, or inter-token latency, and no re-reading of the throughput figure supplies them — the run does not retain per-request arrival times or per-request completion times, so the distribution simply was never recorded.
Getting a tail-latency answer on the same hardware means running a second, differently shaped experiment: an open-loop load generator at a fixed request rate below the saturation point, with per-request timestamps retained, reported as percentiles. The throughput run is still useful input to that experiment — it tells you where the device saturates, which bounds the operating range worth sweeping — but it is an input, not a substitute. Treating a saturated-throughput number as a latency number is the specific error this definition exists to prevent.
The framing that helps
AI inference latency tracks per-request, end-to-end elapsed time under specified batch, concurrency, and load conditions—a distinct physical phenomenon from delays in network hops, disk seeks, query execution, or HTTP round-trips. A useful AI latency report names percentiles under declared conditions, not an average without context, and for generative serving it separates the prefill and decode components rather than folding them together. Before the next AI latency number anchors a deployment decision, which percentile — under what batch policy, what concurrency, what arrival distribution, on which AI Executor — produced the figure, and was that figure produced by an instrument that measured latency at all?
Frequently Asked Questions
What do TTFT, TPOT, and ITL each measure in LLM inference?
Time-to-first-token (TTFT) is the elapsed time from request arrival to the first emitted token, so it carries queue time plus prefill computation. Time-per-output-token (TPOT) is the average interval across the decode phase, and inter-token latency (ITL) is the per-step interval whose spread exposes scheduling jitter that TPOT’s average conceals. They fail independently: a request can show a healthy TPOT alongside a user-hostile TTFT.
How should latency be reported across percentiles so tail behaviour stays visible?
Report p50, p90, p95, and p99 together — and p99.9 for genuinely latency-sensitive systems — each tied to a stated batch-size policy, concurrency level, and request arrival distribution. As load approaches the executor’s saturation point the distribution turns heavy-tailed, so the median moves only slightly while the upper percentiles climb sharply. An average taken over that distribution reports neither the typical experience nor the bad one.
If a benchmark reports saturated throughput, what extra measurement gives a tail-latency answer?
An open-loop load generator driving a fixed request rate below the saturation point, retaining per-request arrival and completion timestamps, and reporting the resulting percentile distribution. The saturated-throughput run cannot be re-read to supply this because it never recorded per-request timings — it counted completed iterations inside a timed window. The throughput figure does bound the useful operating range to sweep, so it is an input to the latency experiment rather than a replacement for it.
Does AI inference latency mean the same thing as networking or storage latency?
No. The units are shared — usually milliseconds — but the physical quantities are not. Networking latency measures packet round-trip transit; AI inference latency measures the per-request elapsed time from request arrival to output completion, including queue time, framework dispatch, kernel execution, and serialization. A 5 ms networking number and a 5 ms inference number (illustrative figures) describe different operations on different infrastructure and are not comparable as “system performance.”
Why every millisecond counts differently
Wall-clock delay from submission to final token delivery defines AI inference latency, sweeping compute cycles, memory bandwidth, and queueing delays into a single observable. So the question to carry forward is this: do you know the executor, the rules, and the release behind the number in front of you — and if not, what would it take to find out?
