Inference latency is an engineering problem, not a hardware problem

A team deploys a Transformer model for real-time inference. The target latency is 20 milliseconds per request. The model runs on an A100 GPU. The measured latency is 85 milliseconds. The first proposal: upgrade to H100. The upgrade delivers 45 milliseconds — a significant improvement from the hardware, but still more than double the target. The latency budget is not a hardware problem. It is a software optimisation problem that happens to run on hardware.

Inference latency is the sum of multiple components: data preprocessing, model execution (which itself comprises multiple kernel launches and memory operations), post-processing, and serving infrastructure overhead. Optimising total latency requires understanding which component dominates and what interventions are available for each. In our experience, the majority of production AI deployments exceed their initial latency targets — often by 2× or more — with model serving overhead frequently accounting for a third or more of total end-to-end latency.

MLPerf Inference v4.0 benchmark results demonstrate this directly: across submissions on the same hardware class, differences in software stack, compilation strategy, and serving configuration produce throughput variations of 2–5×. The hardware is identical; the software configuration determines the performance.

NVIDIA’s TensorRT documentation reports 2–6× inference speedup over native PyTorch on Transformer architectures, depending on model size and GPU target. MLPerf Inference v4.0 benchmark results show that INT8 quantisation achieves within 1% accuracy of FP32 for ResNet-50 and BERT, while delivering 2–4× latency reduction on A100 and H100 hardware.

The first three inference optimisation moves

1. Model compilation (TensorRT): Export the model to ONNX and compile with TensorRT to eliminate training-mode overhead, fuse kernels, and select GPU-optimised implementations. Expected improvement is 2–5× over PyTorch eager execution. Apply this first on any GPU inference workload — it is the highest-impact, lowest-effort intervention.

2. FP16 quantisation: Reduce weight and activation precision from FP32 to FP16 for a 1.5–2× additional latency reduction. Accuracy loss is typically below 0.1 percentage points. Apply by default to every inference workload unless FP32 is required for numerical correctness.

3. INT8 quantisation: Calibrate and quantise to INT8 for a further 2–4× latency reduction when FP16 alone does not meet the target. Requires a calibration dataset and accuracy validation — large language models may lose 1–3 percentage points. Apply when the latency budget is tight and the accuracy trade-off is acceptable.

Why is model compilation the first optimisation step?

Running a PyTorch or TensorFlow model in its training-time execution mode for inference is the most common source of unnecessary latency. Training-mode execution includes dynamic graph construction, eager operation dispatch, and runtime type checking — none of which are needed at inference time. Compilation eliminates this overhead.

TensorRT is NVIDIA’s inference optimisation compiler. It takes a trained model (from ONNX, PyTorch, or TensorFlow), analyses the computation graph, applies kernel fusion (combining multiple operations into single kernels to reduce launch overhead and memory traffic), selects optimised kernel implementations for the target GPU architecture, and produces a compiled inference engine. The speedup from TensorRT compilation is typically 2–5× over PyTorch eager execution, with no accuracy change for FP32 compilation.

torch.compile (PyTorch 2.x) provides graph-mode compilation within the PyTorch ecosystem. It captures the computation graph, applies graph-level optimisations, and generates optimised kernels through backends like Triton. The speedup is typically 1.5–3× over eager execution — less than TensorRT, but the workflow is simpler (no ONNX export, no separate compilation step) and the compiled model remains a PyTorch object that can be debugged and modified.

ONNX Runtime provides cross-platform inference optimisation with execution providers for multiple hardware targets (CUDA, TensorRT, DirectML, OpenVINO). For workloads that need to run on heterogeneous hardware — GPU and CPU inference targets — ONNX Runtime offers a single inference API with hardware-specific optimisation.

Our recommendation: compile the model with TensorRT as the first optimisation step. The effort is minimal (export to ONNX, run the TensorRT compiler) and the speedup is typically the largest single improvement available. We have seen production inference pipelines where TensorRT compilation alone brought latency from above-target to within-target, eliminating the need for further optimisation.

Quantisation: trading precision for speed

Quantisation reduces the numerical precision of model weights and activations from FP32 to FP16, INT8, or INT4. The latency benefit comes from two sources: reduced memory bandwidth (the dominant bottleneck for many inference workloads — moving half as many bytes per operation directly halves the memory-bound execution time) and higher compute throughput (tensor cores execute FP16 operations at 2× the rate of FP32, and INT8 at 4× the rate).

FP16 quantisation is nearly loss-free for most models — the accuracy degradation is typically less than 0.1 percentage point — and provides a 1.5–2× latency reduction. It should be applied by default for any inference workload that does not require FP32 for numerical correctness.

INT8 quantisation provides a 2–4× latency reduction but requires calibration: running representative data through the model to determine the quantisation parameters (scale and zero-point) for each layer. Post-training quantisation (PTQ) applies INT8 quantisation to an already-trained model; quantisation-aware training (QAT) trains the model with quantisation constraints, producing better accuracy preservation. The accuracy impact of INT8 varies by model architecture: large language models typically lose 1–3 percentage points of accuracy; image classification models lose 0.5–1.5 percentage points; object detection models lose 1–2 percentage points on mAP.

The accuracy-latency trade-off must be evaluated against the application’s acceptance criteria. A model that meets the latency target after FP16 quantisation with negligible accuracy loss does not need INT8 quantisation. A model that requires INT8 to meet latency targets must be validated for acceptable accuracy at INT8 precision.

Batching strategy: throughput vs latency

Batching — processing multiple inference requests in a single forward pass — improves GPU throughput (requests per second) by amortising kernel launch overhead and enabling more efficient memory access patterns. However, batching increases latency for individual requests, because each request must wait for the batch to be assembled before processing begins.

Static batching collects a fixed number of requests before processing. The latency impact is bounded: maximum additional latency equals (batch_size - 1) × inter-arrival time. For high-throughput, latency-tolerant workloads (offline batch processing, search indexing), static batching with large batch sizes maximises GPU utilisation.

Dynamic batching collects requests for a configurable time window and processes whatever has arrived when the window expires. This bounds the additional latency to the window duration (typically 5–20 milliseconds) while allowing the batch size to vary with demand. NVIDIA Triton Inference Server, TorchServe, and most production serving frameworks support dynamic batching.

Continuous batching (also called inflight batching) is specific to autoregressive models (LLMs, sequence generators) where different requests are at different stages of generation. Instead of waiting for all requests in a batch to complete before accepting new ones, continuous batching inserts new requests into the batch as in-progress requests complete. This eliminates the padding waste of static batching for variable-length sequences and maintains higher GPU utilisation.

The batching strategy depends on the latency target: if the target is strict (sub-20ms per request), dynamic batching with a short window is appropriate. If the target is relaxed (sub-500ms), larger batches improve throughput and reduce per-request cost.

Memory management: eliminating allocation overhead

GPU memory allocation and deallocation (cudaMalloc, cudaFree) are expensive operations — typically 100–1000 microseconds per call. In an inference pipeline that allocates and frees intermediate buffers on each request, the cumulative allocation overhead can be significant relative to the inference time itself.

Memory pooling pre-allocates a block of GPU memory at startup and serves individual allocation requests from the pool. The allocation cost drops from hundreds of microseconds to nanoseconds. PyTorch’s caching allocator does this automatically for PyTorch workloads; custom CUDA pipelines should use memory pools (CUDA’s cudaMemPool API or a custom pool implementation) rather than direct cudaMalloc/cudaFree calls.

Static memory planning determines the complete memory layout of the inference graph at compilation time, assigning fixed memory addresses to each intermediate tensor. TensorRT performs this automatically. For custom pipelines, static memory planning eliminates runtime allocation entirely — the memory layout is fixed, and each inference pass reuses the same buffers.

The optimisation sequence

Apply the interventions described above in this order, profiling after each step to determine whether the latency target has been met:

1. Compile the model (TensorRT or torch.compile) — see Why is model compilation the first optimisation step? above.
2. Quantise to FP16 — see Quantisation: trading precision for speed above.
3. Profile the compiled, quantised model to identify whether the remaining bottleneck is compute-bound or memory-bound.
4. Quantise further to INT8 if the latency target is not yet met and accuracy at INT8 is acceptable.
5. Optimise batching to balance throughput and latency for the serving pattern.
6. Optimise memory management if allocation overhead is significant.

If this sequence does not achieve the latency target, the next level of intervention is algorithmic restructuring — changing the model architecture, pruning, or adopting a more efficient architecture (e.g., replacing a Transformer with a linear-attention variant). These optimisation techniques become even more critical when deploying CV models on edge devices, where compute and memory budgets are a fraction of data centre hardware.

What makes inference infrastructure reliable and cost-efficient?

Latency optimisation solves one half of the inference problem. The other half is keeping the serving infrastructure stable and economically viable once traffic is live.

Reliability in inference serving means redundancy and observability. Production deployments run multiple model replicas behind a load balancer, with health checks that route traffic away from unhealthy replicas automatically. Serving frameworks like Triton Inference Server and KServe provide liveness and readiness probes, automatic restart on failure, and request queuing that absorbs traffic spikes without dropping requests. The observability layer tracks latency percentiles (p50, p95, p99), throughput, error rates, and GPU utilisation per replica — these metrics feed alerting rules that catch degradation before it breaches SLAs.

Cost-efficiency is measured as cost-per-inference, not cost-per-GPU-hour. Every optimisation in the sequence above — compilation, quantisation, batching, memory management — reduces cost-per-inference by extracting more throughput from the same hardware. A model optimised from 85ms to 18ms per request serves roughly 4.7× more requests per GPU-second. That is a 4.7× reduction in per-request infrastructure cost without adding a single GPU. When the serving load is variable, autoscaling replica counts based on request queue depth keeps GPU utilisation high during peaks and reduces spend during troughs.

Inference latency optimisation is also a critical step when moving a GenAI prototype into production — prototypes tolerate multi-second response times, but production systems rarely can. If your team has an inference latency target that the current deployment does not meet, a GPU Performance Audit identifies the specific bottleneck and the optimisation sequence that addresses it across the full inference stack.