Reasoning-heavy AI workloads have changed the shape of inference. Token-by-token decoding, long context windows, and tight latency SLAs mean the inference tier now competes with training for the most capable GPUs — and for engineering attention. CUDA sits at the centre of that story, not because it is glamorous, but because almost every practical decision about latency, throughput, and cost per request resolves down to how CUDA kernels move data through a GPU. This article works through what that means in practice. We focus on the operational layer: where latency actually goes, which CUDA-level levers move it, and where the temptation to scale out hardware obscures a cheaper algorithmic fix. For a structured walkthrough of the diagnostic side of that work, our companion piece on how to optimise AI inference latency on GPU infrastructure covers the profiling sequence in detail. What CUDA actually is, in operational terms CUDA is NVIDIA’s parallel computing platform and programming model. Kernels run as grids of thread blocks across the streaming multiprocessors of a GPU, with memory tiered into global, shared, and register storage. Streams and events let independent work overlap so the device is not stalled waiting on host-device transfers. That is the textbook view, and it has been stable since the early programming guides. The operationally relevant point is narrower: almost no production team writes raw CUDA kernels for an inference path. They use PyTorch, TensorFlow, or vLLM, which call into cuDNN, cuBLAS, FlashAttention, and TensorRT, which in turn launch CUDA kernels. The performance you see in production is the performance of those library kernels under your specific request mix, not the performance of CUDA in the abstract. In our experience across GPU engagements, the largest latency wins come from changing which library path is invoked, not from rewriting kernels. That framing matters because it disciplines where to spend engineering time. We treat the CUDA layer as a contract: the libraries promise certain throughput envelopes on certain hardware at certain precisions, and our job is to keep the serving stack inside those envelopes. For a deeper look at how that contract shapes tool choice, see CUDA, frameworks, and ecosystem lock-in. Why reasoning workloads stress the GPU differently Classical batch inference — image classification, embedding generation — is throughput-bound and forgiving on tail latency. Reasoning workloads invert this. A long-context LLM serving a chat or agent loop has three properties that change the optimisation target: Output is autoregressive. Each token depends on the previous, so latency is the sum of many small kernel launches rather than one large one. KV-cache memory dominates. For long contexts, the attention cache can exceed model weights in VRAM footprint, which constrains batch size more than compute does. Request shapes vary. Prompts range from tens to tens of thousands of tokens, so static batching wastes either compute or memory. This is why FP8 Tensor Cores on Hopper-class GPUs and the Transformer Engine library matter more for reasoning than for older workloads. FP8 reduces both compute time and the bytes pushed across the memory hierarchy per token. As reported in the NVIDIA Hopper architecture documentation (benchmark-class measurement, vendor-published), FP8 paths roughly double matmul throughput over BF16 on H100 silicon, and independent benchmarking work (Luo et al., arXiv 2402.13499, 2024) confirms the directional gain on representative kernels. Where inference latency actually goes Diagnosing the bottleneck Latency component Symptom First CUDA-level check Model compute High GPU SM utilisation, latency scales with model size Precision (FP8/BF16/INT8), kernel fusion via TensorRT Memory bandwidth Low SM utilisation, high DRAM traffic KV-cache layout, attention kernel choice (FlashAttention) Batching High p99 vs p50 gap, low GPU utilisation under load Continuous batching scheduler, request admission policy Host-device transport CPU-side waits, PCIe saturation Pinned memory, CUDA streams, GPUDirect RDMA Inter-GPU collectives Latency rises non-linearly with tensor-parallel degree NVLink/NVSwitch topology, NCCL configuration This is an observed pattern across our GPU optimisation engagements, not a benchmarked rate — actual proportions vary heavily by model, context length, and traffic shape. The point of the table is to discipline the diagnosis: profile before changing anything, and confirm which row you are actually in. A common mistake is to assume the bottleneck is compute when it is memory. Long-context attention is memory-bandwidth-bound on most current hardware, which is why FlashAttention’s restructuring of the attention kernel — fusing the softmax and the matmul to keep intermediate state in shared memory — yields larger wins than scaling out. The lesson generalises: algorithmic restructuring often beats hardware scaling. We discuss this pattern at greater depth in algorithmic restructuring vs kernel tuning for GPU speedups. Precision: when quantisation actually helps FP8 and INT8 are not free. They reduce memory footprint and compute time, but they require calibration, and not every model preserves accuracy at every precision on every layer. The honest framing has three cases: Memory-bound regimes. Long-context decoding with a large KV-cache. Here FP8 cache storage roughly doubles effective batch size, which translates almost directly into throughput and lower cost per token. The latency win is real. Compute-bound regimes. Short-context, high-batch serving on dense matmuls. FP8 Tensor Cores accelerate the matmul itself, with measurable latency reduction at iso-accuracy when calibration is done properly. Latency-bound regimes with small batches. Single-stream chat at batch size 1. FP8 helps less here because kernel launch overhead and host-device synchronisation dominate, not raw matmul cost. CUDA Graphs and kernel fusion matter more than precision. A widely cited evaluation (Zhou and Yang, Texas State University, ICESS 2022, published-survey) reported TensorRT achieving substantial throughput improvements with INT8 at preserved accuracy on a range of image and language models. Our operational reading: quantisation pays off most when you can verify the accuracy delta on your evaluation set, not on the original paper’s. Skipping that step is the most common reason quantisation rollouts fail in production. Interconnects: why NVLink matters before NCCL settings do Once a model is sharded across multiple GPUs, the interconnect becomes the latency floor. An H100 NVLink fabric inside a DGX-class node provides up to ~900 GB/s per GPU of intra-node bandwidth (NVIDIA Hopper documentation, vendor-published benchmark). InfiniBand at 400 Gb/s class handles inter-node collectives. The order of operations matters: Topology first. If tensor-parallel shards cross a slow link, no amount of NCCL tuning recovers the loss. Confirm that tensor-parallel groups stay inside NVLink-connected GPU sets, and pipeline-parallel boundaries cross the slower inter-node fabric. GPUDirect RDMA second. Confirm tensors move device-to-device without staging through host memory. This is the single most common configuration mistake we see in multi-node inference deployments. NCCL tuning third. Algorithm and protocol selection inside NCCL can shave further latency, but only after the topology is correct. This sequence is operationally specific to multi-GPU reasoning workloads. For single-GPU inference, none of it applies — which is why right-sizing the deployment before tuning matters as much as the tuning itself. Our piece on matching H100 GPU servers to AI inference deployment covers when the multi-GPU step is justified at all. When to optimise rather than scale out The choice between “add more GPUs” and “optimise the inference path” is genuinely a decision, and getting it wrong is expensive in both directions. A rough decision frame we use across engagements: Signal Likely correct response GPU SM utilisation >80%, latency at floor Scale out — you have a real compute ceiling GPU SM utilisation <40%, p99 latency high Optimise — batching, scheduling, or kernel choice is wrong KV-cache hits VRAM ceiling before SM utilisation does Optimise — FP8 cache, paged attention, or smaller model Inter-node collective time dominates Optimise topology first, then consider whether sharding is justified Cost per inference is the constraint, not absolute latency Almost always optimise — hardware procurement has a long payback This is an observed-pattern heuristic from our GPU engagement work, not an industry benchmark. The underlying principle is durable: hardware scaling adds capacity linearly with cost, while algorithmic optimisation often produces step-function improvements. The boundary between the two is workload-specific, which is why profiling precedes any commitment to either path. The operational picture Pulling the threads together, a CUDA-aware inference stack for reasoning workloads has four layers that have to be consistent with each other: Node design. GPUs with strong Tensor Cores and adequate memory bandwidth for the target context length. NVLink/NVSwitch inside the node when tensor parallelism is needed. Fabric. RDMA-capable interconnect across nodes, with GPUDirect end-to-end so tensors do not stage through host memory. Software. CUDA-aware frameworks calling into TensorRT, FlashAttention, or vLLM. Precision selected per layer based on accuracy validation, not assumption. Operations. End-to-end metrics — tokens per second, p95 and p99 latency, energy per request — tracked together. PUE reported under ISO/IEC 30134-2 categories when site comparisons matter. Each layer can absorb a poor choice from the layer below it for a while, then the inconsistency surfaces as a latency spike under load. The diagnostic discipline that prevents this is to measure end-to-end before adjusting any one layer in isolation. FAQ How do I diagnose where AI inference latency is being spent — model compute, memory, batching, or transport? Profile with Nsight Systems and Nsight Compute against representative production traffic, not a synthetic benchmark. The bottleneck table above maps symptoms to likely causes; the discipline is to identify which row you are in before changing anything. Compute and memory bottlenecks need different interventions. What is the most efficient GPU infrastructure for low-latency inference today? For reasoning workloads at scale, Hopper-class (H100/H200) GPUs with NVLink/NVSwitch inside the node and 400 Gb/s-class RDMA fabric across nodes is the current operational answer. The right size — single GPU, single node, multi-node — depends on context length, request volume, and model size, not on a universal recommendation. When does FP8 / INT8 quantisation actually reduce serving latency, and when does it only save memory? FP8 reduces latency when the workload is compute-bound or when KV-cache pressure was constraining batch size. It mainly saves memory when the workload is small-batch latency-bound at batch size 1 — in that regime, kernel launch overhead dominates, and CUDA Graphs or kernel fusion help more than precision. How do batching strategies (continuous, dynamic, static) trade throughput against tail latency? Static batching maximises throughput but harms tail latency when request shapes vary. Continuous batching, as implemented in vLLM and similar serving stacks, keeps GPU utilisation high while bounding p99 latency, and is the operational default for variable-shape reasoning traffic. Dynamic batching with strict deadlines suits real-time chat better than static. When should I optimise the inference path rather than scale out to more GPUs? When GPU SM utilisation is below ~50% under production load, when KV-cache memory is the binding constraint rather than compute, or when cost per inference (not absolute latency) is the operational target. In each case, optimisation typically delivers larger and faster gains than procurement. How do I measure cost-per-inference before and after optimisation to justify the engineering work? Track tokens per second per GPU under representative traffic, multiply by the fully loaded GPU-hour cost (compute, power, cooling, amortised hardware), and divide. Report the same metric before and after each optimisation change. PUE reported under ISO/IEC 30134-2 categories provides the energy component when site costs are part of the comparison. Where this leaves us CUDA is the layer where reasoning workloads either fit the hardware or fight it. The discipline is unglamorous: profile, identify the actual bottleneck, change one thing, measure again. The pattern we keep returning to is that algorithmic and configuration choices — precision, attention kernel, batching scheduler, topology — usually move latency more than hardware scaling does. Hardware scaling is the right move when the GPUs are genuinely saturated, and the wrong move whenever they are not. TechnoLynx works with teams on exactly this sequence: profiling inference workloads, identifying where CUDA-level choices are leaving latency on the table, and producing the kind of engineering plan that justifies (or replaces) the next round of GPU procurement. If you want to walk through your inference stack with us, get in touch. References NVIDIA Developer. “CUDA Platform for Accelerated Computing.” NVIDIA Documentation. “CUDA Programming Guide.” NVIDIA. “Hopper GPU Architecture.” NVIDIA Docs. “TensorRT Best Practices.” NVIDIA Docs. “Transformer Engine User Guide.” Luo et al. “Benchmarking and Dissecting the Nvidia Hopper GPU Architecture.” arXiv:2402.13499, 2024. Zhou & Yang. “Exploring TensorRT to Improve Real-Time Inference for Deep Learning.” Texas State University, ICESS 2022. Uptime Institute. “Global Data Center Survey 2024.” Harvard Kempner Institute. “Distributed Inference Handbook.” Image credits: Freepik.
