Why wall-clock timing is not profiling

A CUDA kernel takes 12 milliseconds to execute. Is that slow? The question is unanswerable without context. If the kernel is performing 500 billion floating-point operations and the GPU’s peak throughput is 312 TFLOPS, the theoretical minimum is 1.6 milliseconds — the kernel is achieving 13% of peak, and there is significant room for improvement. If the kernel is memory-bandwidth-bound, performing element-wise operations on 4 GB of data on a GPU with 2 TB/s memory bandwidth, the theoretical minimum is 2 milliseconds — the kernel is achieving 17% of the memory bandwidth ceiling, and the optimisation target is memory access patterns rather than compute efficiency.

Wall-clock timing tells you how long the kernel took. Profiling tells you why it took that long — and what specific intervention would make it faster. Without profiling, GPU optimisation is guessing. With profiling, it is engineering.

The roofline model: where does your kernel sit?

The roofline model is the foundational framework for GPU kernel performance analysis. It maps every kernel to a two-dimensional space defined by two hardware limits: compute throughput (FLOPS) and memory bandwidth (bytes/second). The kernel’s arithmetic intensity — the ratio of compute operations to memory accesses — determines which limit the kernel is constrained by.

Compute-bound kernels have high arithmetic intensity: they perform many operations per byte of data accessed. Large matrix multiplications (GEMM), convolutions with large filter sizes, and dense attention computations fall into this category. For these kernels, the optimisation target is compute efficiency: are the tensor cores being used? Is the warp execution efficient? Are there unnecessary branches or divergent execution paths?

Memory-bound kernels have low arithmetic intensity: they perform few operations per byte accessed. Element-wise operations (ReLU, sigmoid, addition), batch normalisation, and small matrix operations fall into this category. For these kernels, the optimisation target is memory throughput: are memory accesses coalesced? Is the data layout cache-friendly? Can multiple operations be fused to reuse data in registers or shared memory?

Nsight Compute generates the roofline chart automatically for any profiled kernel. The kernel appears as a point on the chart; its position relative to the compute ceiling and memory bandwidth ceiling indicates which limit is binding and how much headroom remains. A kernel sitting at 80% of the memory bandwidth ceiling is well-optimised for a memory-bound workload — further gains require reducing the memory access volume (through fusion or algorithmic change), not improving the access pattern. A kernel sitting at 20% of the compute ceiling is poorly optimised — there is a 5× potential improvement from better compute utilisation.

The profiling workflow

We follow a systematic profiling methodology that moves from system-level to kernel-level analysis. This progression is important: optimising an individual kernel that contributes 2% of total execution time is wasted effort if the system-level bottleneck is host-device serialisation.

Step 1: System-level timeline (Nsight Systems). Run the workload under Nsight Systems to produce a timeline of GPU activity: kernel launches, memory transfers, synchronisation events, and idle periods. The timeline reveals the macro-level performance structure. Questions this step answers: What fraction of the total execution time is the GPU active? Where are the idle gaps? Which kernels dominate the execution time? Is there host-device serialisation that could be overlapped?

Step 2: Identify the dominant kernels. Sort kernels by cumulative execution time. Typically, 3–5 kernels account for 80% or more of the GPU execution time. These are the optimisation targets. Optimising kernels that do not appear in the top contributors has negligible impact on total execution time.

Step 3: Kernel-level profiling (Nsight Compute). Profile each dominant kernel individually with Nsight Compute. The detailed analysis includes roofline positioning, achieved occupancy, memory throughput breakdown (global, shared, L1, L2), warp execution efficiency, and instruction mix. For each kernel, the profiler identifies the specific bottleneck and often suggests the relevant optimisation.

Step 4: Apply targeted optimisations. Based on the profiling data, apply the specific optimisation that addresses the identified bottleneck. For a memory-bound kernel: improve coalescing, fuse with adjacent operations, or change the data layout. For a compute-bound kernel: enable tensor core usage, reduce warp divergence, or increase ILP (instruction-level parallelism). For an occupancy-limited kernel: reduce register or shared memory usage per thread.

Step 5: Re-profile and iterate. After applying the optimisation, re-profile to verify the improvement and identify the next bottleneck. GPU optimisation is iterative — fixing one bottleneck often reveals the next. The process continues until the kernel’s performance is within acceptable distance of the theoretical ceiling, or until the dominant bottleneck shifts to a different kernel or a system-level constraint.

The GPU underutilisation patterns we encounter in production workloads are identified through this workflow. Without it, the team is optimising by intuition rather than by measurement.

Common profiling findings and their fixes

From our GPU Performance Audit engagements, the most common profiling findings are:

Uncoalesced global memory access. Threads in a warp access non-contiguous memory locations, resulting in multiple memory transactions where one would suffice. The fix: restructure the data layout (Array of Structures to Structure of Arrays conversion) or modify the access pattern to ensure consecutive threads access consecutive memory addresses. Typical improvement: 2–4× memory throughput for the affected kernel.

Tensor core underutilisation. The kernel performs matrix multiplications in FP32 without using tensor cores, despite running on hardware that supports them (Volta and later). The fix: convert to mixed-precision (FP16 input, FP32 accumulate) and use CUDA’s WMMA API or cuBLAS’s tensor core-enabled routines. Typical improvement: 4–8× throughput for the matrix multiplication operations.

Excessive kernel launch overhead. Hundreds or thousands of small kernels are launched sequentially, with the CPU overhead of each launch (5–15 microseconds) exceeding the GPU execution time of the kernel. The fix: fuse small kernels into larger ones, or use CUDA graphs to batch the launch sequence into a single replayable graph. Typical improvement: 2–10× reduction in total execution time for launch-overhead-dominated workloads.

Register pressure causing spills. The kernel uses more registers per thread than the hardware provides, causing register spills to local memory (which is backed by global memory and is 100× slower). The fix: refactor the kernel to reduce register usage — simplify expressions, reduce loop unrolling, or split the kernel into stages that require fewer simultaneous live variables. Typical improvement: 1.5–3× throughput recovery.

What profiling tells you that benchmarks do not

Benchmark numbers — training throughput, inference latency, images/second — tell you the current performance. Profiling tells you the achievable performance and the specific path to reach it. The gap between current and achievable is the optimisation opportunity, and its magnitude determines whether the intervention is worth the engineering investment.

We have seen profiling reveal opportunities ranging from negligible (the workload is already well-optimised, within 10% of the hardware ceiling) to transformative (the workload is at 15% of peak, with a clear path to 60% through targeted kernel optimisation and memory layout changes). The profiling data determines which situation you are in — and prevents the expensive mistake of planning GPU memory for training or procuring more hardware when the existing hardware is underutilised.

The first step is always the same: profile the workload, identify the dominant bottleneck, and quantify the gap between current and achievable performance. Everything else follows from that data. Our GPU engineering practice can help you get started.