GPU utilization percentage is a poor AI performance metric
nvidia-smi reports GPU utilization as a percentage. Teams running AI workloads often treat this number as a performance indicator: high utilization equals good, low utilization equals wasted compute. That interpretation is wrong often enough to cause real engineering and procurement mistakes — and it is a clean example of why synthetic benchmarks systematically fail to match real production workloads.
GPU utilization percentage measures how often the GPU was executing at least one kernel in the last sampling window. A GPU running a single inefficient kernel 100% of the time shows 100% utilization. A GPU running the same workload 10× faster also shows 100% utilization. The headline numbers are identical; the performance is not. This is a benchmark-realism problem disguised as a metric: the number is honest, but the question it answers is not the one most teams think they are asking.
What does nvidia-smi GPU utilization actually measure?
NVIDIA’s published documentation defines GPU utilization as “Percent of time over the past sample period during which one or more kernels was executing on the GPU”—a binary measure asking whether the GPU was doing anything in each tick of the sampling window, not how efficiently it was working, not how many of its compute units were active, and not whether the work was useful for the model being served.
The metric was designed for fleet monitoring — was this card idle, or did something run on it? — and not for performance characterisation. Reading it as a performance metric is a category error. The card can be 100% busy moving memory around, or 100% busy executing a poorly fused custom kernel, and nvidia-smi cannot tell the difference between either case and a well-optimised compute-bound training step.
Where the utilization interpretation breaks
A small table sharpens the failure modes. Each row shows a real workload shape that produces a misleading headline number, and what the actual state of the hardware is underneath.
| Situation | GPU utilization (nvidia-smi) | Actual state |
|---|---|---|
| Training well-optimised large model | ~100% | Efficient — compute-bound |
| Training with data-loading bottleneck | ~100% (during compute) | Inefficient — bubbles between compute bursts |
| Inference at batch=1 | Often 40–70% | Expected for latency-optimised serving |
| Memory-bandwidth-bound operation | ~100% | Expected — limited by memory, not compute |
| Poorly optimised custom kernel | ~100% | Inefficient — many compute units idle |
100% GPU utilization (as a definitional ceiling) can mean either “the hardware is being used efficiently” or “there is a kernel running that is not efficiently using the GPU’s compute units.” The metric does not distinguish. This is the structural mismatch that benchmark consumers most often miss: utilization is a presence signal, not a productivity signal, and a benchmark that reports only utilization is benchmark-shaped without being workload-shaped.
Better metrics for AI GPU performance
“Is this GPU doing useful work for my model?” requires an answer from a different layer of the stack. The metrics below are what we collect alongside utilization in any serious benchmarking pass.
| Metric | What it measures | How to get it |
|---|---|---|
| MFU (Model FLOPS Utilization) | Fraction of theoretical FLOPS achieved | Manual calculation from throughput |
| SM Occupancy | Fraction of SMs with active warps | NSight Compute |
| Memory bandwidth utilization | Fraction of peak bandwidth used | NSight Compute / DCGM |
| Actual throughput (items/sec) | The outcome the system exists for | Application-level measurement |
Reporting any single one of these in isolation is also misleading — that is the broader benchmark-realism point — but together they form a triangulation that exposes which resource is actually the bottleneck.
Why does high GPU utilization not mean high performance?
100% utilization can coexist with poor performance. The utilization metric from nvidia-smi indicates that at least one CUDA kernel was active during each sampling period — it says nothing about what that kernel was doing. A memory-copy kernel, a poorly parallelised custom kernel, or an inefficient attention implementation all show as 100% utilization while leaving most of the GPU’s compute units idle.
The distinction matters for capacity planning. A system reporting 95% GPU utilization (illustrative threshold) appears to have no headroom, but profiling can reveal that a large fraction of that time is spent on suboptimal kernels that could be replaced with fused or vendor-optimised alternatives. We have seen, in our engagements, cases where replacing a hand-written CUDA kernel with a cuDNN-optimised equivalent cut inference time substantially — with no change in the headline GPU utilization percentage. The card stayed near 100% busy (as observed in our project work); each unit of busy time simply processed more useful tokens (observed-pattern, from project work; not a general benchmark claim).
For benchmark testing, GPU utilization should always be reported alongside throughput in samples per second or tokens per second. If two configurations both show roughly 98% utilization but configuration A processes meaningfully more samples per second than configuration B, configuration A is more efficient despite identical utilization. This is the typical shape of a result when one configuration uses optimised paths — FlashAttention, torch.compile, TensorRT, fused kernels via Triton — that extract more useful work from each GPU cycle.
Batching moves the number without moving the bottleneck
Turn on the dynamic batching that serving stacks such as vLLM apply — continuous batching, where new requests join an in-flight batch rather than queueing behind a fixed one — and the same hardware reads far closer to 100% (as a methodological contrast to single-stream) while the reported per-request latency distribution changes shape entirely.
Nothing about the silicon changed between those two readings. What changed was the arrival pattern and the batch composition, which is to say the workload shape. That is the general result worth carrying: workload shape dominates the observed number, and utilization is the metric that hides this most effectively, because both the single-stream and the continuously batched case can be described with the same one-line summary.
The practical consequence for anyone reading a published figure is that a utilization percentage without a declared concurrency and batching policy is not interpretable. Neither is a throughput figure without them, but at least throughput fails loudly — it moves when the concurrency moves. Utilization can pin near its ceiling across a wide band of very different operating points.
Workload shape dominates the signature you see
Understanding the utilization signatures produced by different AI workload types separates useful production proxies from synthetic artefacts.
Training workloads typically show high, steady GPU utilization with periodic dips that correspond to gradient synchronisation in distributed training. Shallow dips suggest efficient collective communication via NCCL or NVLink; deep dips point at a communication bottleneck, often a PCIe topology issue or a poor sharding choice. The dip shape, not the average, carries the signal.
Inference serving is the inverse. Utilization tracks request load: at low concurrency, both utilization and latency are low; as concurrency rises, utilization climbs while latency stays roughly flat — until a saturation point, beyond which utilization plateaus near 100% and tail latency rises sharply. The benchmark question is not “what is the peak utilization?” but “where is the saturation knee, and what is the tail latency just before it?” That knee is workload-specific, and a single-stream synthetic benchmark cannot see it at all. The realism question here is not binary; it is about whether the test reproduces the concurrency, queuing, and request-size variability the production system actually experiences.
For teams setting up GPU benchmarking practices, the practical rule is to collect all three metrics — utilization percentage, achieved memory bandwidth, achieved arithmetic throughput — from the first benchmark run, even if only one seems relevant. The complete dataset enables retrospective analysis when a regression appears months later. Collecting incomplete metrics initially and bolting on more later produces a fragmented history with no consistent baseline across time periods, which is one of the most common procurement-grade benchmark failures we encounter.
What a scoped instrument can and cannot tell you
Any published measurement should state its scope rather than imply coverage it does not have. Why benchmarks fail to match real AI workloads sets out the general structure of that gap; the specific case is easier to reason about. A LynxBenchAI score covers a fixed model catalogue at fixed precisions for one named release, measured inside a continuous timed window after a discarded warm-up. It scales the workload to each device’s own saturation point rather than fixing one workload across very different hardware, which removes the fixed-workload bias that flatters one device class. It does not reproduce your request mix, your queuing, or your concurrency profile, and it does not claim to.
That is the difference between a scoped instrument and a prediction. The scoped instrument tells you something checkable about a declared set of conditions and names what falls outside them; the prediction quietly asserts that its conditions were yours all along. GPU utilization is not performance walks through the reasoning behind why the headline number misleads even inside a well-declared scope.
A correct interpretation pattern
When diagnosing AI performance, the order of operations matters:
- Measure actual throughput first (tokens/sec, images/sec, requests/sec at a declared concurrency).
- Check whether memory bandwidth is saturated, using NSight Compute or DCGM.
- Only then interpret GPU utilization as context — never as the headline metric.
The question to put to any GPU-utilization-driven performance claim is whether the utilization number is paired with throughput on the actual workload at a declared operating point, or whether it is being read as a proxy for productive work it does not measure. Utilization percentage is one of the cleanest examples of this failure class: a number that is technically correct, widely reported, and almost always read as something it is not. Is GPU-utilization percentage actually the binding constraint on what your workload delivers under sustained load at your real concurrency — or is it a number that flatters a different question entirely?
Frequently Asked Questions
How does request batching change observed GPU utilization compared to a single-stream benchmark?
On AI GPU Utilization Testing, the short answer is this. A single-stream benchmark leaves gaps between requests, so utilization reads well below saturation even on hardware that is far from its limit. Dynamic or continuous batching in serving stacks such as vLLM lets new requests join an in-flight batch, which pushes the same hardware close to 100% utilization (as an observed ceiling under continuous load) and reshapes the latency distribution. The silicon did not change; the arrival pattern and batch composition did, which is why a utilization figure without a declared concurrency and batching policy is not interpretable.
What does benchmark saturation mean, and why does a suite everything scores well on stop being useful?
Saturation here has two senses worth keeping apart. At the workload level it is the knee where utilization plateaus and tail latency starts rising sharply — the operating point a procurement decision actually cares about. At the suite level, a benchmark saturates when every candidate device scores near the top, at which point the numbers no longer separate the options and the comparison has to move to a harder or more workload-specific test.
How should I reason about a published score whose model catalogue, precisions, and measurement window are all fixed while my stack differs?
Treat the score as a statement about a declared set of conditions, not a forecast for your deployment. A LynxBenchAI figure, for instance, covers a fixed model catalogue at fixed precisions for one named release, measured inside a continuous timed window after a discarded warm-up. Use it to narrow the candidate set on grounds it genuinely measures, then measure the parts it explicitly does not cover — your request mix, your queuing, your concurrency — on your own stack.
Can I compare GPU utilization figures between two different benchmark releases?
No, and this is the same class of error as comparing scores across releases. A utilization number is only meaningful against a stated model set, precision, concurrency, and measurement window; change any of those and the reading moves for reasons unrelated to the hardware. If you need a cross-release comparison, re-run both configurations under one release rather than lining up two published headline numbers.
Does high utilization guarantee speed?
Utilization pinned at 98% (illustrative threshold) can coincide with throughput lagging by half when memory bandwidth starvation occurs. So the question to carry forward is this: do you know the executor, the rules, and the release behind the number in front of you — and if not, what would it take to find out?
