Neither TOPS nor GPU utilization predicts AI throughput
TOPS and GPU utilization percentage both fail the capacity planning test despite widespread quotation in AI hardware specifications. TOPS is a theoretical ceiling that real workloads never approach. GPU utilization, as reported by nvidia-smi, tells you whether the GPU was busy during a sampling window, not whether the silicon was doing useful work. Worse, that coarse figure is not even the same quantity as SM utilization — the two can diverge sharply on the same workload, and confusing them is a common source of wrong conclusions. Neither answers the question buyers and operators actually want answered: how much throughput will I get for my workload, on this hardware, under realistic conditions?
This piece is specifically about TOPS as a metric paired with GPU utilization — what the two together can and cannot tell you about a running workload. Two adjacent questions live in companion articles: what TOPS on the spec sheet measures is covered in AI TOPS on the spec sheet; how the hardware-software stack turns TOPS into achieved throughput is covered in TOPS performance across the stack. This piece treats the metric pair, not the spec sheet or the stack, as the unit of analysis.
The mistake is to treat utilization as an outcome rather than a proxy. It is a proxy — a coarse, one-dimensional signal that lacks workload context. A GPU running a single memory-bound kernel can report 100% utilization while using roughly 15% of its arithmetic capacity. Another GPU running a heavily compute-bound kernel can report 40% utilization and still be saturating the relevant hardware unit. Same number on the dashboard, two completely different operational realities.
Coarse nvidia-smi utilization versus SM utilization
The single percentage nvidia-smi prints is the fraction of the sampling window during which at least one kernel was running on the device. It says nothing about how many of the GPU’s Streaming Multiprocessors were actually working. SM utilization — what NVIDIA Nsight Compute and DCGM expose as SM activity or achieved occupancy — measures the fraction of those SMs doing work. The two diverge sharply on the same workload: a single small memory-bound kernel keeps the device “busy” for the whole window, so coarse utilization reads near 100% while SM utilization sits in the low teens. That gap is exactly where the coarse metric stops being informative — a GPU pinned at 100% on nvidia-smi is not by itself a sign of trouble (a saturated training job should look like that), but it also does not confirm the SMs are saturated. Only the SM-level reading tells you which is the case.
The same divergence explains why memory utilization is a third, separate quantity. The memory figure nvidia-smi prints is the fraction of the window in which the memory interface was being read or written — not how full HBM is, and not what fraction of peak bandwidth was achieved. Three numbers, three different definitions, all commonly quoted as if they were one signal called “utilization”.
What actually determines AI throughput
The binding constraint in the performance roofline (observed-pattern framing from Williams et al.) determines throughput. roofline model, widely adopted across GPU performance literature). For any given operation, one of two things is true:
- The operation is compute-bound. Throughput is limited by GPU FLOPS. The relevant measure is MFU — Model FLOPs Utilization — the fraction of peak FLOPS the workload actually achieves.
- The operation is memory-bandwidth-bound. Throughput is limited by how fast weights and activations can be read from HBM. The relevant measure is achieved memory bandwidth as a fraction of peak.
Most LLM inference at low batch sizes is memory-bandwidth-bound — the model weights have to be streamed from HBM on every token-generation step, and arithmetic intensity per byte loaded is low. Most LLM training at large batch sizes is compute-bound, because the dense matrix multiplications inside attention and MLP layers dominate. Diffusion inference and many vision pipelines sit in between, with some layers compute-bound and others memory-bound.
This is the framing that makes utilization legible. Utilization without a roofline classification is just a number. Utilization paired with “this kernel is memory-bandwidth-bound at (illustratively) 78% of peak HBM bandwidth” is an actionable observation.
A quick map: what to measure, by workload
| Workload | Bound | Relevant metric | What to optimize |
|---|---|---|---|
| LLM inference, batch=1 | Memory bandwidth | GB/s utilization | INT8 / INT4 weight quantization |
| LLM inference, batch=64+ | Compute | MFU | Larger batch, FlashAttention, kernel fusion |
| Diffusion inference | Mixed | Both, per layer | Profile per kernel |
| CNN training | Compute | MFU | Larger batch, mixed precision |
| Embedding / feature extraction | Memory | Bandwidth | Batching, dtype reduction |
Use this as a decision rubric, not a benchmark. The point is that “the right metric” is workload-shaped — there is no single number, including utilization or TOPS, that survives across all five rows.
Why does a GPU show low utilization while still delivering high throughput?
Utilization tracks time occupancy rather than work completed. If the binding constraint for a workload is HBM bandwidth, then the moment memory bandwidth is saturated, additional SMs sit idle by definition — there is nothing for them to compute on until the next batch of weights arrives. The GPU is operating at the workload’s ceiling, but the utilization metric reports the idle SMs as headroom.
This is the canonical case where “make utilization higher” is the wrong optimization goal. Pushing more concurrent work onto a memory-bandwidth-bound kernel does not raise throughput; it raises contention. The honest answer is that the workload’s arithmetic intensity is low and the only paths forward are (a) reduce bytes per operation through quantization, (b) restructure the kernel so more arithmetic happens per byte read, or (c) accept the ceiling.
We see this regularly in production LLM serving. A team chasing a higher nvidia-smi reading by increasing concurrency hits the same tokens-per-second wall, then concludes the GPU is “underused.” It isn’t. The metric is misreading the situation.
How do you actually measure what a GPU is doing?
Three tiers, in increasing fidelity:
import torch
from torch.profiler import profile, record_function, ProfilerActivity
with profile(activities=[ProfilerActivity.CUDA],
with_flops=True) as prof:
with record_function("model_inference"):
output = model(input)
print(prof.key_averages().table(sort_by="cuda_time_total"))
PyTorch’s profiler gives you per-operation time and FLOP counts. Operations with high FLOPs and short time are compute-bound. Operations with low FLOPs but high memory traffic are memory-bound. This is enough to tell you which roofline edge you are sitting on.
For deeper analysis, NVIDIA Nsight Compute reports SM occupancy, achieved-vs-theoretical memory throughput, and achieved-vs-theoretical arithmetic throughput on a per-kernel basis. Nsight Systems gives the timeline view — useful for spotting pipeline stalls, CPU bottlenecks in data loading, and serialization between streams. nvidia-smi dmon -s u is the cheapest first pass: continuous utilization sampling to spot gross underutilization windows during a production run.
The two-stage pattern we use on engagements is: dmon to identify periods of suspicious behaviour, then Nsight Systems on a representative window to find the responsible kernels. Jumping straight to Nsight without first localising in time is how engineers lose afternoons.
There is a fourth option, and it answers a different question than any profiler. Instead of instrumenting the workload you already have, run a fixed workload catalogue under a declared timed window and count the iterations that completed. A LynxBenchAI run does exactly that — completed iterations inside a continuous measurement window, after a discarded warm-up — which makes it an outcome measure rather than an occupancy measure. It reports nothing about utilization, deliberately. Saturation, in that frame, is not read off a dial: the batch size is raised until throughput stops improving inside a defined noise band, and the point where the curve flattens is the answer a utilization percentage was never able to give.
Every such figure is bound to the AI Executor that produced it — device, backend (cuda for NVIDIA, cuda via ROCm for AMD, xpu for Intel, or CPU), driver, framework, and runtime. That binding is also why the same utilization percentage means different things on two stacks: the counter is measuring the same wall-clock property, but the software path deciding what runs during that wall-clock time is not the same.
Where TOPS and nvidia-smi actually belong
Both metrics retain some value. They are mis-deployed when treated as performance measurements.
- TOPS: use as a coarse ceiling check. If a hardware option’s published TOPS is below what your workload theoretically needs, eliminate it. Do not use TOPS to rank two options that both clear the bar — vendors compute it under different sparsity and precision assumptions, and real-world achievable fraction varies by an order of magnitude (an observed pattern across the published MFU literature; not a benchmark of any specific system).
nvidia-smiutilization: use as a coarse efficiency tripwire. Sustained utilization under ~30% during a training loop is a strong indicator of a data-loading or CPU-side bottleneck. Above that threshold, the metric loses discriminative power — it is compatible with both well-tuned and badly-tuned workloads.
The broader argument — that utilization is a proxy, not an outcome — is developed in our analysis of why GPU utilization is not performance. That piece works through the measurement gap in detail.
When does the utilization metric mislead an investigation more than it helps?
Memory-bandwidth-bound workloads expose the limitation when teams optimize for percentage alone. When two hardware options are being compared on nvidia-smi readings rather than achieved throughput per dollar or per watt. When a serving framework reports 90%+ GPU utilization but tokens-per-second has flatlined — almost always a batching pathology that utilization cannot see. And when an autoscaler triggers on utilization thresholds for a workload whose ceiling sits well below 100% by design.
The general rule: utilization is suggestive at the extremes (very low = something is stalling; very high with stable throughput = saturated) and noisy everywhere in between. Drawing conclusions from utilization in the middle of its range, without a roofline classification, is the dominant failure mode.
Why idle GPU time is not automatically waste
Procurement encounters the identical confusion from a different angle. Idle time on a GPU is read as “wasted hardware” — therefore consolidate, therefore push utilization up. But idle time can be the correct outcome of a system with bursty load and latency SLOs. A GPU that runs at 35% average utilization but serves p99 latency targets during traffic peaks is doing its job. Forcing it to higher average utilization (through co-tenancy, more aggressive batching, or workload packing) often breaks the tail-latency property that justified the deployment in the first place.
This is also why throughput per watt is the most actionable single number for production serving: (inferences per second) divided by (GPU power draw in watts). It captures hardware capability, software tuning, and workload fit in one figure, and it does not reward idle-time elimination at the expense of latency. Two systems with identical utilization percentages but different throughput-per-watt values have different optimisation states — the lower one has recoverable waste, the higher one mostly doesn’t.
A dashboard that combines GPU metrics with application metrics
For production AI serving, the operationally useful view combines GPU metrics with application-level metrics on the same timeline:
- Requests per second (application load)
- Tokens per second or images per second (application throughput)
- GPU SM occupancy (compute utilisation)
- GPU memory bandwidth utilisation (memory pressure)
- GPU power draw (energy efficiency)
Correlating these reveals patterns invisible in any single metric. A GPU at 90% utilization with flat throughput under rising request load indicates that the serving framework’s batching is suboptimal — requests are queuing rather than batching. A GPU at 40% utilization with maximum sustained throughput for the model indicates a memory-bandwidth ceiling; further load will not lift output regardless of how many SMs appear idle.
The implementation we typically reach for: NVIDIA DCGM (Data Center GPU Manager) collecting GPU metrics, exposed as Prometheus metrics, visualised in Grafana alongside application telemetry. Two to three hours of setup, and from then on the combined view does the work that a utilization dashboard alone cannot.
What remains genuinely unsettled is where the tripwire threshold belongs. The ~30% figure above is a heuristic drawn from training loops we have looked at, not a number with a population behind it — and the honest way to replace it is to execute the workload, time it, and compare completed work against the same device elsewhere rather than argue about a dial.
Frequently Asked Questions
Is there a “good” GPU utilization percentage to aim for — 70%, 90%, 96% — or does the question itself misread what the metric reports?
The question misreads the metric. Utilization reports the fraction of a sampling window in which at least one kernel was resident, so the “right” value is whatever a correctly-bound workload happens to produce — near 100% for a saturated dense training loop, 35% for a latency-SLO serving deployment that is doing exactly its job. A target percentage only makes sense once you have classified the workload against the roofline; before that, any threshold is arbitrary.
What is the difference between coarse GPU utilization, SM utilization, and memory utilization, and why can they diverge sharply on the same workload?
Coarse utilization is wall-clock occupancy of the device, SM utilization is the fraction of Streaming Multiprocessors actually computing, and memory utilization is the fraction of the window in which the memory interface was active — not how full HBM is and not the achieved fraction of peak bandwidth. A single small memory-bound kernel pins coarse utilization near 100% (illustratively) while SM utilization sits in the low teens, because the three numbers count three different things.
How is GPU utilization actually calculated by the driver, and why does that formula explain how a nearly-idle kernel can register as 100% busy?
The driver samples the device over an interval and reports the fraction of that interval during which one or more kernels were executing — a binary busy/not-busy determination per sample, with no weighting by how much of the hardware the kernel used. A single-block kernel occupying one SM is “busy” by that definition just as much as a kernel filling every SM, which is precisely how a nearly-idle device reads 100% under this definitional binary sampling logic — a rule of thumb that explains the counterintuitive measurement.
If a utilization dashboard cannot settle whether a device is saturated, what measurement does?
Execute the workload and time it. A LynxBenchAI run counts completed iterations inside a continuous timed window after a discarded warm-up, which is an outcome measure rather than an occupancy measure, and saturation is established from the throughput curve — batch size raised until throughput stops improving inside a defined noise band. That curve flattening is the evidence; the dial is not.
TOPS alone hides more than it reveals
Identical TOPS ratings across three accelerators can yield 40 percent throughput variance when memory bandwidth—not arithmetic capacity—constrains your workload. Is that executor close enough to yours for the result to mean anything?
