Introduction

Modern AI workloads demand GPUs because dense linear algebra over batched tensors—the mathematical foundation of training and inference—maps directly onto the parallel execution units a GPU provides. A CPU executes a few threads quickly; a GPU executes thousands of arithmetic operations simultaneously. For a neural network whose forward pass is dominated by matrix multiplies, that throughput difference is the difference between a model that trains in hours and one that trains in weeks.

The interesting question, though, is not whether you need GPUs. It is whether you are actually using the ones you have. Enterprise GPU infrastructure purchased for AI workloads routinely sits underutilised — memory bandwidth unused, compute cores idle during data transfer, batch sizes that leave most of the silicon on the table — and the cost compounds monthly. You cannot know what you are wasting until you profile.

What this means in practice

• The reason GPUs accelerate AI is parallel arithmetic throughput on dense tensors, not a single magic primitive.
• GPU-busy % is the most misleading utilisation metric in common use — a GPU can be “busy” doing memory operations while its compute units stall.
• The honest cost metric is TCO per useful FLOP, not TCO per purchased FLOP.
• Most cloud GPU spend that gets approved for “more capacity” would be undercut by profiling the workloads already running.

How do I calculate the true cost of an underutilised GPU fleet?

Three cost components hide in most balance sheets, never itemized separately. First, the capital or rental envelope — the sticker price of a purchased H100, or the per-hour rate of a rented instance. Second, the utilisation-adjusted denominator — what fraction of that capacity actually ran productive AI work, where productive is defined by your application’s success metric rather than by the operating system’s idle counter. Third, the opportunity cost — what the same money would have bought if redirected to optimising the existing fleet (better data pipelines, larger batch sizes, more aggressive kernel fusion) instead of adding capacity.

A defensible calculation looks like: monthly spend × (1 − fraction-of-time-doing-useful-work) = the hidden waste line item. For a typical enterprise AI team that has not profiled, that figure lands between 30% and 60% of GPU spend as a rule of thumb from field observations. It is rarely zero. The corrective question is not “do we need more GPUs” but “what are the existing ones doing when they are nominally working?”

What does “GPU utilisation” actually measure — and why is the GPU-busy percentage misleading?

nvidia-smi’s GPU-Util field—the most commonly cited metric—reports the percentage of time during the sample window when at least one streaming multiprocessor kernel was active. It says nothing about how many SMs ran, what fraction of memory bandwidth was used, or whether the kernels achieved anything close to peak throughput. A GPU running a small kernel that occupies one SM at 5% achieved throughput will report GPU-Util: 100% for the duration. That number tells the operations team the device is fully loaded; it tells the engineering team almost nothing.

The metrics that matter live one layer deeper: SM occupancy (what fraction of the available warp slots are filled), memory bandwidth utilisation (what fraction of the HBM read/write capacity is consumed), Tensor Core utilisation for the AI-relevant fp16/bf16/int8 paths, and kernel time vs idle time per training step. Tools like NVIDIA Nsight Systems, NCCL profiler outputs, or PyTorch’s built-in profiler expose these directly. The first time a team profiles a workload against these metrics rather than GPU-Util, the typical finding is that “100% utilised” actually means “doing memory copies”.

How do I compute total cost of ownership per useful FLOP rather than per purchased FLOP?

An H100 SXM ships ~3.96 PFLOPS at fp8 with Tensor Cores; that advertised peak defines your purchased FLOPs. Useful FLOPs are what your specific workload achieves end-to-end on real data, end-to-end being the relevant qualifier — it has to include data loading, preprocessing, inter-GPU communication, gradient synchronisation, and the periods when the device is waiting for something else.

The calculation is mechanical once you have profiler output. Take the achieved FLOP rate for a representative training step (the profiler computes this), multiply by the fraction of wall-clock time the step actually ran (the remainder being data-loading stalls, communication waits, evaluation pauses, checkpointing), and divide your monthly amortised cost (capital depreciation plus power plus data-centre overhead, or cloud invoice) by that number. The result is dollars per useful FLOP-hour. Two organisations running the “same” model on the “same” hardware routinely differ by 3-5× on this metric—in our experience, the differential is almost entirely in the data pipeline and the kernel-launch overhead, not in the GPU itself.

Which workload patterns most often leave GPU capacity on the table?

Most waste in production AI infrastructure falls into four recognizable patterns. Small batches — fixed by either increasing the batch size until memory bandwidth saturates, or by switching to a model architecture whose forward pass scales better. CPU-bound data pipelines — fixed by moving augmentation onto the GPU, increasing dataloader workers, or pre-processing data offline. Synchronous evaluation between training steps — fixed by overlapping evaluation with the next training step on a separate stream, or by moving evaluation off the training device entirely. Distributed training with poorly tuned collective communication — fixed by checking the NCCL ring/tree topology against the physical interconnect, and by sizing gradient buckets to hide allreduce behind backward computation.

None of these fixes require buying anything. All of them require profiling first to know which one applies.

Should I procure additional GPU capacity or first profile the utilisation of what I have?

The decision rule we apply in audits is simple: until your existing fleet runs at greater than ~60% achieved FLOP utilisation (a rule of thumb from our audit practice) on your representative workloads, additional capacity will only multiply the waste.

What cost savings are realistic from optimising utilisation versus renting more cloud GPUs?

At this stage, precision in GPU utilization measurement becomes critical. The lower end captures the easy wins (batch sizing, dataloader tuning, mixed-precision adoption); the higher end requires per-kernel optimisation, model architecture adjustment, or a re-think of how the data pipeline meets the device. Beyond 60% (as a rule of thumb) the marginal effort typically exceeds the marginal saving, and the decision shifts to model or hardware refresh.

The cost frame matters: the saving is recurring (every month thereafter), whereas additional cloud capacity is a recurring cost. The net present value of profiling once and using less is consistently larger than the NPV of renting more.

Limitations that remained

What follows is a map to the savings and the metrics that reveal them; execution remains your responsibility. Three honest gaps remain. First, profiler output is only useful if engineering time is allocated to interpret it — many teams have the data and do not have the bandwidth to act on it. Second, the utilisation framing assumes a stable workload mix; teams running heterogeneous workloads (training, inference, ad-hoc experimentation) on shared infrastructure face a scheduling problem that profiling alone does not solve. Third, the savings band quoted above is based on engagements with teams that had at least one engineer willing to own the optimisation work end-to-end; teams without that ownership see the data, agree it is interesting, and then change nothing.

Three deployment patterns that justify the hardware spend

When training cycles compress from weeks into days, when inference latency satisfies SLAs beyond CPU reach, or when research iteration speed becomes the bottleneck, GPUs justify their expense. For AI infrastructure teams we run GPU performance audits that quantify per-workload utilisation against the metrics that actually matter (SM occupancy, memory bandwidth, achieved FLOPs), identify the dominant waste sources in the data pipeline and the kernel mix, and produce a prioritised intervention list with measurable before/after targets. We work with engineering teams that want to spend less on capacity by using more of what they already have. Contact us to discuss your AI infrastructure utilisation.

Image credits: Freepik.