TOPS is a marketing metric that obscures hardware comparison
Every AI accelerator data sheet now includes a TOPS figure. It sounds like a direct measure of AI processing capability. It is not. TOPS is a peak throughput figure for integer operations (INT8 or INT4) measured under ideal conditions that no real workload achieves.
This piece focuses on the hardware-software stack as the reason identical TOPS scores deliver different throughput in deployment. Two adjacent questions live in companion articles: what TOPS on the spec sheet measures, and why no transformation of the headline number predicts performance, is covered in AI TOPS on the spec sheet; how TOPS interacts with GPU utilization as a metric is covered in AI TOPS and GPU utilization. Here the subject is the stack itself — kernels, runtime, memory hierarchy, batching — as the dominant explanation for the gap between a TOPS rating and observed tokens per second.
The metric is useful for understanding the theoretical ceiling of a chip’s integer compute. It is not a basis for comparing chips, selecting hardware, or predicting inference speed. We see this confusion regularly in procurement conversations: a buyer compares two accelerators by their headline TOPS, picks the larger number, and then discovers months later that the deployed model runs slower than on the rejected option. The number was real. The framing was wrong.
What does TOPS actually measure?
The metric quantifies integer multiply-accumulate throughput at specified precision and thermal design power. The calculation is:
TOPS = peak_INT8_ops_per_cycle × clock_frequency × core_count
This measures theoretical peak throughput assuming, by definition of peak FLOPS, 100% utilization of all compute units simultaneously, zero memory bandwidth bottleneck, perfect data distribution across compute units, and no overhead for data movement, control flow, or communication.
Why TOPS comparisons mislead
Four failure modes appear repeatedly when engineers evaluate accelerators by TOPS alone.
Precision mismatches. 100 TOPS at INT8 is not the same as 100 TOPS at FP16 or BF16. Many TOPS figures are quoted at INT4, which provides roughly 2× the TOPS of INT8 but requires more aggressive quantization that may not be acceptable for your model quality requirements. A headline number stripped of its precision regime is decorative, not diagnostic.
Utilization ceilings. Observed real-world utilization typically sits in the 30–70% range of peak TOPS depending on workload structure, memory access patterns, and model architecture (observed pattern across our infrastructure engagements; not a benchmarked rate). The remaining headroom rarely closes without substantial kernel-level work — FlashAttention, kernel fusion in TensorRT or XLA, NCCL topology tuning — and even then it closes asymmetrically across operator types.
Memory bandwidth ignored. A chip with 200 TOPS and 50 GB/s memory bandwidth will be memory-bandwidth-bound for most LLM inference workloads long before it approaches the compute ceiling. This is the most common mismatch we encounter in current procurement: transformer decode is dominated by KV-cache reads, and additional TOPS on a bandwidth-starved part are inert.
Different operation types. The operations that dominate transformer inference — large matrix multiplications, attention with a growing KV-cache, all-reduce across NVLink or PCIe — stress compute and memory bandwidth differently than the small dot products that dominate published INT8 benchmarks.
A worked TOPS comparison
| Chip | Quoted spec | Reality on LLM inference at 8k context |
|---|---|---|
| Chip A | 100 TOPS INT8, 100 GB/s bandwidth | Memory-bound; achieved fraction of peak well below 30% |
| Chip B | 60 TOPS FP16, 400 GB/s bandwidth | Compute-bound; materially faster end-to-end |
Despite its lower rating, Chip B delivers faster end-to-end inference. The reason is structural, not anecdotal: at 8k context, decode is dominated by streaming the KV-cache through HBM. Bandwidth, not TOPS, sets the ceiling. The performance-emerges-from-the-stack framing explains why any single-number metric — TOPS, FLOPS, even memory bandwidth in isolation — is an incomplete characterization of AI hardware performance.
The deeper point is that the same TOPS number lands differently depending on the software stack sitting on top of it. A fixed GPU runs at very different effective throughput depending on the driver version, the runtime (TensorRT-LLM, vLLM, ONNX Runtime), the compiler path (torch.compile, XLA), and the framework’s memory and scheduling behaviour. Swap the runtime under the same silicon and the achieved-over-peak ratio moves — sometimes by a factor of two — without a single hardware change. That is why performance has to be reasoned about as a stack rather than read off a spec line: the layers interact, and the interaction is where real-world AI performance is actually decided.
What is actually being compared when two machines share a GPU?
At this juncture the TOPS comparison ceases to offer predictive value. Two machines can carry the same accelerator and still be different measurement subjects, because the thing that produced the number is the executor: the device together with the backend it ran through (cuda for NVIDIA, cuda via ROCm for AMD, xpu for Intel, or CPU) and the driver, framework, and runtime present on that machine. Place two numbers side by side without those fields and you are not comparing hardware — you are comparing two unlabelled configurations that happen to share a product name.
The practical consequence is a recording discipline rather than a philosophy. For a benchmark figure to stay meaningful to someone reading it later, the record has to carry the device, the backend, the driver version, the framework and runtime versions, the precision, and the release name under which the run was made. A LynxBenchAI result is bound to that whole tuple and is not attributed to silicon alone; comparability holds within a release name, so a 26Q3 figure is read against another 26Q3 figure and not against a differently named run. That constraint is what makes the stack argument checkable instead of rhetorical. It also bounds what the run may claim: it does not record thermal state or settled-clock conditions, so it does not speak to them.
There is a related trap on the software side. If one vendor’s hardware gets weeks of kernel tuning and another gets a default path, the resulting spread says more about who did the work than about the parts. The way we hold that constant is to prepare the models once, before any target device is known, and ship the same prepared artefacts to every vendor’s hardware; where two vendors’ kernels disagree about what is permissible, the stricter constraint is applied to all of them rather than relaxed per vendor. Optimization effort then stops being the hidden variable in the comparison.
When does TOPS mislead hardware decisions?
Cross-architecture comparisons amplify TOPS unreliability, especially when workload profiles differ by design. A part optimised for INT8 edge inference and a part optimised for FP16 training cannot be compared by their TOPS numbers alone. The INT8 part may report 4× the TOPS of the FP16 part, but if your workload requires FP16 precision, the INT8 TOPS figure is irrelevant — the comparison is not just unfavourable, it is undefined.
The second common mislead is the gap between peak TOPS and achieved TOPS. Peak assumes full utilisation of all compute units simultaneously, which requires perfectly parallelisable workloads with no memory stalls, no synchronisation overhead, and no idle cycles. In our experience, AI workloads on well-optimised hardware tend to achieve roughly 30–60% of peak TOPS; on poorly optimised deployments the achieved fraction can sit below 15% (observed pattern, not a published benchmark). The ratio itself — achieved divided by peak, on the target workload at the target precision — is the diagnostic signal. A ratio below 0.3 on a well-maintained system usually indicates a memory-bandwidth bottleneck rather than insufficient compute. Adding more TOPS would not help.
For procurement decisions, we recommend specifying required performance in application-level terms — tokens per second, images per second, P99 inference latency under realistic batch and context — rather than TOPS. Two GPUs with identical TOPS specifications but different HBM bandwidth and NVLink topology will deliver materially different performance on memory-bound workloads.
TOPS at different precisions are not comparable
Vendors publish TOPS across several data types—INT8, FP16, BF16, and FP32 among them. These numbers are not interchangeable. A chip rated at 400 TOPS INT8 and 200 TFLOPS FP16 does not offer equivalent performance for INT8 and FP16 workloads — the 2× ratio reflects the arithmetic simplicity of INT8 operations, not a quality–performance tradeoff that you can dial in freely.
The practical question is which precision your workload actually requires. LLM inference at acceptable quality typically requires FP16 or BF16, sometimes with selective INT8 weight quantization through TensorRT-LLM or vLLM. Vision model inference often works well at INT8 after quantisation-aware training. Edge accelerators optimised for INT8 — Google Edge TPU, Intel Movidius, the NPUs in current laptop SoCs — deliver impressive TOPS numbers that are irrelevant if your model requires FP16; they either cannot run FP16 at all, or run it at dramatically lower throughput.
We evaluate hardware at the precision our target workload requires. A chip with 200 TOPS INT8 but no FP16 support is not useful for LLM serving. A chip with 50 TFLOPS FP16 but mediocre INT8 throughput is not ideal for edge vision deployment. Matching precision capability to workload requirement is the first filter before TOPS numbers become comparable at all.
How to read a TOPS spec without being misled
A short diagnostic checklist for the next time a vendor deck lands on the table:
- Name the precision. INT4, INT8, FP16, BF16, FP32 — and refuse to compare across them.
- Compute the bandwidth ratio. Divide TOPS by GB/s memory bandwidth. If the ratio is high relative to peers, the part is memory-bandwidth-skewed and will under-deliver on transformer decode.
- Ask for sustained, not peak. Under sustained load, on the runtime you will actually deploy (TensorRT, vLLM, ONNX Runtime, custom CUDA), with realistic batch and context.
- Record the executor. Device, backend, driver, framework, runtime, precision, release name. A number without them cannot be re-read later.
- Measure achieved-over-peak on your workload. This single ratio collapses most of the procurement debate into one defensible number.
- Restate the requirement in application terms. Tokens per second at a given P99 latency, frames per second at a given resolution. TOPS is then a sanity-check on the upper bound, not the specification.
The open question is not whether TOPS overstates a chip — everyone concedes that much. It is how much of the remaining spread between two well-run machines is the silicon at all, once the backend, driver, framework, and runtime are written down next to the number. What would you need recorded in your own environment before you would be willing to answer that?
Frequently Asked Questions
If two machines have the same GPU but different drivers, frameworks, and runtimes, what exactly is being compared when their benchmark numbers are placed side by side?
Not the hardware. What produced each number is the executor — the device plus the backend it ran through and the driver, framework, and runtime present on that machine — so two same-GPU machines with different software above the silicon are different measurement subjects. Placed side by side without those fields, the comparison silently attributes a software delta to the chip. This is why a result is bound to the whole tuple rather than to the product name.
What has to be recorded alongside a benchmark number for that number to remain meaningful to someone else later?
The device, the backend, the driver version, the framework and runtime versions, the precision, and the release name of the run. Comparability holds within a release name, so a 26Q3 figure is read against another 26Q3 figure rather than across differently named runs. Anything the run does not measure — thermal state, settled clocks — must not be inferred from it.
How do you keep optimization effort from becoming the hidden variable when the same model is run across hardware from different vendors?
By bounding the effort identically: the models are prepared once, before any target device is known, and the same prepared artefacts ship to every vendor’s hardware. Where two vendors’ kernels disagree about what is permissible, the stricter constraint is applied to all of them rather than relaxed per vendor. Without that discipline, a spread between parts can just be a record of who received more tuning attention.
Where does a hardware-versus-software framing stop being useful for reasoning about AI workload performance?
It stops being useful as soon as the dominant effects are interactions rather than layers — kernel quality against the memory hierarchy, or how the runtime streams a KV-cache through HBM at long context. Specs on one side and code on the other implies the two can be scored separately and added up, which is precisely what achieved-over-peak ratios refuse to do. Past that point the stack is the unit, and the systems framing is the more productive one.
TOPS ratings rarely predict your application speed
Memory bandwidth constraints, sparsity acceleration, and toolchain quality frequently allow a 30 TOPS device to outperform one rated at 45 TOPS. If any of those differ, are you still looking at a comparison, or two unrelated observations?
