Generic benchmarks answer the wrong question for AI

Standard GPU benchmarks miss every metric that matters for transformer inference and diffusion workloads. These benchmarks measure compute throughput on standardized tasks that have very little in common with what AI inference and training actually do — namely, move large tensors across memory hierarchies, sustain matrix-multiplication kernels for minutes at a time, and exercise the software stack (driver, CUDA, cuDNN, framework) in ways a synthetic 30-second test never touches.

A benchmark for AI has to measure execution, not specification. That is the practical consequence of treating GPU performance as an execution property of a running system rather than a static property of the hardware: you have to run representative workloads, on the actual stack you intend to deploy, for long enough that the real bottleneck reveals itself. Everything below is the protocol we run when a team asks whether a particular box is fit for purpose. Empirical execution of real software workloads is the reference standard here; a specification-based estimate is the fallback you reach for only when running the workload is not feasible, and it legitimately supports a rough ceiling and a shortlist — nothing about achieved throughput on your stack.

The three dimensions that actually matter

How quickly your system moves weights through memory while keeping cores fed determines real-world AI performance. Any benchmark protocol that omits one of these dimensions produces misleading numbers — usually flattering ones, because synthetic burst tests light up compute peaks while bandwidth and thermals are still cold.

1. Peak compute throughput (TFLOPS)

Test whether the hardware can sustain the matrix operations AI models require, at the precision your workload uses. FP32 for training without mixed precision, FP16 or BF16 for most modern training, INT8 for production inference. PyTorch on top of CUDA and cuDNN is enough to drive this directly:

import torch, time
a = torch.randn(8192, 8192, device='cuda', dtype=torch.float16)
b = torch.randn(8192, 8192, device='cuda', dtype=torch.float16)
for _ in range(5):
    torch.matmul(a, b)
torch.cuda.synchronize()
start = time.time()
for _ in range(100):
    torch.matmul(a, b)
torch.cuda.synchronize()
elapsed = time.time() - start
flops = 2 * 8192**3 * 100
print(f'{flops / elapsed / 1e12:.1f} TFLOPS FP16')

What you are looking for is the achieved TFLOPS as a fraction of the GPU’s published peak. In configurations we’ve tested, a well-configured RTX-class workstation will land between 60% and 80% of the spec-sheet number on FP16 GEMM; anything below 60% means kernels, drivers, or thermals are leaving compute on the table, not that the silicon is broken.

2. Memory bandwidth at realistic tensor sizes

Most AI inference is memory-bandwidth-bound, not compute-bound. Bandwidth on the tensor shapes your model actually uses is the relevant number — vendor specs quote streaming bandwidth on idealised access patterns that no real model produces.

import torch, time
weights = torch.randn(4096, 4096, device='cuda', dtype=torch.float16)
torch.cuda.synchronize()
start = time.time()
for _ in range(1000):
    result = weights.sum()
torch.cuda.synchronize()
elapsed = time.time() - start
bytes_read = 4096 * 4096 * 2 * 1000
print(f'{bytes_read / elapsed / 1e9:.0f} GB/s effective bandwidth')

A useful threshold (observed-pattern, not a benchmarked rate): effective bandwidth under 50% of the published HBM or GDDR figure typically points at host-to-device transfer overhead rather than the memory subsystem itself.

3. Sustained performance under load

Burst performance tells you what the GPU can do in the first 60 seconds. Sustained performance tells you what it can do during the next 8 hours of production traffic. The two diverge for thermal, power-budget, and driver-management reasons that no spec sheet exposes.

Run inference for 15 minutes continuously and compare throughput at minute 1, 5, 10, and 15. A sustained-to-peak ratio below 0.85 is the threshold at which thermal throttling becomes the dominant bottleneck — at that point cooling, not the GPU, is the constraint.

How should I read the numbers?

Interpret your readings by placing them on a scale anchored to validated reference points. The table below is the reference card we use internally when interpreting a fresh run:

Metric	Good	Acceptable	Investigate
FP16 TFLOPS vs spec-sheet peak	> 80%	60–80%	< 60%
Effective memory bandwidth vs peak	> 70%	50–70%	< 50%
Sustained / burst throughput ratio	> 0.90	0.85–0.90	< 0.85

Two GPUs with nearly identical spec sheets routinely land in different columns of this table on the same model, for reasons that are not visible from the spec sheet — driver version, cooling solution, PCIe topology, power limit, the framework’s kernel choices. That is the empirical version of the broader argument that GPU performance for AI is an execution property, not a static one.

What does a complete benchmark protocol include?

Two dozen more variables influence whether your hardware will meet production requirements. A complete protocol has four phases — hardware validation, software stack verification, burst performance, sustained performance — and skipping any of them produces incomplete data.

Hardware validation. Confirm that the GPU reports the expected model, memory capacity, and driver version via nvidia-smi. Check that PCIe link speed matches the card’s capability — Gen4 x16 for current GPUs; a Gen3 link halves host-to-device transfer bandwidth and silently caps every benchmark downstream. For multi-GPU systems, verify NVLink is active with nvidia-smi nvlink --status. We have seen entire procurement cycles invalidated because a chassis seated the card in a Gen3 slot.

Software stack verification. Run a known-good model — ResNet-50 or BERT-base — and compare throughput against published baselines for the same hardware. If throughput is more than 10% below baseline (as a diagnostic rule of thumb), the stack is misconfigured. Common causes are well-known: wrong CUDA version, missing cuDNN, PyTorch built without GPU support, or aggressive power management capping the GPU clock. None of these will be obvious from the GPU’s own diagnostics.

Burst performance. Run the target workload for two minutes and record peak throughput. This is the ceiling the hardware can reach before thermals take effect. For desktop and workstation GPUs, burst performance usually matches the boost-clock specification — that is what spec sheets are actually describing.

Sustained performance. Run the same workload for 30 minutes and record throughput every 60 seconds. The steady-state value, typically reached after 5 to 10 minutes, is what determines production capacity. We report the ratio of sustained to burst — our “sustain ratio” — as the primary number. Values below 0.8 indicate insufficient cooling for the workload’s power consumption, and no amount of kernel tuning will recover what thermals take away. This is the practical face of the gap between peak and steady-state behaviour.

Documenting results so they remain useful

Keep a dated log of every benchmark run with full system specifications, or accept that results expire immediately. Documented results become reusable engineering data. The template we use captures four blocks: hardware specification (GPU model, VRAM, CPU, RAM, storage type), software stack (OS, driver, CUDA version, framework version), test configuration (model, batch size, sequence length, precision), and the results themselves (burst throughput, sustained throughput, P50/P95/P99 latency, power consumption).

Store the results in a version-controlled file — CSV is enough — alongside the benchmark scripts that produced them. This builds a searchable history of hardware performance across configurations and time. When evaluating new hardware, the honest comparison anchor is the team’s own record on the same workload, not a vendor number measured under undisclosed conditions.

The overhead is roughly five minutes per run. That is a negligible cost to convert an ephemeral test into durable engineering knowledge. When the next “how does the 4090 compare to an A100 for our workload?” question lands, the answer comes from data rather than memory.### Record the release name alongside every number

One field in that record does more work than the rest: the name of the benchmark release that produced the result. A run made under the 26Q3 release of LynxBenchAI is read only against other 26Q3 results; when 27Q1 lands, the catalogue and the harness move with it, and comparing across release names silently compares two different measurements. If you would rather not maintain the harness yourself, pip install lynxbench-ai (Python 3.11 or later, Linux or Windows via WSL2) runs a fixed model catalogue on the machine in front of you and reports a 26Q3 result in 15 to 30 minutes — the free Personal Edition, which is what has shipped. That is the point at which the argument about your box stops being rhetorical: the machine reports what it achieved, and the release name tells you what the number may be compared with.

Why benchmarking is a protocol, not an event

A 5% quarterly decline in sustained throughput — invisible against daily monitoring noise — compounds to a 19% annual reduction in effective capacity (illustrative compounding example).

This is the design intent behind LynxBenchAI: treat the benchmark as a continuously executed protocol re-run across driver, framework, and OS updates against a versioned baseline, because hardware-as-measured drifts with software and a one-time baseline becomes stale inside a single release cycle. The question to put to any AI-PC benchmark protocol: does the methodology specify how often and against what baseline the protocol is re-run as the model, driver, and operating point drift — or is the result being treated as a static property of the box?

Frequently Asked Questions

Which precision should I benchmark a PC for AI in — FP32, FP16, or INT8?

Match the precision to your actual workload. Use FP32 for training without mixed precision, FP16 or BF16 for most modern training, and INT8 for production inference. Benchmarking in a precision you will never deploy produces a number that flatters or penalises the hardware for the wrong reasons.

What sustain-ratio threshold tells me cooling is the bottleneck rather than the GPU?

A sustained-to-burst throughput ratio below 0.85 is where thermal throttling becomes the dominant constraint, and a ratio below 0.8 indicates outright insufficient cooling for the workload’s power draw. Below those thresholds, no amount of kernel tuning recovers what thermals take away — the fix is cooling, not software.

How often should I re-run an AI-PC benchmark protocol?

Treat benchmarking as a recurring protocol rather than a one-time event. We re-benchmark production hardware quarterly against the original baseline, because a 5% quarterly decline in sustained throughput — invisible against daily monitoring noise — compounds to roughly a 19% annual loss of effective capacity.

Can I compare a benchmark result from one release of the harness against a result from another?

No. Every result carries the name of the release that produced it — currently 26Q3, with 27Q1 next — and is only meaningful against results produced under the same release name. The model catalogue and the harness itself change between releases, so a cross-release comparison compares two different measurements rather than two machines.

When execution is not feasible, what can a specification-based estimate legitimately be used for?

Running the real workload is the reference standard; a spec-based estimate is the fallback when that is impossible — during procurement of hardware you cannot yet touch, for example. It legitimately supports a rough ceiling and a shortlist to test later. It does not support a claim about achieved throughput, a sustain ratio, or an ordering between two similar cards on your model.

Running your first local AI performance test

Grab a reference model like Llama 2 7B or Stable Diffusion 1.5 and profile both initial load time and steady-state token generation. So the question to carry forward is this: do you know the executor, the rules, and the release behind the number in front of you — and if not, what would it take to find out?