The benchmark-to-AI-performance prediction problem GPU benchmarks are designed to be reproducible and comparable across hardware. To achieve that, they run standardised workloads — fixed model architectures, fixed batch sizes, fixed precision — under controlled conditions. Real AI workloads are none of these things. They arrive at variable rates, mix request shapes, exercise different precision regimes, and run for hours rather than seconds. The gap between a GPU’s benchmark score and its observed performance on a specific AI workload can comfortably run 2–5× in either direction; this is an observed pattern across the procurement reviews we get pulled into, not a benchmarked rate. A more useful framing: a benchmark is a measurement under a chosen workload shape, and workload shape dominates observed performance. When the shape on the benchmark report and the shape in production diverge — and they almost always do — the published number stops being a useful proxy. That divergence is structural, not a sign that the benchmark was dishonest. Why synthetic shapes diverge from production Four divergences recur often enough to be worth naming explicitly. Each of them shows up in roughly the same way regardless of vendor, and each of them is invisible to the headline number on a benchmark report. 1. Burst vs sustained performance Most GPU benchmarks run for 30–120 seconds, comfortably inside the thermal reservoir of most GPUs. AI training runs for hours; serving runs continuously. GPUs boost clock speed when cool, then throttle as the package heats up and as power-delivery limits engage. A GPU that scores 100% in a 60-second benchmark commonly sustains roughly 70–90% of that throughput over a 6-hour run — an observed range across the hardware reviews we participate in, not a vendor-specified throttle curve. 2. Memory-bandwidth vs compute ratio Standard benchmarks often run compute-bound workloads — dense matrix operations at the sweet-spot sizes for the silicon. Many AI inference workloads, especially autoregressive LLM decoding, are memory-bandwidth-bound: large model weights streamed for every token, with kernel compute time small relative to HBM transfer time. A GPU with high advertised FLOPS but limited HBM bandwidth will underperform its benchmark ranking on memory-bound workloads, sometimes by a wide margin. Per NVIDIA’s published specifications, the FLOPS-to-bandwidth ratio varies considerably across the product line; the right ratio depends on the workload shape, not the headline number. 3. Fixed vs variable batch size Benchmarks tend to run at fixed, often large batch sizes that maximise GPU utilisation. Production inference may run at batch=1 for latency-sensitive endpoints, or at a continuously-batched, request-mixing shape on a serving framework like vLLM or TensorRT-LLM. GPU utilisation at small effective batch sizes is lower, and different architectures behave differently in that regime — kernel-launch overhead, scheduler behaviour, and Tensor Core occupancy all shift. 4. Framework overhead not captured GPU benchmarks run hand-optimised kernels directly. Production AI workloads run through framework dispatch layers — PyTorch eager, torch.compile, TensorFlow, ONNX Runtime — which add Python overhead, graph-capture cost, and dispatch latency that benchmarks elide. Two GPUs with identical raw benchmark scores can show meaningfully different effective throughputs once a framework, with its CUDA-kernel selection heuristics and its cuDNN version, sits between the user and the silicon. Benchmark categories and AI prediction accuracy The shorthand below is the rubric we use when a buyer hands us a stack of benchmark reports and asks which numbers to trust. It is a planning heuristic, not a benchmarked accuracy score — the right column is “how much does this number predict your AI throughput”, not “how well did the vendor execute the test”. Benchmark type AI prediction accuracy Why 3DMark, FurMark (graphics) Very low Rendering access patterns differ structurally from AI compute Geekbench GPU Compute Low Short-burst, generic compute, no framework MLPerf Inference (closed) Medium Real models and frameworks, but fixed shapes, heavy optimisation Framework microbenchmarks (pytorch-benchmark, NCCL tests) Medium-high Captures dispatch and collective overhead Your own workload, on the candidate hardware High The only reliable method The pattern is monotonic: the closer the benchmark sits to your actual workload — same model family, same framework, same batch and concurrency profile, same precision — the better its predictive value. MLPerf is more useful than Geekbench for AI buyers, but it is still not your workload. Why isn’t realism binary? Realism is not a property a benchmark either has or lacks; it is a vector of dimensions a benchmark can approach unevenly. A benchmark can be realistic on model architecture but synthetic on request arrival pattern; realistic on batch size but synthetic on input length distribution; realistic on framework but run on a thermally-idealised chassis. Treating realism as binary — “synthetic” vs “real-world” — collapses a multi-axis decision into a label and hides exactly the dimensions that matter for procurement. A more useful question for any benchmark report: which dimensions of the buyer’s workload has this methodology explicitly bounded, and which has it left to the buyer’s interpretation? When LynxBench AI issues a result, the methodology disclosure names the axes it controlled (model, precision, batch, duration, concurrency, framework version) and the axes it did not — because results published without that disclosure are upper bounds rather than deployment-grade evidence. The structural reasons benchmark numbers drift away from production sit underneath all four divergences above; we treat them as a single failure class in why benchmarks fail to match real AI workloads. What concurrency does to the picture A single-stream benchmark measures the latency of one request against an otherwise-idle GPU. Production serving rarely looks like that. Requests arrive on a distribution — bursty, often Poisson-ish at low scale, more uniform at high scale — and the serving stack batches them dynamically. The measured throughput on a single-stream benchmark and the measured throughput under continuous batching at, say, 32 in-flight requests, can differ by an order of magnitude on the same hardware; queuing, KV-cache pressure, and prefill-vs-decode interference all enter. Concurrency also exposes the tail. P50 latency under load may look fine while P99 collapses, because a handful of long-context requests stall the decode loop for everyone in the batch. A benchmark that reports a single throughput number gives no signal on that, even though the tail is what wakes up the on-call engineer. Recording at least P50, P95, and P99 alongside mean throughput is non-negotiable for serving workloads — it’s a baseline practice in our LLM-serving engagements, not a vendor-specified KPI. What to run for AI capacity planning When the goal is to size hardware for a real workload, the procedure is short and unglamorous. Each step is there because skipping it has burned us, or someone we work with, at least once. Run the actual model at the target context length and batch size, through the target framework and runtime (PyTorch + torch.compile, vLLM, TensorRT-LLM, ONNX Runtime — whichever will run in production). Measure for at least 10 minutes per configuration to capture steady-state, not burst. Treat anything shorter as a smoke test. Record VRAM utilisation, power draw, and SM occupancy alongside throughput. A GPU at 95% reported utilisation can still be memory-bandwidth-bound and leaving compute on the table. Test at the realistic concurrency profile — for serving, a request generator that matches the production arrival distribution; for training, the actual data-parallel or pipeline-parallel topology you intend to deploy. These four steps don’t produce a single comparable number across vendors. They produce something more useful: a workload-specific measurement on the candidate hardware, which is the only kind of measurement that survives contact with production. Building workload-specific benchmarks The most reliable benchmark for any AI workload is the workload itself, run under controlled conditions. We build workload-specific benchmarks by isolating the inference or training loop from the full application, fixing the input data to a representative sample, and instrumenting the loop with per-iteration timing. The scripts are typically 50–100 lines of Python — small enough that anyone on the team can read them, run them, and modify the request distribution. This approach removes the external variables (network latency, storage I/O variability, upstream queuing) that confound production measurements, while preserving the workload characteristics (model architecture, batch size, precision, framework version) that determine GPU behaviour. The standardised output — mean throughput, P50/P95/P99 latency, GPU utilisation, power draw — is comparable across hardware configurations, driver versions, and framework versions without needing interpretation against a proxy score. These same scripts double as regression tests for the AI software stack. After a PyTorch, CUDA, or cuDNN upgrade, re-running them confirms the upgrade did not silently degrade throughput. We wire them into CI as post-deployment validation: an alert fires if any tracked metric drifts more than a configured threshold from the stored baseline, which catches software regressions before they affect production. It is the same artefact, used twice — once for hardware evaluation, once for stack validation. When is a synthetic benchmark still useful? Synthetic benchmarks remain genuinely useful in a few well-bounded cases. They are the right tool for vendor-vs-vendor sanity checks at the silicon level — confirming that a card’s HBM bandwidth, FP16 throughput, or NVLink fabric matches the spec sheet, independent of any particular workload. They are useful for regression-testing the driver and CUDA stack itself, because the stability of the synthetic number across driver versions tells you whether the lower layers shifted. And they are useful as a quick first filter on a long candidate list — eliminating clearly underpowered options before you invest in workload-specific testing on the survivors. What synthetic benchmarks are not useful for is capacity planning for a specific deployment. The further the synthetic shape sits from the deployment shape — and on the four axes above, the distance is usually large — the less the headline number tells you about throughput, latency, or cost-per-request in production. The procurement implication is short: when a GPU benchmark informs a real spending decision, ask which workload dimensions the methodology explicitly bounded for your case, and which it left to your interpretation. The dimensions left open are where the 2–5× surprises live. Which workload dimensions did the GPU benchmark in front of you bound for your sustained-load operating point, and which did it leave for your interpretation?
