What does it mean to say two benchmarks are “comparable”?

The word gets used casually. “We compared the A100 and H100 benchmarks.” “This result is comparable to the vendor’s published number.” “We benchmarked both configurations and compared the results.”

In each case, “compared” hides an assumption: that the two measurements are measuring the same thing, under sufficiently similar conditions, with a declared protocol that makes the numbers commensurable. Comparability is not a property of the scores themselves — it is a property of the methodology that produced them. When that methodology is shared, explicit, and well-specified, comparison tells you something real. When it is absent or divergent, comparison is arithmetic performed on numbers of unknown provenance.

Comparability doesn’t come from the scores. It comes from the methodology that produced them.

Methodology defines what is being compared

A benchmark methodology specifies, at minimum: the workload being executed (model architecture, dataset or input distribution, precision format), the measurement protocol (what’s timed, how timing is performed, what’s excluded), the system configuration (software stack versions, driver, framework, optimization flags), and the reporting protocol (what statistical summary is used, what metadata accompanies the result).

When two measurements share these specifications, their scores can be meaningfully compared. When they diverge — one measured peak throughput during warmup while the other measured steady-state after thermal settling; one used TensorRT-optimized inference while the other used vanilla PyTorch — the comparison produces a number that reflects the methodological difference at least as much as the hardware difference.

This is not a theoretical concern. As explored in how benchmarks function as decision infrastructure, the framing embedded in a benchmark’s methodology shapes what the result means. Two results produced by different methodologies are not “two data points about the same thing” — they are data points about two different things that happen to share a unit of measurement.

Cross-vendor comparability is inherently constrained

Comparing hardware across vendors introduces methodological challenges that are difficult to eliminate.

Each vendor optimizes for their own hardware. NVIDIA publishes benchmark results using TensorRT with CUDA-specific optimizations. AMD publishes results using MIGraphX or ROCm-tuned configurations. Intel publishes results using OpenVINO. Each toolchain is optimized for its target hardware, which means each result reflects the hardware and the vendor’s software stack maturity.

A naïve “apples to apples” comparison — running the same model on both vendors’ hardware — is only methodologically clean if the software stack provides equally optimized execution paths on both targets. In practice, one vendor’s stack is typically more mature for a given operation, and the differential reflects software investment, not silicon capability.

Fair cross-vendor comparison requires either: using a vendor-neutral framework with equivalent optimization on both targets (rare in practice), or acknowledging that the comparison reflects the hardware+software system as a whole (which is actually the relevant comparison for deployment decisions, even if it’s less satisfying as a hardware-specific claim).

This connects with how organizations should approach hardware selection — the comparison that matters for a buying decision is the one that reflects what you’ll actually deploy, not the one that isolates a single variable.

Workload choice dominates outcomes

Of all the methodological variables that affect benchmark results, workload selection is typically the most influential.

A benchmark that evaluates training throughput on ResNet-50 at batch size 64 exercises largely compute-bound operations with moderate memory requirements. The same hardware evaluated on LLM inference with long-context inputs at small batch sizes exercises memory-bandwidth-bound operations with minimal compute utilization. The two benchmarks may produce results that rank the same set of hardware in different orders — not because the hardware changed, but because the workload shifted the performance-limiting resource.

We see this frequently: Hardware A wins on Workload X; Hardware B wins on Workload Y. Neither result is wrong. They’re measuring different things, and the useful question is not “which hardware is faster?” but “which workload is more representative of my deployment?”

A methodology that declares its workload — precisely, including model, input shape, batch size, precision, and optimization level — allows consumers to judge representativeness. A methodology that omits these details, or buries them in footnotes, produces results that look comparable but aren’t.

The methodology as a contract

A well-specified methodology functions as a contract between the benchmark publisher and the benchmark consumer. The publisher commits to measuring a specific thing under specific conditions. The consumer commits to interpreting the result within those conditions.

When both sides honor the contract, comparison works. You can meaningfully say “System A achieved X under this methodology and System B achieved Y under the same methodology, and the difference reflects something real about the systems.”

When the contract is vague or absent — “we ran some tests and these are the numbers” — comparison collapses. The consumer has no basis for knowing whether two measurements are commensurable, and no basis for attributing differences to hardware, software, configuration, or methodology.

This sounds like bureaucracy, and it’s easy to dismiss as unnecessary overhead. But methodological rigor is what separates measurement from marketing. A benchmark without a declared methodology is an anecdote with decimal places.

At minimum, a methodology contract should disclose:

• Workload specification. Model architecture, parameter count, input shape/distribution, batch size, sequence length.
• Precision format. FP32, BF16, FP8, mixed — and whether quantization was applied, with what calibration.
• Software stack versions. Framework, compiler/optimization passes, kernel libraries (e.g., cuDNN version, FlashAttention version), driver version.
• Measurement protocol. What is timed (forward pass only, end-to-end pipeline, including host preprocessing or not), how timing is performed (wall clock, CUDA events), warmup policy, number of iterations.
• Statistical summary. Mean, median, percentiles, number of runs, whether outliers were excluded and how.
• Environmental conditions. Thermal state at measurement start (cold or pre-warmed), power limit setting, cooling configuration if it departs from reference.
• Exclusions. What the benchmark does not measure and why — model loading, network latency, queuing, host-side computation.

When these fields are present, two results from different sources can be meaningfully compared. When they’re absent, the comparison is arithmetic performed on numbers of unknown provenance.

Methodology is not purely technical

Designing a benchmark methodology involves technical choices (what to measure, how to measure it) and non-technical choices (what to prioritize, what to exclude, what audience the benchmark serves).

A methodology designed for vendor comparison will include cross-platform reproducibility constraints. A methodology designed for capacity planning will emphasize steady-state measurement and realistic workload patterns. A methodology designed for procurement decision support will include cost-relevant metrics alongside performance ones.

These design choices reflect the methodology’s purpose, and different purposes produce different methodologies — each valid for their intended use, potentially misleading when applied outside it. The claim that “a benchmark is a benchmark” obscures this design dimension.

The practical discipline is: when consuming a benchmark result, read the methodology before reading the score. If the methodology matches your deployment scenario and evaluation purpose, the result is useful evidence. If it doesn’t — if it measures a different workload, different conditions, or different metrics than what you need — the result may still be interesting, but it isn’t the evidence you need.

As explored in the context of how cost, efficiency, and value are distinct metrics, the choice of what to measure encodes assumptions about what matters. Methodology is where those assumptions are made explicit — or left implicit and dangerous.