To read an MLPerf Inference result honestly, treat it as a comparability signal produced under fixed conditions, not as a forecast of how the hardware will behave on your model. Each MLPerf number is tied to a scenario, a reference model, a quality target and a load pattern, and any of those can differ from your deployment. Used well, a result shows what the hardware reached under MLPerf’s conditions with the submitter’s own tuning. Used naively, it gets mistaken for a promise about production. MLPerf is one type of evidence among several in a buyer’s file, and the Ollama benchmarks and procurement evidence overview sets out where the other types fit.
The rest of this article covers what each scenario encodes, what to check in a submission before you trust its headline, how to turn the figures into latency and cost per decision under your own traffic, and how to put options side by side so a committee is comparing like with like.
How Do You Read MLPerf and Hardware Inference Benchmarks Honestly for a Real Deployment Decision?
An honest reading asks three things of a result. First, under which scenario, division, version and availability category was it produced? Second, how close are those conditions to your model, input distribution, accuracy tolerance and concurrency? Third, what would you have to measure yourself to close the gap?
The benchmark is designed for comparability across systems. MLPerf prescribes rules so that systems with very different architectures can be compared, and it fixes the dataset, the load generator (LoadGen) and the accuracy script for every submission while leaving the implementation to the submitter. That split is the reason the numbers are comparable with each other. It also means each result reflects the submitter’s own implementation, optimised on its preferred toolchain and hardware.
Submissions pass audit tests and a result-review process. The original MLPerf Inference paper (Reddi et al., 2019) reports that the v0.5 round received over 600 submissions from 14 organisations, of which audit tests cleared 595 as valid. The same paper reports that review found about 40 issues in roughly 180 closed-division results, from missed quality targets and latency bounds to misread rules. Audited, then, means checked against MLPerf’s own conditions. It does not mean checked against yours. Our article on reading NVIDIA MLPerf results as procurement evidence starts from a single vendor press slide.
What Do the Four MLPerf Scenarios Measure, and Which One Looks Like Your Traffic?
Each MLPerf Inference scenario pairs a query-arrival pattern with one metric. The four were chosen by surveying MLPerf’s membership of customers and vendors. They behave very differently, and a strong result in one scenario does not carry over to another.
| Scenario | How LoadGen issues queries | Reported metric | Production shape it resembles |
|---|---|---|---|
| Single-stream | One query, waits for completion, then the next | Tail latency of the query stream (90th percentile in the original paper) | One user or one sensor feeding a device, e.g. a phone app |
| Multi-stream | Several streams at a fixed interval | Number of streams supported within the QoS bound | Multi-camera perception, as on vehicles or robots |
| Server | Poisson arrivals | Queries per second achievable within the QoS bound | An online API taking independent requests |
| Offline | All samples available at once | Throughput in samples per second | Batch jobs: nightly scoring, document backfills |
Server and Offline are the scenarios most datacentre benchmarks require, and the gap between them matters. In the Reddi et al. results, every system delivered less throughput under Server than under Offline because the latency constraint makes large batches harder to form. The loss was as small as 3% for some systems and around 50% for others. An Offline figure therefore tells you almost nothing about a latency-bound service until you have seen the same system’s Server result.
The tail percentile is also set per task and scenario. In the original paper a 99th-percentile guarantee for vision tasks in Server and Multi-stream required about 270K queries, and machine translation used a 97th percentile. For LLM inference benchmarks the current rules collect time to first token (TTFT) and time per output token (TPOT), with bounds that differ by category. One LLM benchmark, for example, sets 2000 ms TTFT and 200 ms TPOT for its conversational category but 450 ms and 40 ms for its interactive category. Find out which bound a quoted number passed before you compare it with your own SLO.
Two scenario details catch readers out. In the Server scenario, the final reported metric is scheduled samples per second, not completed samples per second. And edge submitters may infer a Multi-stream result from a Single-stream run and an Offline result from either, so check whether an edge Offline figure was measured or inferred.
Which Parts of a Submitted MLPerf Config Must You Check Before Trusting a Headline Number?
Check the conditions that change what the number means before you look at the number itself. This checklist is built to be used on its own:
| Check | What to look for | Why it changes the reading |
|---|---|---|
| Division | Closed or Open | Closed requires a model equivalent to the reference and allows quantisation calibration but no retraining. Open allows a changed model and quality target, and Open results are generally not directly comparable with each other or with Closed |
| Accuracy target | The quality target the run had to meet | Almost all implementations must reach within 1% of the FP32 reference model’s accuracy, and each benchmark is defined by a dataset and a quality target |
| Precision | The numerical format the submitter quantised to | Closed submitters may quantise to any format that reaches the quality target, using the supplied calibration data |
| Scenario and sub-scenario | Server, Offline, Interactive and so on | Throughput and latency bounds differ by scenario |
| Version | The MLPerf round | Results are compatible only for the same benchmark and scenario on versions MLCommons deems compatible |
| Availability category | Available, Preview or RDI | Available systems can be bought or rented today. Preview systems must be submittable as Available in the next round. RDI covers experimental or internal hardware and software |
| Verified status and chip count | Whether the result was verified, and how many accelerators ran it | MLCommons requires comparisons to disclose any difference in version, division, category, verification status, scenario or chip count |
| Batching and parallelism | The batch settings in the submission’s system and implementation description | Larger batches raise throughput and also per-request latency, and the best setting depends on model, hardware and application |
Precision deserves more than a box ticked. Kurtic et al. (2024), working across the Llama-3.1 family, found FP8 (W8A8-FP) effectively lossless and well-tuned INT8 at 1 to 3% accuracy degradation. Within a 1% budget against an FP32 reference, then, the format behind a headline can be the difference between a number you can reproduce at your accuracy tolerance and one you cannot. If your domain cannot absorb the format the submitter chose, discount the result.
Version deserves the same attention. MLCommons said in April 2026 that five of the eleven datacentre tests in MLPerf Inference v6.0 were new or updated.
Where Do MLPerf and Hardware Benchmark Results Belong in a Procurement-Grade Evaluation Pack, and Where Do They Mislead?
In a procurement-grade evaluation pack, MLPerf and other hardware suites belong in the operational-feasibility and cost-per-decision sections. They do not belong in the task-accuracy or failure-mode sections. In the right place they answer questions such as whether a platform can plausibly meet a latency-bounded throughput at a given chip count, and whether its Server-to-Offline drop is small or large. Evidence of that kind, which a non-engineer on a committee can read and sign against, is an engineering output in its own right.
Results mislead in predictable places. MLCommons’ messaging guidelines say that derived metrics such as cost or accuracy comparisons must state their basis and must not be presented as official or verified MLPerf metrics. Any use of results that are not verified must also be disclosed as unverified. So a slide showing “cost per inference from MLPerf” is someone’s calculation built on an MLPerf input. It is not an official MLPerf metric. Power claims follow the same rule: only system power measured with the MLPerf Power methodology is sanctioned for comparisons, not TDP or a PSU rating. The MLPerf Power measurement methodology covers how that figure is produced.
Interpolation is the other trap. MLCommons’ March 2026 description of MLPerf Endpoints explains that it plots only verified operating points as step functions, because interpolating between measured points can suggest performance that was never achieved and can hide memory overflows or P99 spikes. A vendor curve drawn through two MLPerf points has the same weakness.
Why Can’t MLPerf or a Vendor Hardware Benchmark Answer Task Accuracy or Failure Modes?
MLPerf checks that a reference model on a reference dataset stays within a fixed quality margin. It runs the reference model on MLPerf’s fixed dataset, not your model on your inputs. Reddi et al. make the point themselves: trading 1% accuracy for 50% lower TCO is sensible when identifying cat photos and risky when detecting pedestrians, because the trade-offs are specific to each application. A 2025 benchmarking-methodology paper likewise treats accuracy as outside the scope of performance benchmarking and potentially independent of the hardware option.
Task accuracy and failure behaviour have to be evaluated on your own data. Our guide to running a task-specific LLM evaluation that survives procurement review covers that half of the evidence, and the reasoning-model case is treated separately in what DeepSeek-R1 benchmarks and reasoning evals test.
How Do You Turn MLPerf Queries per Second and Percentile Latency into p95 Latency and Cost per Decision?
An MLPerf figure is not a substitute for measuring p95 latency and cost per decision on your own workload. You measure them on the candidate stack under your own offered load and SLO, and use the MLPerf result to decide which candidates deserve that measurement. MLPerf reports latency at its own percentile and bound, at its own arrival pattern, and its Server QPS is the rate at which that bound held. Your latency and utilisation depend on your offered rate, SLO, batch limits and deployment configuration.
A June 2026 arXiv study found that, on identical H100 hardware with model, precision and GPU allocation held fixed, effective cost per token varied 17.5 to 36.3 times purely as a function of offered request rate, across 42 vLLM benchmark runs. The same authors argue that GPU utilisation is an output of arrival rate, model, quantisation and SLO, not an input to guess, and that a cost number quoted without a load attached is meaningless. Signing a latency SLA fixes the operating point on that cost curve, so the SLA is itself a price.
The quantity to carry forward is goodput: work completed while meeting the latency requirement, as distinct from raw throughput. Throughput normally saturates near the engine’s maximum batch size while latency keeps rising, so the peak tokens-per-second figure is often a configuration you could not run for an interactive service.
The Load-Test-to-Cost Procedure
- Deploy the configuration you would run: model, quantisation and hardware matched to your constraints. Published benchmark corpora are a starting reference, not a substitute for measurement on the hardware.
- Load-test with your real prompt-length and output-length distribution, your expected arrival pattern and your declared TTFT, TPOT and end-to-end SLO. Sweep concurrency or request rate, record p95 alongside the percentiles your SLO names, and note the highest arrival rate at which the SLO holds. That rate is your goodput.
- Divide hourly infrastructure cost by useful output per hour at that goodput. For tokens, cost per million is C / (T × 3,600) × 1,000,000, where C is cost per hour and T is usable tokens per second. For non-generative workloads, cost per successful request is often more useful.
Running that sweep as a repeatable job, so the numbers are regenerated when the model or serving engine changes, is ordinary MLOps engineering work rather than a one-off spreadsheet.
Worked Example: Throughput Versus Goodput
Hivenet’s explainer on inference metrics gives a comparison: System A produces 10,000 output tokens per second with 60% of requests meeting the SLO, and System B produces 8,500 with 98% meeting it. Assume, for simplicity, that SLO compliance applies evenly across tokens. A’s useful output is then 10,000 × 0.60 = 6,000 tokens per second, and B’s is 8,500 × 0.98 = 8,330. At the same hourly cost C, A costs C / 21.6 per million useful tokens (6,000 × 3,600 = 21.6M per hour) and B costs C / 29.99. B is about 28% cheaper per useful token despite the lower headline throughput. In this example, the system that wins on raw throughput costs more per useful token.
How Should Hardware and Serving Options Be Tabulated so a Committee Can Compare Like with Like?
Tabulate options only on rows where every condition matches, and show each condition explicitly so a mismatch is visible rather than buried. Hivenet’s guidance says results compare only when model, precision, hardware, request shape, load, serving configuration and metric definitions are sufficiently alike. Its minimum disclosure list includes model and revision, precision, GPU count, serving engine and version, input and output lengths, concurrency or request rate, duration, latency metrics, throughput and error rate.
| Row | Option A | Option B | Must match? |
|---|---|---|---|
| Model and revision; precision | Yes | ||
| Accelerator, count, interconnect | Disclosed | ||
| Serving engine and version; max batch size | Disclosed | ||
| Input / output length distribution | Yes | ||
| Arrival pattern and rate (or concurrency) | Yes | ||
| SLO: TTFT, TPOT, end-to-end at stated percentiles | Yes | ||
| Which throughput figure: per-request or system TPS, raw or goodput | Yes | ||
| p95 latency at goodput | Result | ||
| Error rate (timeouts, OOMs, preemptions) | Result | ||
| Cost per million useful tokens or per decision at goodput | Result | ||
| MLPerf reference: version, division, category, scenario | Context only |
The “which throughput” row matters because tokens per second has several definitions. NVIDIA’s GenAI-Perf, for example, defines TPS as total output tokens over the span from the first request to the last response, which is a system figure and not what one user experiences. Error rate belongs in the table because a throughput reported only from successful requests, with failures omitted, misleads. Where two configurations trade cost against latency, Pareto thinking helps: an option that is slower, dearer and lower-throughput under the same workload can be dropped, and only real trade-offs go to the committee.
LynxBenchAI’s argument that methodology is what makes benchmarks comparable is the background to the “must match” column.
The open question is not which accelerator topped a round. It is whether the cost per decision at your SLO justifies a step up in hardware tier at the load you actually carry. The same June 2026 study found that a lower-priced A100 was cheaper per million tokens near idle, while H100 won above a crossover load. Where your load sits on that curve is something only your own measurement can establish.
FAQ
Is MLPerf Open Source?
Yes. MLCommons describes the MLPerf Inference benchmark suite as open source, intended to measure performance in an architecture-neutral, representative and reproducible way.
What Does MLCommons Do for MLPerf?
MLCommons owns the MLPerf mark, and a review committee that includes submitter representatives and working-group chairs oversees review and publication of results.
Can Benchmark Tests Evaluate Both Hardware and Software?
MLPerf Inference was built to measure the inference performance of ML hardware, software and services, and its Closed division is meant to compare hardware platforms or software frameworks like for like.
Is MLPerf Inference the Same as MLPerf Training?
No. MLPerf Training is a complementary suite that measures training rather than inference, and MLPerf Training for sizing AI infrastructure covers what its time-to-train metric tells a buyer.