MLPerf Client: What the Benchmark Measures and How to Read It

MLPerf Client benchmark results report TTFT, tokens per second and agentic task time.

MLPerf Client: What the Benchmark Measures and How to Read It
Written by TechnoLynx Published on 11 Jul 2026

MLPerf Client is the MLCommons benchmark for running large language models, generative image models and agentic AI workloads locally on laptops, desktops and workstations. For LLM tests it reports two numbers: time to first token (TTFT) and tokens per second (TPS) for the rest of the response. Version 2.0 adds an end-to-end duration for agentic tasks. MLPerf Client benchmark results are best read as a controlled comparison of vendor inference paths running int4-quantised models on fixed prompts. They are not a verdict on whether a given machine will run your workload.

A single headline tokens-per-second figure hides most of what decides whether a project works on a device. It doesn’t tell you which model and quantisation were used, how long the prompt was, whether your user waits on the first token or on the stream that follows, or whether your model fits in memory at all.

How Does MLPerf Client Actually Work, and What Does That Mean in Practice?

The benchmark ships in three tiers. Base components are required, run on most client systems and are the only ones that can produce an official “tested by MLCommons” score. Extended components (long prompts, larger models) may not fit every machine. Experimental components ship with the benchmark but haven’t been accepted into the base or extended set yet. Results from extended or experimental components are marked as such in the output.

The language models are quantised to 4-bit integer weights (int4) so they fit on a wide range of client devices. To check that quantisation and other optimisations haven’t degraded quality too far, the working group imposes accuracy requirements based on MMLU and IFEval. Vendors have room to optimise. When the v0.5 release came out, Signal65’s analysis noted that vendors were encouraged to modify the model for their hardware, provided the result met the MMLU accuracy threshold.

A score is valid only when both the executable and the JSON configuration file are versions the MLPerf Client working group has tested and approved. That approval involves peer review among the participating organisations and MMLU-based accuracy testing. You can change configurations to experiment, but scores from modified executables, libraries or config files are not comparable to the approved ones. In the default configs each test runs four times: one warm-up run and three performance runs, and the application reports the average of the three.

The benchmark’s prompt categories have these approximate token lengths:

Category Approx. input tokens Approx. output tokens
Content generation 128 256
Creative writing 512 512
Structured text 830 256
Code analysis 1,856 128
Summarisation, intermediate (~4K) 3,906 128
Summarisation, substantial (~8K, extended) 7,631 128

In v2.0 the roughly 4K-token summarisation task moved into the base set, and Llama 3.1 8B and Phi 4 Mini Instruct became the mandatory base models.

In practice, a score tells you that a specific model at int4, on a specific vendor path, handled that prompt mix at that speed. Every one of those conditions has to match your project before the number transfers.

What Does MLPerf Client Actually Measure, and What Does It Not Tell You About Your Workload?

MLCommons defines TTFT as the wait in seconds before the system produces the first token of its response to a prompt. It is reported separately, following industry practice; that first wait is usually the longest in the response, and lower is better. TPS is the average rate at which the system produces the rest of the tokens, excluding the first, and higher is better. In a chat interface, TTFT is how long you wait for the first character and TPS is how fast the rest of the answer appears.

Because TPS excludes the first token, the two numbers combine directly into a response time: response time ≈ TTFT + (output tokens − 1) ÷ TPS.

Independent reviews of v2.0 also point out that the published LLM figures are geometric means across the prompt categories, not per-category values. So a headline TTFT blends a 128-token prompt with much longer ones, and you can’t tell from the headline which one your workload resembles.

Worked Example: Turning the Two Numbers into a Wait

Assumptions: we take a token rate StorageReview published for Llama 3.1 8B on a Lenovo ThinkPad P1 Gen 9 in MLPerf Client v2.0, which was 64.8 tokens per second. We apply it to the output lengths of two categories. Since that figure is a geometric mean across categories, the example only shows how the calculation works.

  • Content generation, 256 output tokens: (256 − 1) / 64.8 ≈ 3.9 seconds of streaming after the first token appears.
  • Code analysis, 128 output tokens: (128 − 1) / 64.8 ≈ 2.0 seconds of streaming, but this follows a prompt of about 1,856 input tokens, so a larger share of the total wait sits in TTFT.

The same device, at the same token rate, spends its time in different places depending on the shape of the prompt. That shape is the first thing to match against your own workload.

What the Scores Do Not Tell You

  • Your model and precision. The scores cover the benchmark’s model list at int4, with vendor-specific optimisations allowed inside the accuracy gate.
  • Your context length. The largest standard LLM prompt is about 7,631 input tokens. StorageReview also observed that MLPerf’s 8B models run on smaller GPUs because MLPerf sizes its context to the card.
  • Whether your model fits. In StorageReview’s laptop comparison, one GPU generated tokens about 2.3 times faster than another on every model that fit in its 24GB, but only the machine with 64GB of unified memory available to the GPU could run the larger models at a usable speed.
  • Concurrency. On a serving system, total throughput rises with concurrent requests while per-user TPS falls as latency grows.

How Do Time-to-First-Token and Sustained Tokens/Second Differ, and Why Does the Distinction Matter for Agentic Versus Generative Use Cases?

TTFT and TPS map onto the two phases of LLM inference. Prefill processes the whole input at once and is compute-bound. Decode generates tokens one at a time, reads the key-value (KV) cache repeatedly and is memory-bound. A phase-aware study of GPUs and emerging accelerators found that no single platform dominated both phases. It also found that aggregate end-to-end metrics can hide phase-specific trade-offs, so prefill and decode have to be evaluated separately for a meaningful hardware comparison. That study ran Llama2-7B in FP16, but its authors say the qualitative prefill/decode trade-offs persist for transformer-based models.

For long-form generation, such as drafting or summarising into a long output, TPS is the number to weigh. For interactive short calls, the user mostly feels TTFT.

In agentic loops, the answer depends on the runtime. For background on what an agentic workload is, see that piece; here the question is narrower. One characterisation of ReAct-style agents found that with effective context caching, 84.6–99.5% of input tokens were reused across turns. In that setting decode took 91.0–98.6% of LLM time, and each turn prefilled only the newly appended text. Those authors also note that if the cached state is evicted, a decode-dominated workload turns into expensive recomputation. A second study of production agent sessions shows the other side. At the final step of one session, the model produced 151 tokens for its next action but first had to process 166,721 context tokens.

So whether an agent loop is limited by TTFT or by TPS on a laptop depends on whether the inference runtime keeps the KV cache resident between turns, and whether the device has the memory to hold it. That depends on the system design, and the benchmark can’t settle it for you.

MLPerf Client v2.0 measures agentic work directly. Its Software Engineering and Data Analyst agent scenarios report total task time across all LLM inference turns and tool executions, with a breakdown of inference time and tool time.

Why Can a Device That Leads on Throughput Still Be the Wrong Choice for an Agentic Orchestration Loop?

A throughput leader can lose an agentic loop because the loop pays costs that a sustained-TPS figure doesn’t measure. Each turn pays a prefill for its newly appended context. Context grows and KV-cache memory piles up. Tools take time of their own.

Per-turn waits add up. A loop of many short calls pays the TTFT (or the prefill of newly appended context) once per turn, so when the cache is lost or the appended context is large, a device with a strong decode rate and a weak first-token time loses ground on every step. Turn counts vary widely. In one characterisation most configurations averaged 12–62 turns per task, with long tails reaching hundreds.

Memory can also end the run outright. In StorageReview’s v2.0 testing, one laptop did not finish either agentic scenario because DirectML ran out of memory on the agentic prompts, even with 32GB reserved for the GPU, and the result was marked DNF. A strong TPS figure on short prompts says nothing about that case.

The bottleneck may not be the model at all. AgentSysBench found that tool execution took up to 84% of total time in some request types, while LLM inference took up to 90% in others. It also found that the bottleneck shifts whenever the workload, model or serving configuration changes. StorageReview also noted that agent wall time depends on how many tool calls the model makes, which is why the generation rate is the cleaner hardware number.

This cuts both ways. In MLPerf Client’s own agentic scenarios on one set of laptops, task times followed the token rates, and tool phases differed by only about two seconds across the four machines. In those runs, the machine with the highest token rate also finished both agent tasks first. The risk comes when your loop makes more calls, carries longer context, or runs heavier tools than the benchmark’s agents. For the parts of an agent loop that produce those differences, see the components of an LLM agent.

How Do You Run MLPerf Client, and Which CPU, GPU and NPU Paths Does It Cover?

Since v1.0, MLPerf Client has offered a command-line interface and a graphical one. The GUI shows compute and memory use in real time, keeps a results history, builds comparison tables across runs and exports CSV files for offline analysis. The CLI suits automation and regression testing. To run a test, you call the benchmark executable with the -c flag and a JSON config file. The config specifies the acceleration path and options recommended by that hardware vendor.

MLCommons recommends using each vendor’s own config for that vendor’s device. For example, use the NVIDIA ORT GenAI config for a discrete GeForce GPU and the AMD ORT GenAI GPU config for an integrated AMD GPU in the same machine. By default the GPU-focused configs don’t name a device, so the benchmark runs on every available GPU in turn and reports each separately. Device enumeration and selection for multi-GPU systems arrived in v0.6.

Hardware coverage has changed with nearly every version:

Version Workloads Hardware and software paths
v0.5 Early access; Llama 2 7B with four prompt scenarios GPUs via ONNX Runtime GenAI with DirectML, and Intel OpenVINO
v0.6 Same workloads as v0.5 Adds Intel NPUs; GPU acceleration on AMD, Intel and NVIDIA; MLCommons calls it the first open benchmark to span GPU and NPU on consumer platforms
v1.0 Llama 2 7B Chat, Llama 3.1 8B Instruct, Phi 3.5 Mini; Phi 4 Reasoning 14B experimental; code analysis and experimental 4K/8K summarisation AMD NPU and GPU via ONNX Runtime and Ryzen AI SDK; Intel NPU and GPU via OpenVINO; AMD, Intel and NVIDIA GPUs via ONNX Runtime GenAI with DirectML; Qualcomm NPU and CPU hybrid via Genie and QAIRT; Apple Mac GPUs via MLX; experimental Windows ML (OpenVINO EP), llama.cpp with CUDA, llama.cpp with Metal
v1.5 LLM workloads, plus experimental power and energy measurement through MLPerf Power and SPEC PTDaemon Windows ML support and vendor runtime updates; Windows x64, Windows on Arm, macOS, Linux (CLI, experimental) and an iPad app (GUI only)
v2.0 Agentic SWE and Data Analyst scenarios; experimental image generation with Flux.2 klein 4B; Llama 3.1 8B and Phi 4 Mini mandatory, Qwen 3 8B experimental, Phi 4 Reasoning 14B extended; 4K prompts mandatory Adds NVIDIA CUDA for LLM, image generation and agentic workloads on Windows ARM64

Platform support also differs by execution provider. According to the project README, native OpenVINO runs on Windows x64 and Linux x64. Windows ML runs on Windows x64 and Arm. Native QNN runs on Windows Arm only. The Ryzen AI path runs on Windows x64 only. MLX and llama.cpp with Metal run on macOS and iOS, while llama.cpp with CUDA runs on Windows x64 and Arm. On Linux x64, the README lists only the native OpenVINO path.

MLCommons also lists minimum memory per platform. Examples include NVIDIA GeForce RTX 2000 series or newer with 8GB VRAM (16GB for extended prompts), Qualcomm Snapdragon X Elite platforms with 32GB of system memory, and Apple M-series Macs with 16GB. If you want to measure power draw, how the MLPerf Power energy measurement works is covered separately.

Columns in a Results Table Are Not Always Like-for-Like

Two machines side by side in a review may be running different inference engines. StorageReview’s v2.0 testing used llama.cpp on CUDA for the NVIDIA GPUs and Intel’s native OpenVINO path for the Arc GPU, and the reviewers said that column showed what that configuration delivers, not an engine-to-engine comparison. Another review covered OpenVINO, Windows ML with DirectML and llama.cpp with CUDA in one table, and called the rows comparisons of “platforms as delivered”.

The runtime alone can move the result a long way. In Signal65’s v0.5 testing on the same Intel Core Ultra 7 258V laptop, the OpenVINO path achieved over twice the speed of the ONNX Runtime path on first-token latency. Comparisons across versions break as well: MLCommons’ release notes say the v0.6 workloads were the same as v0.5, but performance could differ because of updated software components. Model lineups and mandatory prompt lengths changed again in v2.0.

Before you compare two numbers, check that they come from the same benchmark version, the same model and a comparable execution path. For the wider discipline of reading MLPerf and hardware inference benchmarks for deployment, see that guide. The datacentre suite, with its different scenarios, is covered in MLPerf Inference and what its numbers mean for procurement.

How Should You Read MLPerf Client Scores When Scoping On-Device Inference for a Real Project?

Use the scores to shortlist devices and execution paths, then decide on your own measurements.

TechnoLynx takes on this kind of on-device feasibility work through its Generative & Agentic AI R&D practice. Work through these checks in order:

  1. Match the model. Is your model on the benchmark’s list, at int4? If not, the scores only bound the problem.
  2. Match the prompt shape. Find the category closest to your input and output lengths, and remember that headline figures are geometric means across categories.
  3. Pick the metric your users feel. For short interactive calls, weight TTFT. For long outputs, weight TPS. For multi-step agents, weight the agentic end-to-end time and its split between inference and tools.
  4. Check fit before speed. Look for DNF or not-run results, which point to memory limits, and compare the device’s memory with your model plus your largest context.
  5. Check the execution path. Confirm the vendor path you would actually ship (OpenVINO, Ryzen AI, Genie, DirectML, llama.cpp, MLX) is the one that produced the score, and that it’s supported on your operating system.
  6. Check validity. Only scores carrying the “configuration tested by MLCommons” notice count as valid MLPerf Client scores.
  7. Profile your own loop. Run your model with your context lengths and call pattern on the candidate device before committing to an architecture.

The last step matters most. The context that makes a benchmark number meaningful comes from measuring your workload, not from the leaderboard. The work to do is the profiling itself: per-turn latency, KV-cache residency and tool time in the actual orchestration loop. That data shows whether an on-device design is viable.

For the broader question of whether a generative AI use case is technically feasible, see that guide. LynxBenchAI handles the separate question of benchmarking AI performance on your own hardware.

FAQ

Is MLPerf Client Open Source?

MLCommons describes MLPerf Client as an open benchmark.

What Is MLCommons MLPerf?

MLPerf Client is one of MLCommons’ MLPerf benchmarks. MLPerf Client is the member that targets personal computers, while the main MLPerf suites run on GPU-accelerated AI servers. For what the full family measures, see MLPerf and what its benchmarks measure for AI procurement.