Phoronix provides reproducible AI-relevant GPU benchmarks
Phoronix Test Suite (PTS) diverges from Geekbench and 3DMark by shipping test profiles that invoke real AI framework code: TensorFlow training benchmarks, PyTorch inference tests, ONNX Runtime profiles, and quantised LLM runs via llama.cpp. For comparing GPU hardware in a documented, reproducible way, PTS is more relevant to AI than most consumer benchmark alternatives — provided you read the numbers correctly. The trap is treating a PTS score as a forecast of production throughput. It is not. It is a controlled snapshot of one fixed workload running on a particular stack, and the gap between that snapshot and what your inference service will actually see can be larger than the gap between two competing GPUs.
We run PTS regularly in our engagements, mostly as a driver and stack validation tool. That framing — scaffolding, not verdict — is the one most teams miss.
Setting up Phoronix for GPU AI testing
# Install Phoronix Test Suite
wget https://phoronix-test-suite.com/releases/phoronix-test-suite-10.8.4.tar.gz
tar xzf phoronix-test-suite-10.8.4.tar.gz
cd phoronix-test-suite
sudo ./install-sh
# Run TensorFlow benchmark
phoronix-test-suite benchmark tensorflow
# Run PyTorch benchmark
phoronix-test-suite benchmark pytorch
# Run ONNX Runtime benchmark
phoronix-test-suite benchmark onnxruntime
Each profile pins a model, a batch shape, and a precision. That pinning is what makes the run reproducible — and also what makes it narrow. A benchmark is usable to the extent its scope is declared, so read the profile definition as the scope statement it is: this model catalogue, these precisions, this measurement window, and nothing about your application. The profile that runs on your test node is the same profile that ran on the published comparison node, which is exactly the property production workloads do not have.
What are the key AI-relevant Phoronix test profiles?
| Profile | Model tested | Metric | What it measures |
|---|---|---|---|
| tensorflow-benchmark | ResNet-50 | Images/second | Training throughput, fixed batch |
| pytorch-benchmark | ResNet-50, BERT | Items/second | Training/inference, fixed batch |
| onnxruntime | ResNet-50 | Latency/throughput | Inference framework path |
| llama.cpp | Quantised LLM | Tokens/second | CPU+GPU LLM inference |
Every profile follows the same template: single-stream, fixed batch, fixed model. That is the right shape for a reproducible test. It is the wrong shape for predicting an inference service that sees concurrent requests, variable sequence lengths, and queuing.
Interpreting Phoronix GPU benchmark results
We enforce a set of interpretive rules each time we analyse PTS results.
ResNet-50 training throughput is a useful relative comparison across training infrastructure but does not transfer cleanly to modern architectures. ViTs, diffusion U-Nets, and decoder-only transformers stress different parts of the GPU — attention kernels, memory bandwidth into HBM, tensor-core utilisation under FlashAttention — in different proportions than ResNet’s convolutional stack. A GPU that wins on ResNet-50 by a comfortable margin can lose on long-context attention because the bottleneck has moved.
ONNX Runtime inference tests use smaller models and batch sizes than production. GPU efficiency at batch=1 versus batch=32 is non-linear, and the curve is model-dependent. Scaling a PTS latency number linearly to production batch size is one of the most common misreadings we see.
Cross-submission comparisons require matching the software environment. PTS publishes community benchmark results, and the temptation is to compare your number to the leaderboard. Driver version, CUDA version, cuDNN version, and framework version each move the result. In our experience, differences in the software stack alone produce roughly 20–40% variation on identical hardware (observed pattern across the engagements where we have pinned hardware and varied stacks; not a published benchmark).
Quick-answer block: what PTS does and does not tell you
| Question | PTS answer quality |
|---|---|
| Is my GPU driver stack functional end-to-end? | High — a failing PTS run reliably indicates a problem |
| Will this GPU outperform that GPU on my workload? | Low — only on PTS’s fixed workload, not yours |
| How will the GPU scale to my batch and concurrency? | None — PTS is single-stream, fixed batch |
| Did a driver update regress AI throughput? | High — controlled before/after on identical hardware |
| What is my absolute production throughput? | None — PTS measures a proxy workload |
Rigorous protocol in PTS application distinguishes credible findings from deceptive ones. The suite is a controlled-environment instrument. It is not a production predictor.
What does a Phoronix GPU test tell you about AI readiness?
Three families compose Phoronix’s GPU benchmark catalogue: OpenGL rendering (Unigine, GpuTest), Vulkan compute (vkpeak), and framework-specific AI tests (PyTorch, TensorFlow, ONNX Runtime, llama.cpp). The AI-specific tests are the only ones that predict AI workload performance with reasonable accuracy — and even those measure a narrow slice of the surface a production system actually traverses.
The PyTorch benchmark in PTS runs ResNet-50 inference at a fixed batch size. That tells you whether the GPU, driver, and CUDA/cuDNN stack are correctly installed and functioning. It does not tell you how the GPU will behave on your specific model architecture, sequence length, or batch configuration. A Stable Diffusion run, an LLM inference run, and a ResNet-50 inference run stress different subsystems — compute units, memory bandwidth, tensor cores, the attention kernel path — in different proportions. The PTS profile gives you one point in a high-dimensional space.
We use PTS primarily as a driver validation tool. After installing or updating NVIDIA drivers on Linux, running the PTS PyTorch test confirms that the full chain — driver, CUDA runtime, cuDNN, PyTorch, model execution — is functional. A passing PTS result does not guarantee production readiness. A failing PTS result reliably indicates a stack problem. That asymmetry is the useful property.
For cross-vendor comparison (NVIDIA versus AMD), PTS provides a controlled environment where both vendors run the same test code. This eliminates the software-stack variable that confounds most ad-hoc cross-vendor comparisons. The catch: PTS framework tests typically do not use vendor-specific optimisations — no FlashAttention on the NVIDIA side, no MIOpen tuning on the AMD side — so the results reflect unoptimised baseline performance rather than what a production-tuned deployment would reach. The more defensible variant of that design is bounded, disclosed optimisation effort: models prepared once before any target device is known, the stricter constraint applied to every vendor rather than the friendliest one per vendor, and a workload that fails or times out recorded as a zero instead of quietly dropped from the average. That is also the discipline LynxBenchAI applies, whose 26Q3 release covers a fixed model catalogue at fixed precisions inside a continuous timed window after a discarded warm-up — a stated scope, not a forecast of your serving stack. Where you can execute the real software, that execution is the reference standard; specification-based estimation from clock rates and memory bandwidth is the fallback, not the equal. In rough terms, we see the gap between PTS results and production-tuned performance run at roughly 20–40% on NVIDIA (where framework optimisations are mature) and 40–60% on AMD (where additional tuning effort is required). These are observed patterns from our deployments, not published benchmarks.
That is enough margin to make PTS useful for sanity-check ballpark comparison and unreliable for procurement decisions where the difference between two configurations is being weighed in single-digit percent.
Comparing PTS results across driver versions
Tracking AI performance regressions and gains across driver updates represents an underexploited use case for PTS. Running the same profile before and after a driver update on identical hardware produces a controlled comparison that isolates the driver’s performance impact — exactly the kind of variable that disappears into noise in production traffic.
We maintain a PTS result database for our production GPU configurations. When evaluating a driver update — for example, moving from a 535.x line to a 550.x line — we run the PTS PyTorch and TensorFlow profiles on a test node before updating, then again after. A throughput change of more than 3% triggers investigation: either the new driver has introduced a regression (which we then report upstream) or it has enabled an optimisation worth understanding before rolling forward.
This approach has caught three significant driver regressions before they reached production over an 18-month window. In each case the PTS test showed a 5–12% throughput drop (we measured this in our internal testing history) that had been invisible in manual production observation because it fell inside the normal variation of live traffic. The controlled, identical-workload comparison made the regression visible (observed in our own engagement history; not a benchmark you can run against our environment).
Why the PTS number is not the production number
Here lies the central challenge. PTS pins the workload. Production does not.
That is also why a GPU can report 96-98% utilisation (as an illustrative observed range) under concurrent, variable load and still deliver poor real-world throughput: the device is busy, but a large share of that work is padding, stalled KV-cache reads, and partially-filled batches rather than useful tokens.
That structural mismatch is why otherwise-honest published benchmark numbers — PTS or otherwise — routinely fail to predict what a team sees in production. The benchmark is not wrong. The benchmark is answering a different question than the one procurement is asking. Realism is not a binary property the benchmark either has or lacks; it is a question of how close the benchmark’s workload shape sits to the production workload shape on the axes that matter for the architecture in question. For an extended treatment of that mismatch, why benchmarks fail to match real AI workloads covers the structural gap in more detail.
When PTS is the right tool
PTS earns its place when the question matches its shape:
- Driver and stack validation. After any change to the driver, CUDA, or cuDNN, a fixed PTS profile is the cheapest end-to-end confirmation that the chain still functions.
- Regression detection across software updates. Identical hardware, identical profile, before-and-after — the controlled comparison PTS makes possible is hard to get any other way.
- Baseline documentation for a new deployment. Recording PTS results for a freshly provisioned node creates a reference point for the next time something looks off.
- Cross-vendor sanity check. Same test code on NVIDIA and AMD, unoptimised path — useful for ballpark, not for procurement margin.One mismatch is fixable inside a benchmark’s own design, though. A fixed workload run on every device flatters whichever device class the workload was sized for; scaling the workload until each device reaches its own saturation point removes that specific bias, which is what per-device saturation is for. It does not pretend to reproduce production request patterns — concurrency, queuing, and arrival variability stay outside the instrument. And a related trap: a score produced under one release name cannot be compared against one produced under another, because the catalogue and precision set moved underneath it.
PTS is the wrong tool when the question is “how will this GPU perform on my model under my traffic?” That question requires running your model under your traffic, on the candidate hardware, with the precision and executor configuration that production will actually use.
Frequently Asked Questions
How does dynamic batching in a serving stack like vLLM change observed GPU utilisation versus a single-stream Phoronix run?
A single-stream PTS profile pins one request at a time, so utilisation reflects a fixed batch on a fixed model. A stack like vLLM coalesces concurrent requests into batches dynamically, which means observed utilisation becomes a function of arrival rate, queue depth, and sequence-length spread. The same GPU can sit far below its PTS occupancy when traffic is sparse, or saturate with mostly-padding work when sequence lengths vary widely — neither of which the benchmark captures.
Why can a GPU show 96-98% utilisation and still deliver poor real-world throughput under concurrent load?
Utilisation measures occupancy — whether the device has work scheduled — not goodput, the useful tokens or items it actually completes. Under concurrent, variable request load, much of that occupied time can be padding inside partially-filled batches, stalled KV-cache reads, or scheduling overhead. A GPU can therefore read 96-98% busy (as an illustrative range) while a large fraction of its cycles produce no useful output.
How should I reason about a published GPU score whose measurement window, model catalogue, and precision set are all fixed when my stack differs on all three?
Treat the fixed elements as a declared scope rather than a defect, then check each axis against your own stack. A score that covers one named release’s model catalogue at fixed precisions, measured inside a continuous timed window after a discarded warm-up, tells you about that catalogue on that hardware and not about your model, your precision, or your traffic. Scores carrying different release names are not comparable to each other either, since the catalogue moved underneath them.
Does scaling a benchmark workload to each device’s saturation point make it a production predictor?
No — it removes one specific bias, not the mismatch as a whole. A single fixed workload run on every device flatters whichever device class it was sized for; letting each device run up to its own saturation point takes that distortion out of the comparison. Concurrency, queuing, and request-arrival variability remain outside the instrument, which is exactly the part you still have to measure on your own stack.
Should I use Phoronix Test Suite results to make a GPU procurement decision?
Not on their own. PTS runs single-stream, fixed-batch, fixed-model profiles without vendor-specific optimisations, and we observe the gap between PTS results and production-tuned performance running roughly 20-40% on NVIDIA and 40-60% on AMD (observed across our deployments; not a published benchmark). That margin is wide enough to make PTS a useful sanity check but unreliable when a decision turns on single-digit-percent differences between configurations.
Adapting Phoronix methods for transformer loads
Suites built for graphics workloads require prompt libraries and batch-size sweeps before they can reliably evaluate LLM hardware. So the question to carry forward is this: do you know the executor, the rules, and the release behind the number in front of you — and if not, what would it take to find out?
