Precision is not a free model-design parameter
Deployment architects select precision formats—FP16, BF16, FP8, INT8—assuming uniform runtime support across accelerator targets. The runtime does support the precision; on hardware that does not natively accelerate it, the support is by emulation, and emulation runs at a performance cost large enough to negate the reason the precision was chosen in the first place. The precision regime that delivers its expected throughput is the regime the target accelerator generation actually accelerates in hardware. The regime the target generation only emulates is, for performance purposes, a regime the target hardware does not support.
This makes precision a hardware-conditional design decision, not a free model-design parameter. The decision and the hardware decision interact, and choosing one without the other locks in implications the chooser may not have intended. In our experience reviewing procurement matrices alongside model teams, this is the single most common place where a deployment plan written in isolation collides with the accelerator fleet it eventually lands on.
What does “supported” mean at the hardware level?
Contemporary AI accelerators ship dedicated matrix engines (NVIDIA tensor cores, AMD CDNA matrix units, Intel Gaudi equivalents) hardwired for specific numeric formats. The set of natively-supported precisions differs by accelerator generation and is the practical determinant of which precisions the deployment can use at peak throughput. Frameworks like PyTorch, JAX, and TensorRT will happily compile a graph against a precision the hardware does not natively accelerate; the compilation succeeds, but what runs at inference time is not what the model architect drew on the design document.
Three categories of “support” matter:
Native acceleration. The matrix engine has dedicated paths for the precision. Throughput at this precision approaches the device’s design-target peak for that format, and the precision is the operationally usable one for high-throughput workloads.
Software emulation. The precision is supported by the runtime via composition of operations on a different native precision — for example, emulating FP16 by sequences of FP32 operations on a device that lacks FP16 tensor cores, or expressing FP8 matmuls as BF16 tensor-core calls with scaling on a generation that only natively accelerates BF16. Functionally correct; performance-wise, often slower than just running the workload natively at the supported precision in the first place.
Unsupported. The runtime does not implement the precision at all on the target hardware. The workload either falls back to a different precision automatically — with the framework’s mixed-precision logic making the decision out of view of the operator — or fails outright at graph-compile time.
A precision regime that delivers its expected speedup on one accelerator generation can be silently emulated on another, producing throughput that is worse than running the workload at a higher precision the older hardware does support natively. The “FP8 is roughly 2x faster than BF16” statement is a property — typically observed under conditions like dense matmul-bound transformer inference on accelerators that natively accelerate FP8 (observed-pattern; not a benchmarked rate across a device family); on accelerators that emulate it, the same statement can be false, and frequently is.
Generation-conditional precision support
Accelerator generations exhibit uneven precision support with historical additive layering: new devices add formats, legacy hardware receives no retrofits. A simplified picture, per NVIDIA’s published architecture specifications:
| Format | Native acceleration first appeared in | Notes |
|---|---|---|
| FP32 | All generations | Universally supported |
| FP16 tensor cores | Volta (compute capability 7.0) | Mixed-precision standard for several generations |
| INT8 tensor cores | Turing (compute capability 7.5) | Strong inference support |
| BF16 tensor cores | Ampere (compute capability 8.0) | Wide dynamic range; preferred for training |
| TF32 | Ampere (compute capability 8.0) | Reduced-precision FP32 training format |
| FP8 tensor cores | Ada Lovelace (8.9) and Hopper (9.0) | E4M3 and E5M2 variants |
| FP4 tensor cores | Blackwell-class generations | Aggressive inference quantization |
Equivalent capability tables exist for AMD’s CDNA generations and for Gaudi accelerators with different generation boundaries and different specific format support. The pattern that recurs across vendors is the same: precision support is generation-conditional, and “the hardware supports X” is a question that has to be answered per-generation, not per-vendor.
The procurement consequence is that hardware choice and precision-regime choice are coupled. A deployment built on FP8 cannot run on hardware older than the FP8-introducing generation without emulating, which means the procurement decision to buy older hardware retires the FP8 deployment option for that fleet. A deployment built on FP16 + mixed precision can run on most modern hardware, which means a precision-regime choice that constrains the deployment to FP8 also constrains the procurement choice to FP8-supporting hardware.
Why the specification sheet and the machine disagree
One level beneath the generation compatibility table lies a second discontinuity. A card can belong to an FP8-accelerating generation and still fail to produce an FP8 result, because the format has to be reachable through the software stack actually installed: the backend (cuda for NVIDIA, cuda via ROCm for AMD, xpu for Intel, or CPU), plus the driver, framework, and runtime present on that machine. This is why a score is only meaningful when it is bound to the AI Executor — the device together with the stack it ran through — never to silicon on its own.
The released 26Q3 LynxBenchAI run handles this by declaring support through execution rather than through a table: each precision in the catalogue is attempted on the device, and a precision the machine cannot execute is recorded as a zero rather than quietly skipped. The unsupported format appears in the result instead of disappearing from it. That zero is not a verdict on the device; it records the absence of that regime under one release’s conditions, on one executor. When a specification sheet says FP8 and the run says zero, the disagreement is information — about the driver, the framework build, or the runtime, not about the die.
One further property matters for cross-vendor reading: a single catalogue and a single rule set run across NVIDIA, AMD, Intel, and CPU. A per-precision gap between two vendors is therefore measured by one instrument rather than assembled from each vendor’s preferred test. pip install lynxbench-ai turns a specification-sheet claim about precision support into something the owner of the card can check on the card.
Why are precision decisions inherently hardware-conditional?
Conventional reasoning treats precision and hardware as orthogonal: select the accelerator, then choose among its supported formats. The mental model is wrong in both directions, and the symmetry of the failure is what makes it costly.
Picking precision first locks the procurement window. A deployment that requires native FP8 acceleration to meet its throughput target cannot be run on accelerators older than the FP8-introducing generation. The procurement candidate set is therefore constrained by the precision choice — and once the model is trained or fine-tuned against an FP8 numerical envelope, switching to a higher-precision deployment is not free.
Picking hardware first locks the precision option set. A deployment running on accelerators that do not natively accelerate a given low-precision format cannot adopt that format later without buying new hardware. The precision-regime evolution is therefore constrained by the hardware choice, and the constraint is not visible until the team tries to migrate to a newer quantization scheme that the existing fleet emulates rather than accelerates.
The two decisions are not independent; they are a joint decision that has to be made together. The framing that produces durable infrastructure choice is to enumerate the precision regimes the deployment will need over the planning horizon and the hardware generations that natively accelerate them, and to pick from the intersection. Picking from one set without considering the other produces deployments where one of the two becomes the constraint that closes off the other.
A benchmark methodology that supports this joint decision must report the precision regimes the candidate hardware natively accelerates and the throughput at each. A benchmark that reports a single throughput number without the precision regime is reporting on an unspecified part of the joint decision, and a procurement decision built on that benchmark is locking in implications the benchmark did not characterize.
What a precision-by-hardware matrix looks like in a benchmark
Joint decision-making requires matrix presentation: precision formats label one axis, candidate accelerators the other, with throughput and accuracy populating each intersection. The matrix exposes:
- Which precisions each accelerator natively accelerates.
- Where emulation is happening (cells where throughput is far below the format’s expected peak).
- Where the precision could not be executed at all (cells scored zero under that release’s conditions).
- The trade-off space across the (precision, hardware) joint decision rather than along either axis alone.
A benchmark that produces a row — single precision across hardware — supports a hardware-only comparison. A benchmark that produces a column — single hardware across precisions — supports a precision-only investigation. A benchmark that produces a matrix supports the joint decision the procurement actually faces. Our position is that the matrix form is the minimum disclosure for hardware-evaluation benchmarks that intend to inform procurement, not an optional enrichment of a simpler scorecard.
Two reading rules keep such a matrix honest. A per-precision figure carries the name of the release that produced it, so a figure from one release is not comparable with a figure from another. And the aggregate score attached to a device is ordinal — a rank-style aggregate over Training, Inference, and Compute scores — not a physical quantity and not a 0–100 rating. Results accumulate on a public leaderboard, where a device’s absence is itself readable: it means nobody has run it under these conditions yet, which is a different statement from a low score.
The broader treatment of why precision choices are constrained by hardware architecture makes the structural case; the operational expression here is that the set of viable precision regimes is an artifact of the hardware-architecture choice and of the executor assembled around it — making precision and hardware a single joint decision rather than two independent ones, and making the LynxBenchAI per-precision-per-AI-Executor matrix the reporting form that surfaces that fact rather than hiding it.
The framing that helps
Native format acceleration delivers advertised throughput; precision support depends on accelerator generation, and emulated operations fail to meet performance expectations. Precision regime and hardware choice are coupled — picking either first locks implications for the other. Procurement and architecture decisions about AI deployments must therefore be made jointly, against the precision-by-hardware matrix the candidate set actually presents, not against a single throughput number that hides which precision regime produced it.
Because a precision is declared supported only by being exercised, the matrix that matters is the one built from runs on the executors you actually own. On the candidate set in front of you, which cells carry native acceleration of the precision regime your workload requires, which are quietly emulating it under a headline that hides the difference, and which came back as zeros that your specification sheets say should not exist?
Frequently Asked Questions
How do block scaling and blockwise quantization interact with hardware support?
Silicon vendors often expose a precision format in their ISA before adding dedicated acceleration for the quantization and dequantization steps that format requires in practice. FP8 matmuls, for example, often rely on per-block or per-tensor scaling, and a generation that lacks native scaling paths must compose that machinery in software. The result is a format that compiles and runs but does not reach the throughput the format is marketed for, because the scaling overhead is absorbed off the fast path.
If a specification sheet lists FP8 support but a run records a zero for that precision, what does the gap tell an architect?
It tells the architect to look at the stack, not the die. A score binds to the AI Executor — the device plus the backend, driver, framework, and runtime on that machine — so a zero against a listed capability points at something in that chain being unable to reach the format. The zero is not a judgement on the device; it records the absence of that regime under one release’s conditions, and it is the signal to check the installed runtime before revising the procurement shortlist.
Why does a per-precision comparison across NVIDIA, AMD, Intel, and CPU only hold when one catalogue is applied to all four?
Because a gap assembled from each vendor’s own precision test measures the tests as much as the hardware. Running a single catalogue and a single rule set across all four means the difference between two cells is attributable to the executors rather than to differing methodologies. Comparisons stitched together from vendor-preferred benchmarks cannot support that attribution.
What is the practical difference between tensor-core execution and general shader execution for a given precision?
Tensor-core execution uses the dedicated matrix-multiply path and approaches the device’s design-target peak for that format; general shader or CUDA-core execution composes the same arithmetic on the general-purpose path. A per-precision score produced off the matrix-engine path reflects that composition, not the format’s headline capability — which is why the same precision label can appear on two very different numbers for the same silicon generation.
Six Hardware Limits That Override Software Optimizations
Whenever operation formats mismatch tensor core specifications, these specialized units remain unutilized. So the question to carry forward is this: do you know the executor, the rules, and the release behind the number in front of you — and if not, what would it take to find out?
