The hardware capability number that the toolkit version doesn’t override
NVIDIA assigns each architecture a compute capability number that inventories the low-level operations and memory mechanisms available in that generation. It is reported as a major-minor version (7.5, 8.0, 8.6, 9.0, and so on) and it tells the software stack which instructions, precision formats, and tensor-core operations are available on the target hardware. For AI workloads, this number matters more than the headline CUDA toolkit version, because it is what determines whether the kernels the framework wants to run will actually find hardware to run on.
The common confusion is to treat the CUDA toolkit version as the binding constraint — to assume that a workload that “supports CUDA 12” will run equivalently on any GPU the toolkit accepts. The toolkit accepts a range of compute capabilities, but the workload’s actual behavior on each one depends on what that compute capability supports, not on what the toolkit version is. We see this confusion regularly in procurement discussions where teams treat a “CUDA-compatible” line item as if it were a uniform substrate. It is not.
What does CUDA compute capability actually control?
Capability versioning sits at the hardware layer, beneath drivers and runtimes. Each compute capability adds, removes, or changes specific architectural features:
- Precision-format support. Tensor-core operations on FP16 require compute capability 7.0+. BF16 tensor-core operations require 8.0+. FP8 (E4M3, E5M2) tensor-core operations are available starting at 8.9 (Ada) and 9.0 (Hopper). INT8 tensor cores are available on 7.2/7.5+. A workload that uses BF16 matrix multiplications on tensor cores cannot do so on a 7.5 GPU because the tensor cores there do not implement BF16, regardless of which CUDA toolkit is installed.
- Tensor-core matrix shapes and precisions. The matrix shapes (warp-level matrix-multiply-accumulate) and the precision types each generation supports differ. Newer compute capabilities expose larger matrix shapes and more precision options, which the framework’s kernel selection logic uses to choose between tensor-core and general-purpose CUDA-core paths.
- Memory model features. Asynchronous copies, distributed shared memory, thread-block clusters, and similar features are introduced at specific compute capabilities. Workloads optimized to use them on newer hardware fall back to non-optimized paths on older hardware.
- Maximum threads per block, registers per thread, shared memory per block. These hardware limits change across generations and constrain how kernels are launched.
The framework’s kernel-selection logic respects compute capability: it picks the best kernel for the target hardware among those compiled into the binary. If the binary does not include a kernel for the target compute capability, the framework either falls back to a generic CUDA-core path or fails outright. PyTorch wheels, for instance, are shipped against a specific list of compute capabilities; running them on a target outside that list produces exactly the fallback or failure behavior just described.
Why compute capability matters more for AI than the toolkit version
Developers specify a CUDA toolkit when they build; the GPU checks its capability at launch. Compute capability is what the hardware provides. A toolkit version is meaningful only in conjunction with the compute capability the target hardware supports.
Two GPUs accepted by the same CUDA toolkit can still differ substantially in observed behavior. A workload that uses BF16 tensor cores will run them on a compute-capability-8.0 GPU and fall back to FP32 CUDA cores on a 7.5 GPU, producing dramatically different throughput on identical software. A workload that uses FP8 tensor cores will run on a 9.0 GPU and either fall back or fail on an 8.0 GPU. A workload that depends on thread-block clusters (a 9.0 feature) will not run at all on older hardware, regardless of toolkit version.
This is why a benchmark report that names the CUDA toolkit version but not the compute capability of the target hardware is under-specifying the executor. The toolkit accepts; the hardware delivers; the kernel chosen is the intersection of what the toolkit knows how to compile and what the hardware can execute. Stacks like TensorRT and cuDNN sit on top of this intersection — they expose tuned kernels per compute capability, which is part of why CUDA’s ecosystem depth is hard to substitute, not just its surface API.
Compute capability mapping at a glance
| Compute capability | Generation | Notable AI-relevant features |
|---|---|---|
| 7.0 / 7.2 | Volta | First-generation tensor cores; FP16 matrix-multiply accumulation |
| 7.5 | Turing | INT8 / INT4 tensor cores |
| 8.0 | Ampere (A100) | BF16 tensor cores; structured sparsity; TF32 for training |
| 8.6 | Ampere (consumer / RTX 30) | Subset of 8.0 features; different shared-memory budget |
| 8.9 | Ada Lovelace | FP8 (E4M3 / E5M2) tensor cores |
| 9.0 | Hopper | Thread-block clusters; distributed shared memory; new tensor-core APIs |
| 10.0+ | Newer generations | Per-generation additions; consult the architecture’s documentation |
According to NVIDIA’s architecture documentation, workloads optimized for 9.0 cannot map onto 8.0 hardware without degradation, and FP8-dependent code has no execution path on 8.0 at all. The toolkit’s role is to compile code paths for each target the binary intends to run on. The hardware’s role is to actually execute the path the framework selects.
What this means for benchmark interpretation
Reporting a CUDA AI benchmark requires both the toolkit version used to compile and the compute capability assumed by the binary. The toolkit version determines what was compiled. The compute capability determines what was executed. Two benchmarks with the same toolkit version on different compute capabilities are reporting on different executors, and the difference can be larger than any reasonable hardware-only comparison would suggest — because the precision regime that ran is itself different.
The reverse case also matters: a benchmark on the same compute capability across different toolkit versions can show non-obvious shifts because the framework’s kernel selection logic has access to a different set of compiled kernels. Bounded optimization in benchmarking — the principle that the optimization effort applied to the system under test must be named and bounded — therefore extends to both the toolkit version and the compute capability.
Bounding that effort is also what keeps a cross-ecosystem reading honest. In LynxBenchAI the models are prepared once, before any target device is known, and where two vendors’ kernels disagree about what is permissible the stricter constraint applies to all of them — NVIDIA (cuda), AMD (cuda via ROCm), Intel (xpu), and CPU alike. Ecosystem depth still shows up in the numbers, but as execution rather than as effort spent, which is precisely what stops a mature stack’s tuning headroom from being smuggled in as a hand-tuning advantage over a younger one. It also means a bounded-effort result says nothing about what a funded migration would eventually achieve on the far side of a switch.
One more boundary is easy to miss: comparability holds within a release name, not across them. The 26Q3 results are readable against each other because one catalogue, one set of correctness thresholds, and one optimization budget produced every side of the comparison; the catalogue itself changes between releases, so a 26Q3 CUDA number set against a later ROCm number is not a comparison at all. The published leaderboard makes the coverage visible in both directions — which devices have accumulated runs, and which are simply absent.
Building on CUDA, frameworks, and ecosystem lock-in, the practical content is that the ecosystem’s value comes from the depth of the kernel library across (toolkit version × compute capability) combinations, and that depth is what a benchmark exercises when it runs. This is also why CUDA vs ROCm comparisons on a single workload are misleading as procurement signals: the relevant question is not “which ran faster here” but “how deep does the kernel library go across the (precision, compute capability) matrix your workloads actually touch”. Our own view, from working through these comparisons with clients, is that the kernel-coverage question dominates the headline-throughput question almost every time.
A quick diagnostic checklist before quoting a CUDA benchmark
Before treating a CUDA AI benchmark as a procurement signal, confirm each of the following is explicit in the report:
- CUDA toolkit version used to compile the binary.
- Compute capability of the target GPU (not just the marketing name — 8.0 and 8.6 are not interchangeable).
- Precision regime that actually executed (FP32, TF32, FP16, BF16, FP8, INT8). The advertised precision and the executed precision can differ when the hardware doesn’t support the requested path.
- Kernel provider (cuBLAS, cuDNN, FlashAttention, TensorRT, a custom kernel) — different providers have different per-compute-capability coverage.
- Fallback behavior — was there a fallback to CUDA cores, and if so, on which operations?
A report missing any of items 1–3 is under-specifying the executor. Items 4–5 are needed to interpret the magnitude of any cross-GPU difference.
The framing that helps
Which data types, tensor-core modes, and memory instructions a GPU supports is entirely a function of its compute capability. The CUDA toolkit version is what the application requests; the compute capability is what the hardware provides; the kernel that runs is the intersection. Benchmark reports must declare both for the result to be interpretable.
LynxBenchAI treats the (toolkit version, compute capability, precision regime) tuple as part of the AI Executor specification — alongside the GPU model — because the precision regime that actually executes is determined by that tuple, and the per-precision performance the benchmark measures depends on which regime ran. For the CUDA AI number you are about to cite, does the report disclose which compute-capability-specific kernels the runtime actually dispatched — the kernel coverage the production software stack will load — or is the per-precision throughput being read off a GPU marketing name whose advertised precision the kernel path may not honour?
Frequently Asked Questions
Which compute capability does my GPU need for BF16 tensor-core operations?
BF16 tensor-core operations require compute capability 8.0 or higher. A 7.5 (Turing) GPU does not implement BF16 on its tensor cores, so a workload requesting BF16 falls back to FP32 CUDA cores regardless of which CUDA toolkit is installed. FP8 (E4M3, E5M2) tensor-core support starts higher still, at 8.9 (Ada) and 9.0 (Hopper).
Why does the same PyTorch build behave differently on two GPUs the CUDA toolkit both accepts?
PyTorch wheels are shipped against a specific list of compute capabilities, and the kernel-selection logic picks the best compiled kernel for the target hardware. If the two GPUs differ in compute capability — 8.0 versus 7.5, for example — they expose different precision formats and tensor-core shapes, so the executed kernel and precision regime differ even though the toolkit accepts both. That is enough to produce dramatically different throughput on identical software.
What must a CUDA AI benchmark disclose before I treat it as a procurement signal?
At minimum it must name the CUDA toolkit version, the target GPU’s compute capability (not just the marketing name, since 8.0 and 8.6 are not interchangeable), and the precision regime that actually executed. The kernel provider — cuBLAS, cuDNN, FlashAttention, TensorRT, or a custom kernel — and any fallback behaviour are needed to interpret the magnitude of cross-GPU differences. A report missing the first three under-specifies the executor.
Why can a CUDA-versus-ROCm result only be read within a single release name?
Comparability comes from the fact that one instrument, one catalogue, and one bounded optimization budget produced both sides of the comparison. The catalogue changes between releases, so pairing a 26Q3 CUDA number with a number carrying a different release name compares two different measurements rather than two devices. Cite the release name alongside the result, and keep both sides inside it.
Why does capping the optimization budget identically across ecosystems matter here?
Because otherwise a mature ecosystem’s depth arrives as tuning effort rather than as execution. Models are prepared once before any target device is known, and where two vendors’ kernels disagree about what is permissible the stricter constraint binds all of them, so the compute-capability-specific kernel coverage described above shows in the result on its own merits. The trade-off is that a bounded-effort number does not predict what a funded migration would reach on that ecosystem.
Compute capability determines which instructions you can call
A 7.5 device cannot execute Hopper’s asynchronous memory copy or thread-block-cluster instructions; binaries must either provide fallback code or exit. Is that executor close enough to yours for the result to mean anything?
