A common confusion with concrete consequences

A team installs PyTorch, runs torch.version.cuda, and sees a CUDA version that doesn’t match what nvcc --version reports for the system-installed CUDA toolkit. The instinct is that something is misconfigured. Usually it isn’t — the two numbers describe different things, and the discrepancy is the expected behaviour of how PyTorch ships CUDA support. The trouble starts when a benchmark is reported as “PyTorch X.Y on GPU Z” and that string under-specifies the AI Executor in ways that prevent the result from reproducing on a different host.

Understanding the relationship between torch.version.cuda, the system CUDA toolkit, the installed driver, and the GPU’s compute capability is the prerequisite for reading or producing reproducible PyTorch benchmark results. It is also where ecosystem depth stops being an abstraction and starts behaving like a procurement constraint: the specific tuple that produced a number is the thing the buyer is actually committing to, not the framework name on the slide.

What does torch.version.cuda actually report?

PyTorch wheels ship with pre-built CUDA runtime libraries inside the wheel itself. When PyTorch is installed via pip install torch, the wheel contains compiled CUDA kernels and the CUDA runtime libraries (cuBLAS, cuDNN, NCCL, and others) that PyTorch needs. These vendored libraries are built against a specific CUDA toolkit version, and that toolkit version is what torch.version.cuda reports.

This is the toolkit version PyTorch was compiled against — not the system-installed CUDA toolkit. A system that has CUDA 11.8 installed system-wide can run a PyTorch wheel built for CUDA 12.1, and torch.version.cuda will report 12.1 because that is what PyTorch was compiled against. The system toolkit is irrelevant to the operation of PyTorch in the typical configuration; PyTorch uses its own vendored runtime.

What PyTorch does require from the host is a sufficiently recent NVIDIA driver — one whose forward-compatibility range covers the toolkit version PyTorch was built with. The driver is what actually talks to the GPU, and the driver version determines which CUDA toolkit versions can be used through it. If the system driver is too old for PyTorch’s vendored toolkit, PyTorch fails at GPU initialization regardless of what is installed system-wide.

The three numbers that all matter, separately

For a PyTorch workload’s behaviour to be specified, three (effectively four) numbers have to be captured:

• torch.version.cuda — the toolkit version PyTorch was built against. This determines the CUDA APIs PyTorch uses and the cuBLAS/cuDNN versions vendored in the wheel.
• System driver version (nvidia-smi) — what the kernel-mode driver supports. This sets the upper bound on toolkit versions that can be used via this driver.
• GPU compute capability — the hardware capability the GPU exposes. This determines which precision regimes and kernels can actually execute.
• PyTorch wheel source — different builds of the same PyTorch version (PyPI default, NVIDIA-published, conda-forge, distribution-vendored) can vendor different toolkit components and produce different runtime behaviour.

A workload that runs on one combination of (torch.version.cuda, system driver, GPU compute capability) is not guaranteed to reproduce on a different combination, even when the PyTorch version string is identical. The reasons are concrete:

• A different torch.version.cuda means different vendored cuDNN/cuBLAS, which can select different kernels for the same operation.
• A different driver can expose different runtime features or carry different bug-fix profiles.
• A different compute capability can route the workload through a different kernel path or fall back to a less-optimized implementation.
• A different wheel source can vendor a different cuDNN or NCCL version even at the same torch.version.cuda.

The torch.version.cuda number alone tells the reader nothing about whether their host will reproduce the result. This is an observed pattern across the PyTorch benchmark reports we encounter in practice — under-specification of the runtime tuple is the modal cause of “I can’t reproduce that number” exchanges, not floating-point nondeterminism or hardware variance.

A reproducibility checklist for PyTorch CUDA benchmarks

A benchmark reporting “PyTorch X.Y on GPU Z” is missing the dimensions the reader needs to reproduce or interpret the result. The minimum disclosure surface for a PyTorch CUDA benchmark to be reproducible includes:

• PyTorch version string (e.g. 2.3.0)
• PyTorch wheel source (PyPI / NVIDIA NGC / conda-forge / distribution package)
• torch.version.cuda (the toolkit version PyTorch was built against)
• System NVIDIA driver version (nvidia-smi)
• GPU model and compute capability
• cuDNN version reported by torch.backends.cudnn.version()
• Whether a custom CUDA toolkit was prepended to the runtime path (LD_LIBRARY_PATH manipulation that overrides PyTorch’s vendored libraries)
• OS and kernel version, since some driver behaviour is OS-conditional
• Workload precision (FP32 / TF32 / FP16 / BF16 / FP8) and whether tensor cores were exercised

A report that satisfies this list can be reproduced by a different team. A report that satisfies a subset cannot, and the subset that’s missing is the difference between informative and indeterminate.

Common patterns that produce non-reproducible PyTorch benchmarks

Three patterns in particular are responsible for most of the “I can’t reproduce that PyTorch number” experience we see across engagements.

The first is silent toolkit override. A user who has set LD_LIBRARY_PATH to include a system CUDA toolkit can shadow PyTorch’s vendored libraries with the system-installed versions. The benchmark then runs on a different cuDNN/cuBLAS than torch.version.cuda suggests. This is invisible from torch.version.cuda alone, and it is one of the more frustrating failure modes because the disclosed numbers all look consistent.

The second is wheel-source variance. PyTorch wheels published by NVIDIA NGC, the PyTorch project, conda-forge, and Linux distribution packages can differ in which cuDNN they vendor and how they were compiled. A benchmark reported with a PyTorch version string but no wheel-source disclosure can be reproduced only if the reader installs from the same source — and there is no reliable runtime probe that recovers the source after install.

The third is driver mismatch. Different system drivers can produce different observed behaviour on the same PyTorch + same GPU, particularly for workloads that exercise newer driver features. A driver version disclosed only as “current” or “latest” produces non-reproducible results six months later, when “current” means something different.

Why this matters beyond debugging

The stack tuple isn’t just a debugging artefact — it is the thing being procured. An accelerator decision that looks like “PyTorch on Vendor X versus PyTorch on Vendor Y” is really a decision about two different runtime tuples, each with its own cuDNN-equivalent library, its own kernel selection logic, and its own driver behaviour. On NVIDIA hardware, the depth of cuDNN, NCCL, FlashAttention kernels, TensorRT export paths, and Triton compatibility is the substance of what makes the published numbers achievable in production. On a competing accelerator, the equivalent tuple has its own depth, its own gaps, and its own kernels that may or may not match the operations a given model uses.

This is why we treat the runtime tuple as part of the AI Executor specification rather than as setup detail. The strategic framing — why ecosystems amplify or suppress raw hardware capability — sits in our hub article on CUDA, frameworks, and ecosystem lock-in; the measurement-side question of why a single GPU benchmark misleads procurement covers the same theme from the measurement-design angle. The operational counterpart is concrete: if you can’t write down the tuple, you can’t compare two accelerators on anything other than marketing. ## Closing

torch.version.cuda reports the toolkit version PyTorch was built against, not the system CUDA toolkit. The relationship between this number, the system driver, and the GPU’s compute capability determines which kernels execute and what performance the workload achieves. Benchmark reproducibility requires all three numbers — plus the wheel source and the cuDNN version — to be disclosed; a report that names only torch.version.cuda is under-specifying the AI Executor in ways that block reproduction on any other host.

LynxBench AI treats the (PyTorch version, wheel source, torch.version.cuda, driver, compute capability, cuDNN) tuple as part of the AI Executor specification — alongside the GPU model — because that tuple is what determines whether a benchmark result transfers to any other host that nominally has the same PyTorch and GPU. Which exact (PyTorch version, wheel source, torch.version.cuda, driver, compute capability, cuDNN) tuple was the cited benchmark run against — and does any host you plan to deploy on match it kernel-for-kernel?