A common confusion with concrete consequences

Teams frequently encounter a mismatch: torch.version.cuda returns one CUDA version while nvcc --version reports another for the system toolkit. The instinct is that something is misconfigured. Usually it isn’t — the two numbers describe different things, and the discrepancy is the expected behaviour of how PyTorch ships CUDA support. The trouble starts when a benchmark is reported as “PyTorch X.Y on GPU Z” and that string under-specifies the AI Executor in ways that prevent the result from reproducing on a different host.

Understanding the relationship between torch.version.cuda, the system CUDA toolkit, the installed driver, and the GPU’s compute capability is the prerequisite for reading or producing reproducible PyTorch benchmark results. It is also where ecosystem depth stops being an abstraction and starts behaving like a procurement constraint: the specific tuple that produced a number is the thing the buyer is actually committing to, not the framework name on the slide.

What does torch.version.cuda actually report?

Pre-built CUDA runtime libraries ship embedded inside PyTorch wheels. When PyTorch is installed via pip install torch, the wheel contains compiled CUDA kernels and the CUDA runtime libraries (cuBLAS, cuDNN, NCCL, and others) that PyTorch needs. These vendored libraries are built against a specific CUDA toolkit version, and that toolkit version is what torch.version.cuda reports.

This is the toolkit version PyTorch was compiled against — not the system-installed CUDA toolkit. A system that has CUDA 11.8 installed system-wide can run a PyTorch wheel built for CUDA 12.1, and torch.version.cuda will report 12.1 because that is what PyTorch was compiled against. The system toolkit is irrelevant to the operation of PyTorch in the typical configuration; PyTorch uses its own vendored runtime.

What PyTorch does require from the host is a sufficiently recent NVIDIA driver — one whose forward-compatibility range covers the toolkit version PyTorch was built with. The driver is what actually talks to the GPU, and the driver version determines which CUDA toolkit versions can be used through it. If the system driver is too old for PyTorch’s vendored toolkit, PyTorch fails at GPU initialization regardless of what is installed system-wide.

The three numbers that all matter, separately

For a PyTorch workload’s behaviour to be specified, three (effectively four) numbers have to be captured:

• torch.version.cuda — the toolkit version PyTorch was built against. This determines the CUDA APIs PyTorch uses and the cuBLAS/cuDNN versions vendored in the wheel.
• System driver version (nvidia-smi) — what the kernel-mode driver supports. This sets the upper bound on toolkit versions that can be used via this driver.
• GPU compute capability — the hardware capability the GPU exposes. This determines which precision regimes and kernels can actually execute.
• PyTorch wheel source — different builds of the same PyTorch version (PyPI default, NVIDIA-published, conda-forge, distribution-vendored) can vendor different toolkit components and produce different runtime behaviour.

A workload that runs on one combination of (torch.version.cuda, system driver, GPU compute capability) is not guaranteed to reproduce on a different combination, even when the PyTorch version string is identical. The reasons are concrete:

• A different torch.version.cuda means different vendored cuDNN/cuBLAS, which can select different kernels for the same operation.
• A different driver can expose different runtime features or carry different bug-fix profiles.
• A different compute capability can route the workload through a different kernel path or fall back to a less-optimized implementation.
• A different wheel source can vendor a different cuDNN or NCCL version even at the same torch.version.cuda.

The torch.version.cuda number alone tells the reader nothing about whether their host will reproduce the result. This is an observed pattern across the PyTorch benchmark reports we encounter in practice — under-specification of the runtime tuple is the modal cause of “I can’t reproduce that number” exchanges, not floating-point nondeterminism or hardware variance.

A reproducibility checklist for PyTorch CUDA benchmarks

“PyTorch X.Y on GPU Z” reports omit the reproducibility dimensions readers require. The minimum disclosure surface for a PyTorch CUDA benchmark to be reproducible includes:

• PyTorch version string (e.g. 2.3.0)
• PyTorch wheel source (PyPI / NVIDIA NGC / conda-forge / distribution package)
• torch.version.cuda (the toolkit version PyTorch was built against)
• System NVIDIA driver version (nvidia-smi)
• GPU model and compute capability
• cuDNN version reported by torch.backends.cudnn.version()
• Whether a custom CUDA toolkit was prepended to the runtime path (LD_LIBRARY_PATH manipulation that overrides PyTorch’s vendored libraries)
• OS and kernel version, since some driver behaviour is OS-conditional
• Workload precision (FP32 / TF32 / FP16 / BF16 / FP8) and whether tensor cores were exercised

A report that satisfies this list can be reproduced by a different team. A report that satisfies a subset cannot, and the subset that’s missing is the difference between informative and indeterminate.

Common patterns that produce non-reproducible PyTorch benchmarks

Most “I can’t reproduce that PyTorch number” cases we encounter across engagements trace to three specific patterns.

The first is silent toolkit override. A user who has set LD_LIBRARY_PATH to include a system CUDA toolkit can shadow PyTorch’s vendored libraries with the system-installed versions. The benchmark then runs on a different cuDNN/cuBLAS than torch.version.cuda suggests. This is invisible from torch.version.cuda alone, and it is one of the more frustrating failure modes because the disclosed numbers all look consistent.

The second is wheel-source variance. PyTorch wheels published by NVIDIA NGC, the PyTorch project, conda-forge, and Linux distribution packages can differ in which cuDNN they vendor and how they were compiled. A benchmark reported with a PyTorch version string but no wheel-source disclosure can be reproduced only if the reader installs from the same source — and there is no reliable runtime probe that recovers the source after install.

The third is driver mismatch. Different system drivers can produce different observed behaviour on the same PyTorch + same GPU, particularly for workloads that exercise newer driver features. A driver version disclosed only as “current” or “latest” produces non-reproducible results six months later, when “current” means something different.

Why the tuple travels badly across ecosystems

Procurement targets the stack tuple, not just an abstraction layer. An accelerator decision that looks like “PyTorch on Vendor X versus PyTorch on Vendor Y” is really a decision about two different runtime tuples, each with its own cuDNN-equivalent library, its own kernel selection logic, and its own driver behaviour. On NVIDIA hardware, the depth of cuDNN, NCCL, FlashAttention kernels, TensorRT export paths, and Triton compatibility is the substance of what makes the published numbers achievable in production. On a competing accelerator, the equivalent tuple has its own depth, its own gaps, and its own kernels that may or may not match the operations a given model uses.

The awkward part is that the tuple has no cross-vendor translation. A ROCm build of PyTorch reports something in torch.version.hip rather than a CUDA toolkit number; an Intel XPU build reports a oneAPI-side stack. So when a migration estimate says “same PyTorch, different device”, it is quietly asserting that two non-comparable tuples behave the same way — which is exactly the assumption the migration is supposed to test. This is why we treat the runtime tuple as part of the AI Executor specification rather than as setup detail: it is the only unit at which a cross-ecosystem claim can be checked at all.

A cross-ecosystem comparison is therefore only worth having if one instrument produced both sides of it, with the same catalogue, the same correctness thresholds, and the same bounded preparation effort applied to every backend. The strategic framing — why ecosystems amplify or suppress raw hardware capability, and where switching costs actually live — sits in our hub article on CUDA, frameworks, and ecosystem lock-in. The measurement-design angle, why a single GPU benchmark misleads procurement, covers what an equal-footing comparison has to hold constant. The operational point here is narrower: if you can’t write down the tuple, you can’t compare two accelerators on anything other than marketing.

One further constraint on reading such comparisons: comparability holds within a release name and not across release names. The catalogue and the thresholds change between releases, so a number carried forward from an earlier release into a later table stops being a like-for-like figure even when the device is identical — and the tuple beneath it will have moved too, because drivers and vendored libraries move on their own schedule.

Closing

The toolkit version PyTorch was compiled against appears in torch.version.cuda, which differs from your installed system CUDA toolkit. The relationship between this number, the system driver, and the GPU’s compute capability determines which kernels execute and what performance the workload achieves. Benchmark reproducibility requires all three numbers — plus the wheel source and the cuDNN version — to be disclosed; a report that names only torch.version.cuda is under-specifying the AI Executor in ways that block reproduction on any other host.

LynxBenchAI treats the (PyTorch version, wheel source, torch.version.cuda, driver, compute capability, cuDNN) tuple as part of the AI Executor specification — alongside the GPU model — because that tuple is what determines whether a benchmark result transfers to any other host that nominally has the same PyTorch and GPU. Which exact tuple was the cited benchmark run against, and does any host you plan to deploy on match it kernel-for-kernel?

Frequently Asked Questions

When evaluating a CUDA alternative such as ROCm, which parts of the migration cost live in framework integration versus in your own kernels?

On torch version cuda Explained, the short answer is this. The framework-integration half is bounded and largely someone else’s work: a ROCm build of PyTorch exists, and models that stay inside standard operators often run with no source change. The unbounded half is anything you wrote against CUDA directly — custom kernels, CUDA-specific launch and stream logic, TensorRT or NCCL-shaped assumptions in the serving path — plus every performance property you tuned against NVIDIA’s kernel selection. In our experience the second half is where migration estimates go wrong, because it is invisible in a framework-level port test.

What does AMD’s HIP translation layer change about switching cost — does source-level CUDA compatibility remove the ecosystem problem or just relocate it?

HIP genuinely lowers the porting cost of source that uses ordinary CUDA constructs, so the “rewrite everything” framing overstates the problem. What it does not translate is ecosystem depth: library-for-library equivalence, kernel coverage for the exact operations your model uses, profiler and debugging maturity, and the accumulated tuning knowledge around them. The cost moves from rewriting code to re-establishing performance and confidence — which is why a successful compile is a milestone, not an answer.

Why does capping the optimization budget identically across ecosystems matter when comparing a mature ecosystem against a younger one?

Without a cap, the mature ecosystem wins on the effort available to it rather than on what it executes — hand-tuning depth gets smuggled into the number as if it were hardware or runtime behaviour. LynxBenchAI prepares models once, before any target device is known, and where two vendors’ kernels disagree about what is permissible the stricter constraint applies to all of them. Ecosystem depth still shows up in the result, but as execution rather than as effort spent on one side of the comparison.

Is there a runtime probe that recovers which wheel source PyTorch came from?

Not reliably. torch.__version__ and torch.version.cuda describe the build, and torch.backends.cudnn.version() gives you the vendored cuDNN, but none of them identify whether the wheel came from PyPI, NVIDIA NGC, conda-forge, or a distribution package. This is why the wheel source has to be recorded at install time as part of the disclosure, not inferred afterwards from the running process.

Matching PyTorch builds to your GPU driver

Bundled CUDA libraries in wheel binaries can silently disable optimizations or trigger import crashes when the embedded runtime conflicts with your driver. If any of those differ, are you still looking at a comparison, or two unrelated observations?