PyTorch and CUDA are not competing options. PyTorch is a tensor and autograd framework; CUDA is NVIDIA’s compute API and toolkit that PyTorch’s GPU backend is built on top of. When you run a model on an NVIDIA card in PyTorch, you are using both — the question is never which one to pick, it is which layer a given problem lives in.
That distinction matters because it decides where optimisation effort goes. A model that is slow because of dataloader stalls or graph breaks will not get faster if you rewrite an attention kernel. A model that is bound by occupancy, memory movement, or precision choices will not get faster if you tune batch size and worker counts.
Which layer owns what?
| Concern | Layer | Where you fix it |
|---|---|---|
| Autograd, module composition, optimiser step | PyTorch | Model and training code |
| Dataloader throughput, host-to-device copies, graph breaks | PyTorch / runtime | DataLoader, pinned memory, torch.compile |
| Operator dispatch, kernel selection, memory allocator behaviour | PyTorch ↔ CUDA boundary | Backend flags, cuDNN/cuBLAS settings, allocator config |
| Kernel occupancy, shared-memory use, warp divergence | CUDA | Custom kernels, Triton, TensorRT |
| Numerical precision (FP32 / TF32 / BF16 / FP8) | Both | Autocast policy above, kernel support below |
Profile first; every optimization hypothesis fails without measurement. If the GPU timeline shows long idle gaps between kernel launches, the bottleneck is above CUDA. If the timeline is dense but individual kernels are slow relative to the memory bandwidth they consume, the bottleneck is below PyTorch. We use that split as the first cut in a GPU performance review because it removes whole classes of speculative rewrite before anyone touches a kernel.
Does PyTorch tie you to NVIDIA?
Custom kernels help in specific, measurable circumstances—not everywhere developers expect. Application code written against torch.Tensor is largely portable — ROCm and XPU backends run the same model code. The lock-in accumulates in what sits underneath: custom CUDA kernels, TensorRT engines, NCCL-specific collective assumptions, and any operator whose fast path only exists on one vendor’s stack. Framework choice, not application code, is often what actually pins a project to a vendor, which is why an honest inventory of “which parts of our stack are NVIDIA-dependent” is worth keeping before any hardware-diversity decision is costed.
torch.compile shifts the boundary rather than removing it. Generated kernels mean fewer hand-written CUDA files, but the compiled path still has to be reasoned about at the kernel level when it underperforms — you inspect what was generated instead of what you wrote.
When a custom kernel is actually justified
Justify a CUDA kernel when fusion opportunities exist that no operator composition captures, profiling confirms that bottleneck owns your step time, and the tensor shape regime will remain stable. Outside those three conditions, composing existing operators and letting the compiler fuse them is the cheaper bet.
We go deeper into the vendor-lock consequences of that choice, and how it interacts with the CUDA versus OpenCL versus SYCL question, in our GPU acceleration engineering work.
The open question for most teams is not which API to standardise on. It is whether anyone has written down which layer their last three performance wins came from.
