Two different compute units, two different jobs

NVIDIA GPUs contain two distinct types of processing units that serve fundamentally different purposes:

CUDA cores are general-purpose parallel processors. Each executes one floating-point or integer operation per clock cycle. A GPU with 16,384 CUDA cores can process 16,384 independent scalar operations simultaneously. They handle everything: graphics rendering, scientific simulation, data processing, and the non-matrix-multiply portions of AI workloads.

Tensor cores are specialised matrix-multiply-accumulate (MMA) units. Each tensor core processes an entire small matrix multiplication (typically 4×4 or larger, depending on generation, as representative examples) in a single operation. They exist specifically to accelerate the dense linear algebra that dominates neural network computation.

CUDA cores handle general-purpose parallel computation; tensor cores accelerate matrix-multiply-accumulate operations — AI inference throughput depends primarily on tensor core utilisation, not CUDA core count.

Why tensor cores dominate AI performance

Matrix multiplication dominates the computational profile of neural network inference. A transformer model processing a single token performs thousands of matrix-vector multiplications (in attention layers and feed-forward layers). Each of these maps directly to tensor core operations.

The performance difference is substantial:

GPU	CUDA cores	Tensor cores	FP16 CUDA core TFLOPS	FP16 Tensor core TFLOPS	Ratio
A100	6,912	432	19.5	312	16×
H100	16,896	528	51	989	19×
L40S	18,176	568	36.7	366	10×

Tensor cores deliver 10–19× more throughput for AI workloads than CUDA cores on the same chip (benchmark figures from published NVIDIA datasheets for A100, H100, and L40S). A GPU with more CUDA cores but fewer or older-generation tensor cores will perform worse on AI inference than a GPU with fewer CUDA cores but more capable tensor cores.

Tensor core generations and precision support

Precision format support—FP8, BF16, INT8—varies by tensor core generation and outweighs raw core count in determining AI throughput. Each generation adds support for additional data formats:

Generation	First appeared in	Supported precisions
1st gen	V100 (Volta)	FP16
2nd gen	A100 (Ampere)	FP16, BF16, TF32, INT8, INT4
3rd gen	H100 (Hopper)	FP16, BF16, TF32, FP8, INT8
4th gen	B200 (Blackwell)	FP16, BF16, TF32, FP8, FP4, INT8

This matters because quantised models (INT8, FP8) offer 2× or more throughput improvement over FP16 — but only on hardware with tensor cores that support those formats. Running an INT8-quantised model on V100 tensor cores gains nothing because those tensor cores only accelerate FP16.

When do CUDA cores still matter for AI?

General-purpose CUDA cores still contribute meaningfully to AI workloads. They handle:

• Non-linear activations (ReLU, GELU, SiLU) — element-wise operations that don’t map to matrix multiply
• Normalization layers (LayerNorm, RMSNorm) — reductions and element-wise computations
• Attention score softmax — exponential and normalization operations
• Tokenization and pre/post-processing — data preparation before and after model execution
• Custom kernels — any operation that doesn’t decompose into standard matrix multiplication

For models where these non-MMA operations constitute a significant fraction of execution time (small batch inference, models with many activation-heavy layers), CUDA core performance affects total throughput. But for the dominant case — large matrix multiplications in transformer attention and FFN layers — tensor cores determine performance.

Implications for hardware selection

When evaluating GPUs for AI workloads, the relevant specifications are:

1. Tensor core generation — determines which precision formats are hardware-accelerated
2. Tensor core count — determines peak matrix-multiply throughput
3. Memory bandwidth — determines how fast data reaches the tensor cores (often the actual bottleneck)
4. CUDA core count — determines throughput of non-MMA operations (secondary factor)

Understanding this hierarchy explains why the CUDA ecosystem’s influence on hardware selection extends beyond software compatibility — tensor core capabilities and their framework support determine what optimisations your workload can access. The unit exists in silicon; whether your stack can reach it is a software question, and that is where the switching-cost argument actually lives.

Marketing materials emphasise CUDA core counts because they are large numbers (16,384 sounds impressive). Tensor core counts are smaller (528) but more relevant. Memory bandwidth (3.35 TB/s on H100 HBM3) is often the actual performance determinant for inference workloads, because models are frequently memory-bandwidth-bound rather than compute-bound at typical batch sizes.

LynxBenchAI treats the breakdown of work across tensor units and general-purpose cores as part of the AI Executor disclosure, because the headline core count answers a question that few AI workloads actually ask. Comparability there is scoped to a single release name — the catalogue changes between releases, so a core-count-versus-throughput reading taken from one release cannot be carried into another. When evaluating any CUDA-core or tensor-core marketing comparison, ask: does the workload that determines the buyer’s outcome actually run on the units being counted within the kernel coverage your stack provides, or does the comparison flatter a metric the deployment will never exercise?

Frequently Asked Questions

When evaluating a CUDA alternative such as ROCm, which parts of the migration cost live in framework integration versus in your own kernels?

For CUDA Cores vs Tensor, the question is straightforward. Framework integration is the part that has usually been done for you: PyTorch carries a ROCm backend, so the standard operators that decompose into matrix multiply and land on matrix units generally come across. Your own kernels are the part that does not — anything hand-written against CUDA intrinsics, warp-level primitives, or a specific tensor core MMA shape has to be rewritten and re-validated against the new hardware’s equivalent units. The split matters because the first cost is amortised across the whole ecosystem and the second is yours alone.

What does AMD’s HIP translation layer change about switching cost — does source-level CUDA compatibility remove the ecosystem problem or just relocate it?

It relocates it. HIP addresses the syntactic layer, so a large share of CUDA source can be translated mechanically, but translation says nothing about whether the resulting kernel reaches the target’s matrix units at the precision and tile shape the original was tuned for. The residual cost moves from porting to performance recovery — and performance recovery is the expensive half.

Why does capping the optimization budget identically across ecosystems matter when measuring a mature ecosystem against a younger one?

Because otherwise the number you get back is a measurement of tuning effort, not of hardware or software capability. When models are prepared once before the target device is known and the stricter of two vendors’ kernel constraints applies to both, ecosystem depth still shows up — but as execution quality rather than as hours spent hand-tuning one side. That is the only condition under which “mature ecosystem wins” is an observation rather than an artefact of who got more attention.

Does a high tensor core count guarantee the precision formats a quantised model needs?

No — count and generation are independent axes. A GPU can carry many tensor cores of an older generation that accelerate only FP16, in which case an FP8 or INT8 model gains nothing from the quantisation and may run slower than the FP16 baseline after dequantisation overhead. Check generation first, then count.

Choosing Between CUDA and Tensor Architectures

Optimal throughput depends on matching your workload’s matrix density and precision needs to the appropriate core architecture. Is that executor close enough to yours for the result to mean anything?