“Just halving the precision” doesn’t describe what FP16 does The intuition that FP16 is FP32 with half the bits is correct as a bit-count statement and misleading as a numerical-properties statement. Going from 32-bit to 16-bit floating-point did not uniformly halve the format’s capabilities. The reduction was unbalanced: FP16 cuts the exponent budget more than half (from 8 bits to 5) and cuts the mantissa less than half (from 23 to 10). The consequence is that the property that’s most reduced — dynamic range — is the property whose loss matters most for typical training workloads. This is why standalone FP16 training is unstable for many workloads where FP32 is comfortable, and why mixed-precision schemes exist to recover the lost dynamic range without paying the FP32 storage cost. Understanding the FP16 format precisely — what its bit allocation provides and what mixed-precision schemes pair it with — is the prerequisite for reading FP16-related performance claims, because “FP16” by itself does not describe a numerical regime; the surrounding scheme does. We see this confusion regularly when teams compare results from PyTorch’s torch.cuda.amp against a paper that reported “FP16 throughput” without specifying whether loss scaling and FP32 accumulation were enabled. What is the IEEE-754 half-precision format? IEEE-754 half-precision allocates 16 bits as follows: 1 sign bit · 5 exponent bits · 10 mantissa bits The 5 exponent bits encode powers of two between roughly 2⁻¹⁴ and 2¹⁵. This is the format’s dynamic range — and it is much narrower than FP32’s range of roughly 2⁻¹²⁶ to 2¹²⁷. In practical terms, FP16 cannot represent very small values (they underflow to zero) or very large values (they overflow to infinity) in regions where FP32 has no difficulty. The 10 mantissa bits encode the significant digits. This gives FP16 about three decimal digits of precision for distinguishing values of similar magnitude — substantially less than FP32’s seven decimal digits, but still sufficient for many computations where the relative ordering matters more than the fine numerical differences. The reduction relative to FP32 is therefore not symmetric: Property FP32 FP16 What’s reduced Sign bit 1 1 Unchanged Exponent bits 8 5 Reduced from 256 to 32 representable powers of two Mantissa bits 23 10 Reduced from ~7 decimal digits to ~3 Dynamic range ~2⁻¹²⁶ to ~2¹²⁷ ~2⁻¹⁴ to ~2¹⁵ Massive narrowing Mantissa precision ~7 decimal digits ~3 decimal digits Moderate reduction Bit total 32 16 Halved The “halving” is in the bit total. The dynamic range is reduced far more than half — by about ten orders of magnitude on each end. That asymmetry is what governs FP16’s behavior on real workloads. Why dynamic range is the limiting property For most training workloads, the values that arise during forward pass and backward pass span many orders of magnitude. Activations in a deep network can range from very large (in early layers or in unnormalized regions) to very small (in late layers or after compression operations). Gradients, particularly on parameters far from the loss, can be many orders of magnitude smaller than the activations they correspond to. Loss values themselves can shrink as training converges. FP32’s exponent budget covers this span comfortably. FP16’s does not. Two failure modes appear when an FP16 computation encounters values outside its representable range: Underflow. Small values (gradients, late-layer activations after compression) below 2⁻¹⁴ round to zero. Once a gradient becomes zero, the corresponding parameter does not update at all, and the training trajectory loses information that the FP32 version would have retained. Underflow is silent — there is no warning that a contribution was lost. Overflow. Large values (early-layer activations, unnormalized intermediate computations) above 2¹⁵ saturate to infinity. Once a value becomes infinity, downstream computations that reference it produce NaNs, and the training trajectory destabilizes catastrophically. The mantissa reduction is the secondary issue, not the primary one. A workload that survives the dynamic-range reduction can usually tolerate the mantissa reduction with negligible accuracy impact. A workload that fails on FP16 fails because of range, not precision. This is an observed pattern across the engagements where we have profiled mixed-precision rollouts — the diagnostic that resolves the failure is almost always a histogram of gradient magnitudes, not a check on mantissa rounding. This is why a workload runs comfortably in BF16 (1+8+7 — same exponent as FP32) but breaks in FP16 (1+5+10 — different exponent): the dynamic range, not the bit total, is what changed. What mixed-precision schemes are doing Mixed-precision training pairs FP16 computation with FP32 accumulation and adds two specific mechanisms designed to recover the dynamic range FP16 cannot provide alone: FP32 master weights. The model parameters are stored at FP32. The forward and backward passes are computed at FP16 (using the FP32 weights cast to FP16 for the computation), but the parameter updates accumulate against the FP32 master copy. This protects the long-term accumulation from FP16’s mantissa-precision limit. FP32 accumulation in matrix-multiply. The intermediate accumulation in matrix-multiply operations runs at FP32 even when the input operands are FP16. NVIDIA’s tensor cores are designed for this pattern: FP16 inputs, FP32 accumulator, FP16 output. This protects the within-operation accumulation from precision loss, and it is the reason a CUDA kernel issued through cuBLAS or a fused TensorRT engine can deliver FP16 throughput without the accuracy collapse that a naive cast would produce. Loss scaling. Before backward pass, the loss is multiplied by a large scalar (typically a power of two). All gradients computed during backward pass are scaled by the same factor, which shifts them into the FP16 representable range and prevents underflow. After backward pass, the scaled gradients are unscaled (by dividing by the same scalar) before the parameter update is applied. This recovers the dynamic range FP16 lacks for very small gradients. A workload running under mixed precision is not running at FP16. It is running at a hybrid (FP16 + FP32 + loss-scaling) regime that has different numerical properties than either FP16 or FP32 alone, and the reported performance and accuracy of the workload are properties of the hybrid regime, not of FP16. The broader logic for why this works at all — that not every operation in a model carries the same numerical sensitivity — is the subject of mixed-precision mechanics, and is the reason these mechanisms compose rather than fight each other. This is why “FP16 result” without specifying the surrounding scheme under-specifies what the result represents. A pure-FP16 result (computation and storage entirely in FP16) is one regime; a mixed-precision result (FP16 compute, FP32 accumulation and master weights, with loss scaling) is a different regime; and the two can produce substantially different accuracy and stability profiles on the same model. What this means for FP16 benchmark interpretation A benchmark that reports performance “at FP16” must disclose what it actually measured. The interpretable forms are: Pure FP16: computation and storage at FP16 throughout, no loss scaling, no FP32 accumulation. Used in some inference scenarios where the workload tolerates the regime. Mixed precision (FP16 + FP32): FP16 compute and storage with FP32 accumulation in tensor-core operations and FP32 master weights for parameter updates, plus loss scaling for gradients. The standard training-side use of FP16. Framework-managed automatic mixed precision (AMP): the framework’s automatic mixed-precision implementation — PyTorch’s torch.cuda.amp.autocast, TensorFlow’s tf.keras.mixed_precision, or JAX’s jax.default_matmul_precision — which makes per-operation decisions about which ops to run at which precision based on numerical-stability heuristics maintained by the framework authors. These are different regimes with different performance and accuracy profiles. A benchmark report that names “FP16” without specifying which regime is asking the reader to assume one — usually mixed precision, but the assumption is invisible — and the result’s generalization to a different regime is the reader’s problem. It is also worth being explicit that mixed precision is not the same thing as quantization. Quantization replaces the floating-point format with a lower-bit integer representation (typically INT8 or INT4), introduces a separate scale and zero-point, and changes how values map to the number line. Mixed precision keeps floating-point semantics throughout and only varies which floating-point format holds which tensor at which moment. Conflating the two leads teams to expect quantization’s calibration workflow when they adopt mixed precision, or to expect mixed precision’s drop-in characteristics when they adopt quantization. Neither expectation survives contact with the workload. Residual risks worth evaluating Even when a framework supports mixed precision automatically, some risks remain that a team adopting it in production should evaluate before treating the regime as a free win: Operations the framework’s AMP heuristics get wrong for your workload. Layer-norm, softmax-with-large-logits, and reduction-heavy custom ops are common offenders. The framework’s allow-list is a default, not a guarantee. Loss-scaling instability at long training runs. Dynamic loss scaling can settle into an oscillation when gradient distributions drift across training phases. A fixed-scale fallback is sometimes more stable. Inference paths that bypass the training-time AMP context. A model trained with AMP can be exported in a way that loses the precision annotations and runs operations at a precision the training run never validated. Subtle accuracy drift that passes unit tests but degrades downstream metrics. End-to-end evaluation on a representative slice — not just loss curves — is the only reliable check. None of these makes mixed precision a bad choice. They make it a choice that benefits from explicit verification rather than blanket trust in the framework default. ## The framing that helps IEEE-754 half-precision is a 1+5+10 bit allocation whose exponent reduction relative to FP32 narrows dynamic range far more than the bit-total halving suggests. Standalone FP16 is unstable for many workloads because of range, not because of mantissa precision. Mixed-precision schemes pair FP16 compute with FP32 accumulation, FP32 master weights, and loss scaling specifically to recover the range FP16 cannot provide alone. A “half-precision” performance result without disclosure of the surrounding scheme is under-specified. LynxBench AI treats the precision regime — the specific combination of computation precision, accumulation precision, master-weight precision, and loss-scaling configuration — as part of the AI Executor specification, because “FP16” alone is not a regime; the surrounding scheme is, and a benchmark that holds the regime explicit is the one that informs a deployment decision about precision selection. For the FP16 throughput number you are about to act on, does the report name the surrounding scheme — computation precision, accumulation precision, master-weight precision, and loss-scaling configuration — that determines achievable quality at this precision, or is the bare format label being asked to stand in for the precision regime your workload actually requires?
