“Just halving the precision” doesn’t describe what FP16 does

Why does halving bit-width from FP32 to FP16 produce more than a proportional loss in dynamic range? Because exponent bits govern scale, and five bits cannot cover the span that eight bits afford. Going from 32-bit to 16-bit floating-point did not uniformly halve the format’s capabilities. The reduction was unbalanced: FP16 cuts the exponent budget more than half (from 8 bits to 5) and cuts the mantissa less than half (from 23 to 10). The consequence is that the property that’s most reduced — dynamic range — is the property whose loss matters most for typical training workloads. This is why standalone FP16 training is unstable for many workloads where FP32 is comfortable, and why mixed-precision schemes exist to recover the lost dynamic range without paying the FP32 storage cost.

Understanding the FP16 format precisely — what its bit allocation provides and what mixed-precision schemes pair it with — is the prerequisite for reading FP16-related performance claims, because “FP16” by itself does not describe a numerical regime; the surrounding scheme does. We see this confusion regularly when teams compare results from PyTorch’s torch.cuda.amp against a paper that reported “FP16 throughput” without specifying whether loss scaling and FP32 accumulation were enabled.

What is the IEEE-754 half-precision format?

IEEE-754 half-precision allocates 16 bits as follows:

1 sign bit · 5 exponent bits · 10 mantissa bits

The 5 exponent bits encode powers of two between roughly 2⁻¹⁴ and 2¹⁵. This is the format’s dynamic range — and it is much narrower than FP32’s range of roughly 2⁻¹²⁶ to 2¹²⁷. In practical terms, FP16 cannot represent very small values (they underflow to zero) or very large values (they overflow to infinity) in regions where FP32 has no difficulty.

The 10 mantissa bits encode the significant digits. This gives FP16 about three decimal digits of precision for distinguishing values of similar magnitude — substantially less than FP32’s seven decimal digits, but still sufficient for many computations where the relative ordering matters more than the fine numerical differences.

The reduction relative to FP32 is therefore not symmetric:

Property	FP32	FP16	What’s reduced
Sign bit	1	1	Unchanged
Exponent bits	8	5	Reduced from 256 to 32 representable powers of two
Mantissa bits	23	10	Reduced from ~7 decimal digits to ~3
Dynamic range	~2⁻¹²⁶ to ~2¹²⁷	~2⁻¹⁴ to ~2¹⁵	Massive narrowing
Mantissa precision	~7 decimal digits	~3 decimal digits	Moderate reduction
Bit total	32	16	Halved

The “halving” is in the bit total. The dynamic range is reduced far more than half — by about ten orders of magnitude on each end. That asymmetry is what governs FP16’s behavior on real workloads.

Why dynamic range is the limiting property

Deep neural network training generates values that vary by many orders of magnitude as signals propagate forward and error flows backward. Activations in a deep network can range from very large (in early layers or in unnormalized regions) to very small (in late layers or after compression operations). Gradients, particularly on parameters far from the loss, can be many orders of magnitude smaller than the activations they correspond to. Loss values themselves can shrink as training converges.

FP32’s exponent budget covers this span comfortably. FP16’s does not. Two failure modes appear when an FP16 computation encounters values outside its representable range:

Underflow. Small values (gradients, late-layer activations after compression) below 2⁻¹⁴ round to zero. Once a gradient becomes zero, the corresponding parameter does not update at all, and the training trajectory loses information that the FP32 version would have retained. Underflow is silent — there is no warning that a contribution was lost.

Overflow. Large values (early-layer activations, unnormalized intermediate computations) above 2¹⁵ saturate to infinity. Once a value becomes infinity, downstream computations that reference it produce NaNs, and the training trajectory destabilizes catastrophically.

The mantissa reduction is the secondary issue, not the primary one. A workload that survives the dynamic-range reduction can usually tolerate the mantissa reduction with negligible accuracy impact. A workload that fails on FP16 fails because of range, not precision. This is an observed pattern across the engagements where we have profiled mixed-precision rollouts — the diagnostic that resolves the failure is almost always a histogram of gradient magnitudes, not a check on mantissa rounding.

This is why a workload runs comfortably in BF16 (1+8+7 — same exponent as FP32) but breaks in FP16 (1+5+10 — different exponent): the dynamic range, not the bit total, is what changed.

What mixed-precision schemes are doing

Mixed-precision training pairs FP16 computation with FP32 accumulation and adds two specific mechanisms designed to recover the dynamic range FP16 cannot provide alone:

FP32 master weights. The model parameters are stored at FP32. The forward and backward passes are computed at FP16 (using the FP32 weights cast to FP16 for the computation), but the parameter updates accumulate against the FP32 master copy. This protects the long-term accumulation from FP16’s mantissa-precision limit.

FP32 accumulation in matrix-multiply. The intermediate accumulation in matrix-multiply operations runs at FP32 even when the input operands are FP16. NVIDIA’s tensor cores are designed for this pattern: FP16 inputs, FP32 accumulator, FP16 output. This protects the within-operation accumulation from precision loss, and it is the reason a CUDA kernel issued through cuBLAS or a fused TensorRT engine can deliver FP16 throughput without the accuracy collapse that a naive cast would produce.

Loss scaling. Before backward pass, the loss is multiplied by a large scalar (typically a power of two). All gradients computed during backward pass are scaled by the same factor, which shifts them into the FP16 representable range and prevents underflow. After backward pass, the scaled gradients are unscaled (by dividing by the same scalar) before the parameter update is applied. This recovers the dynamic range FP16 lacks for very small gradients.

A workload running under mixed precision is not running at FP16. It is running at a hybrid (FP16 + FP32 + loss-scaling) regime that has different numerical properties than either FP16 or FP32 alone, and the reported performance and accuracy of the workload are properties of the hybrid regime, not of FP16. The broader logic for why this works at all — that not every operation in a model carries the same numerical sensitivity — is the subject of mixed-precision mechanics, and is the reason these mechanisms compose rather than fight each other.

This is why “FP16 result” without specifying the surrounding scheme under-specifies what the result represents. A pure-FP16 result (computation and storage entirely in FP16) is one regime; a mixed-precision result (FP16 compute, FP32 accumulation and master weights, with loss scaling) is a different regime; and the two can produce substantially different accuracy and stability profiles on the same model.

What this means for FP16 benchmark interpretation

Benchmarks claiming “FP16 performance” remain ambiguous until the report specifies accumulator precision, scaling techniques, and whether tensor cores participated. The interpretable forms are:

• Pure FP16: computation and storage at FP16 throughout, no loss scaling, no FP32 accumulation. Used in some inference scenarios where the workload tolerates the regime.
• Mixed precision (FP16 + FP32): FP16 compute and storage with FP32 accumulation in tensor-core operations and FP32 master weights for parameter updates, plus loss scaling for gradients. The standard training-side use of FP16.
• Framework-managed automatic mixed precision (AMP): the framework’s automatic mixed-precision implementation — PyTorch’s torch.cuda.amp.autocast, TensorFlow’s tf.keras.mixed_precision, or JAX’s jax.default_matmul_precision — which makes per-operation decisions about which ops to run at which precision based on numerical-stability heuristics maintained by the framework authors.

These are different regimes with different performance and accuracy profiles. A benchmark report that names “FP16” without specifying which regime is asking the reader to assume one — usually mixed precision, but the assumption is invisible — and the result’s generalization to a different regime is the reader’s problem.

The released LynxBenchAI benchmark deliberately takes the opposite approach: it holds one precision fixed per test case and reports each level separately, so no run in a 26Q3 release measures a mixed-precision configuration at all. That separation is the point. It leaves clean single-precision endpoints — an FP16 number and an FP32 number that were never blended — against which a proposed mixed design can be argued. Those endpoints belong to an AI Executor, meaning device, backend, driver, framework and runtime together, so a tolerance you observe on one stack is not a property of the silicon underneath it, and per-precision endpoints from two different release names are not comparable to each other. What pip install lynxbench-ai gives you is the baseline your own composition has to beat, not a verdict on the composition.

FP8 sits at the far end of the same argument. Where FP16 and BF16 differ in how they split a 16-bit budget between exponent and mantissa, an 8-bit float has too little of both to carry a tensor unassisted, so it is used with per-tensor or per-block scaling factors and with accumulation held at a wider format. Adopting it assumes the workload tolerates a coarser mantissa and that the scaling machinery tracks the tensor’s range closely enough to avoid the underflow FP16 already flirts with. That is a strictly stronger tolerance assumption than FP16 or BF16 require, and it is workload-specific rather than a property of the hardware that supports the format.

It is also worth being explicit that mixed precision is not the same thing as quantization. Quantization replaces the floating-point format with a lower-bit integer representation (typically INT8 or INT4), introduces a separate scale and zero-point, and changes how values map to the number line. Mixed precision keeps floating-point semantics throughout and only varies which floating-point format holds which tensor at which moment. Conflating the two leads teams to expect quantization’s calibration workflow when they adopt mixed precision, or to expect mixed precision’s drop-in characteristics when they adopt quantization. Neither expectation survives contact with the workload.

Residual risks worth evaluating

Even when a framework supports mixed precision automatically, some risks remain that a team adopting it in production should evaluate before treating the regime as a free win:

• Operations the framework’s AMP heuristics get wrong for your workload. Layer-norm, softmax-with-large-logits, and reduction-heavy custom ops are common offenders. The framework’s allow-list is a default, not a guarantee.
• Loss-scaling instability at long training runs. Dynamic loss scaling can settle into an oscillation when gradient distributions drift across training phases. A fixed-scale fallback is sometimes more stable.
• Inference paths that bypass the training-time AMP context. A model trained with AMP can be exported in a way that loses the precision annotations and runs operations at a precision the training run never validated.
• Subtle accuracy drift that passes unit tests but degrades downstream metrics. End-to-end evaluation on a representative slice — not just loss curves — is the only reliable check.

None of these makes mixed precision a bad choice. They make it a choice that benefits from explicit verification rather than blanket trust in the framework default.

The framing that helps

Reducing the exponent field from eight bits to five compresses representable magnitude far more severely than the 50 percent reduction in total bits would suggest. Standalone FP16 is unstable for many workloads because of range, not because of mantissa precision. Mixed-precision schemes pair FP16 compute with FP32 accumulation, FP32 master weights, and loss scaling specifically to recover the range FP16 cannot provide alone. A “half-precision” performance result without disclosure of the surrounding scheme is under-specified.

LynxBenchAI treats the precision regime — the specific combination of computation precision, accumulation precision, master-weight precision, and loss-scaling configuration — as part of the AI Executor specification, because “FP16” alone is not a regime; the surrounding scheme is, and a benchmark that holds the regime explicit is the one that informs a deployment decision about precision selection. For the FP16 throughput number you are about to act on, does the report name the surrounding scheme — computation precision, accumulation precision, master-weight precision, and loss-scaling configuration — that determines achievable quality at this precision, or is the bare format label being asked to stand in for the precision regime your workload actually requires?

Frequently Asked Questions

How does FP8 fit into the mixed precision picture, and what does it assume that FP16 and BF16 do not?

An 8-bit float has too little exponent and too little mantissa to carry a tensor unassisted, so it is always paired with per-tensor or per-block scaling factors and with accumulation at a wider format. That means adopting FP8 assumes both a coarser mantissa is acceptable and that the scaling machinery tracks each tensor’s range closely enough to avoid the underflow FP16 already flirts with. It is a strictly stronger tolerance assumption than FP16 or BF16 need, and whether it holds is a property of the workload, not of the hardware that supports the format.

What does automatic mixed precision decide for you, and what still has to be decided by hand?

Framework AMP — PyTorch’s torch.cuda.amp.autocast, TensorFlow’s tf.keras.mixed_precision, JAX’s matmul-precision settings — decides which operations run at reduced precision using an allow-list of numerically safe ops maintained by the framework authors, and it typically manages dynamic loss scaling for you. What it does not decide is whether its allow-list matches your model: layer-norm, softmax with large logits, and reduction-heavy custom ops are the usual offenders. Export paths, fixed-versus-dynamic loss-scale policy, and end-to-end accuracy verification on a representative slice remain manual.

Why does an inference-only FP16 setup usually not need loss scaling?

Loss scaling exists to keep small gradients from underflowing FP16’s lower bound during the backward pass. Inference has no backward pass and no parameter update, so the value population is activations rather than gradients, and activations rarely sit near 2⁻¹⁴. The remaining inference risk is overflow on large early-layer activations, which loss scaling would not fix anyway — it targets underflow specifically.

Can a LynxBenchAI FP16 result be read as a mixed-precision measurement?

No. Each test case in a release holds one precision fixed and reports it separately, so no run measures a mixed-precision configuration. What you get is a clean single-precision endpoint on a specific AI Executor — device, backend, driver, framework and runtime together — that a proposed mixed design can be argued against. Endpoints from two different release names are not comparable to each other, and the tolerance observed on one stack is not a property of the silicon underneath it.

Anatomy of the 16-bit float

IEEE 754 half-precision allocates its sixteen bits as follows: one for sign, five for exponent, ten for mantissa—a design favoring compute density over numeric fidelity. Is that executor close enough to yours for the result to mean anything?