“Default” is a design decision that got habituated

Throughout the deep-learning era, IEEE-754 single-precision floating-point—FP32—has served as the workhorse format for the majority of AI training runs. The format is so consistently the default that it appears in tutorials and framework configurations as if it were a property of “training” itself rather than a specific design choice with specific assumptions. It is a design choice. The reasons FP32 became the default are concrete properties of its bit allocation, and the reasons to consider trading away from it are concrete properties of the workload’s tolerance for the trade.

Understanding what FP32 actually represents — and what its structure provides — is the prerequisite for reasoning about whether a given workload can move to a lower-precision format and what it will give up in doing so. We see teams treat the FP32 default as a moral commitment, when in our experience it is better understood as a conservative budget that happens to match most training workloads.

What is the IEEE-754 single-precision format?

IEEE-754 single-precision allocates 32 bits as follows:

1 sign bit · 8 exponent bits · 23 mantissa bits

The sign bit determines positive or negative. The 8 exponent bits encode a power of two between roughly 2⁻¹²⁶ and 2¹²⁷, which is the format’s dynamic range — how small or large a number it can represent at all. The 23 mantissa bits encode the significant digits of the value, which is the format’s mantissa precision — how finely it can distinguish two numbers of similar magnitude.

These two properties — dynamic range and mantissa precision — are the axes along which floating-point formats differ from each other. Every choice of bit allocation is a trade between them, and every change to a different format is a trade in those terms.

For FP32, the combined budget is large enough that ordinary numerical work — physical simulation, signal processing, scientific computation, and gradient-based optimization — does not encounter representation-induced instability under normal conditions. Numbers do not silently overflow or underflow in regions a workload is likely to traverse. Two values of similar magnitude can be reliably distinguished. Accumulation of many small contributions stays within the format’s representable range without saturation.

That budget is the property the AI-training default was built around.

Why FP32 became the AI training default

Gradient descent translates, in numerical terms, into a long sequence of tiny updates that must accumulate reliably without catastrophic rounding drift. The optimizer updates each parameter by accumulating contributions across many samples, many batches, and many epochs. Three properties of this accumulation matter for format selection:

• Gradients can be small. Particularly in deep networks, gradients on parameters far from the loss can be many orders of magnitude smaller than activations. The format must represent values across this dynamic range without underflow.
• Updates accumulate. Many small contributions sum to the parameter update. The format must maintain enough mantissa precision that the small contributions are not lost in the rounding when added to a larger running sum.
• Stability matters across iterations. A representation-induced error in one iteration can amplify in the next. The format must be conservative enough that ordinary training trajectories do not encounter pathological numerical behavior.

FP32’s 8-bit exponent and 23-bit mantissa happen to satisfy all three for the typical range of training workloads. The dynamic range comfortably covers the gradient/activation ratio in most networks; the mantissa precision is enough that small-update accumulation remains numerically meaningful. The format was not designed for AI training — it predates deep learning by decades — but its allocations match what training workloads need closely enough that it became the unthinking default in PyTorch, TensorFlow, and JAX configurations alike.

The corollary is that the default is not a moral commitment. It is a budget that happens to be conservative for most workloads. Different workloads have different tolerances, and a workload whose properties differ from the typical training profile may be able to use a smaller budget without losing what FP32 was protecting.

What trading away from FP32 actually trades

Dropping below FP32 forces a tradeoff: you sacrifice either the span of representable magnitudes or the number of significant digits retained in each mantissa. The two reductions have different consequences:

Reducing dynamic range (e.g. moving from 8 exponent bits to 5, as in FP16) means small values can underflow to zero and large values can overflow to infinity in regions where FP32 would still represent them. For a workload whose gradients and activations span many orders of magnitude, this is the more dangerous reduction — the failure mode is silent loss of information, and the symptom is training divergence or accuracy collapse that doesn’t trace cleanly to a single computation. Loss-scaling tricks in mixed-precision recipes exist precisely because FP16’s narrower exponent doesn’t reach the small gradients the optimizer needs to see.

Reducing mantissa precision (e.g. moving from 23 mantissa bits to 7, as in BF16) means values close in magnitude become indistinguishable. For a workload whose accuracy depends on the relative ordering of activations rather than their fine numerical differences, this reduction is often well-tolerated. Tensor Cores on Ampere and later NVIDIA hardware accumulate BF16 multiplies into FP32 registers, which preserves the wider exponent end-to-end while consuming half the memory bandwidth on the input side.

The format choices in modern AI hardware — BF16, FP16, FP8, FP4 — are different points on this trade-off space, each declaring which of the two properties the workload designer believes can be reduced. A workload that needs dynamic range tolerates BF16 (which keeps FP32’s 8-bit exponent) but not FP16 (which cuts to 5). A workload that needs mantissa precision but not dynamic range can use FP16 in a mixed-precision scheme that recovers range elsewhere. A workload tolerant on both axes can move to FP8 with appropriate accuracy validation.

Precision as a design parameter makes the broader case: precision is a design decision about which numerical properties a workload can afford to reduce, not a single-axis “lower = worse” trade.

What a precision-aware benchmark must report

When a benchmark cites FP32 performance, it signals adherence to the numerically conservative baseline that dominated pre-mixed-precision practice. The number is interpretable as a baseline. A benchmark that reports performance at a lower precision must report two coupled numbers: the throughput at the lower precision, and the accuracy of the workload’s output at that precision against the FP32 reference.

Throughput-only reporting at lower precision is structurally incomplete because the throughput gain is conditional on the accuracy holding. As an illustrative example, a workload that runs 4× faster at FP8 with no measurable accuracy loss has gained 4×. A workload that runs 4× faster at FP8 with a 5% accuracy degradation may not have gained anything operationally — the lost accuracy may exceed the value of the speedup. This is an observed pattern across the inference engagements we’ve audited, not a benchmarked rate.

The pair (throughput at precision X, accuracy at precision X relative to FP32) is therefore the minimum reporting unit for any precision-related performance claim. Reporting one without the other defers the trade-off to the reader without the data the reader needs to evaluate it.

A released 26Q3 LynxBenchAI run implements that pairing as a gate rather than a footnote. Precision is a first-class axis of the measurement: each model in the fixed catalogue is evaluated across several precisions — fp64/fp32/fp16/bf16 on the compute category, fp32/bf16 on training, fp16/int8/fp8 on inference — and each precision is reported separately instead of being folded into one headline figure. Before a per-precision result counts at all it has to clear a correctness threshold defined for that precision, so a fast wrong answer scores nothing. Aggregation then weights throughput by the memory a run actually moves, which means a lower-precision run has to earn its speed-up rather than collect credit for being cheap. And when a device cannot execute a precision, that precision is scored as zero rather than quietly skipped — the absence stays visible in the result instead of flattering the average. These figures belong to that named release and to that catalogue; they are not a prediction about the reader’s own model, and a per-precision number from one release name should not be compared against one from another.

Format-by-format trade-off matrix

The formats commonly used in modern AI hardware position themselves on the FP32 trade-off space as follows:

Format	Bits (sign+exp+mantissa)	Dynamic range vs FP32	Mantissa precision vs FP32	Typical workload fit
FP32	1+8+23	Reference	Reference	Conservative training default; numerical-stability baseline
TF32	1+8+10	Same	Reduced	NVIDIA training-throughput format on Ampere+
BF16	1+8+7	Same as FP32	Substantially reduced	Range-sensitive training; activations spanning many orders of magnitude
FP16	1+5+10	Substantially reduced	Moderately reduced	Mixed-precision training and inference where range fits
FP8 (E4M3)	1+4+3	Further reduced	Further reduced	Inference and selected training under careful calibration
FP8 (E5M2)	1+5+2	Same as FP16	Substantially reduced	Gradient or activation paths needing range over precision

Each row trades a different combination of range and precision. Choosing among them is a workload-specific decision about which numerical property the workload can afford to reduce, not a generic lower = worse ranking. The bit allocations are per NVIDIA’s and Arm’s published format specifications.

The framing that helps

Single-precision floating-point under IEEE-754 devotes one bit to sign, eight to the exponent, and twenty-three to the mantissa, yielding both broad range and fine granularity. This budget happens to be conservative for the typical training workload, which is why FP32 became the AI training default. A precision below FP32 reduces dynamic range, mantissa precision, or both, and the choice is a workload-specific design decision — not a generic “smaller is worse” trade. Precision-related benchmark claims must report throughput and accuracy as a pair.

LynxBenchAI treats performance per precision and a declared accuracy criterion as a joint output of the AI Executor specification — because precision is a design parameter that produces a (throughput, accuracy) pair, not a single number. The freely available Personal Edition (pip install lynxbench-ai) puts that per-precision picture on your own device, which turns precision from an argument about principles into a set of numbers you generated yourself. The question to ask of any FP32-to-lower-precision claim is whether the accuracy axis is named explicitly at the workload that matters, or held implicit in a way that hides the trade. On the precision claim in front of you, is the (throughput at precision X, accuracy at precision X) pair reported on the workload that determines achievable quality at this precision regime — or is the accuracy axis being carried by an FP32 baseline the deployment will never actually run?

Frequently Asked Questions

What does the 1+8+23 bit allocation in FP32 actually control?

The single sign bit sets positive or negative, the 8 exponent bits set the dynamic range (roughly 2⁻¹²⁶ to 2¹²⁷ — how small or large a value can be represented at all), and the 23 mantissa bits set mantissa precision (how finely two numbers of similar magnitude can be distinguished). Dynamic range and mantissa precision are the two axes on which every floating-point format trades against the others. Any format below FP32 reduces one or both of these.

Why does FP16 cause training divergence where BF16 does not?

FP16 cuts the exponent from 8 bits to 5, which narrows the dynamic range so far that small gradients can underflow to zero in regions FP32 would still represent. BF16 keeps FP32’s full 8-bit exponent and instead sacrifices mantissa bits, so it preserves the range that range-sensitive training needs. This is why loss-scaling tricks exist for FP16 mixed-precision recipes but are not needed to keep BF16 in range.

How does a per-precision correctness threshold change what a speed-up measurement actually means?

Without a threshold, a lower-precision speed-up is just a throughput number whose validity is assumed. A 26Q3 run gates each per-precision result on a correctness threshold defined for that precision, so a run that is fast but wrong scores nothing rather than topping a table. Aggregation additionally weights throughput by the memory a run actually moves, so the speed-up has to be earned rather than credited for being cheap. The figure belongs to that named release and its fixed catalogue, not to your own model.

What happens in a benchmark result when a device cannot execute a given precision?

It is scored as zero rather than silently omitted. Skipping an unsupported precision would leave a gap the reader cannot see and would quietly flatter the device’s aggregate; recording a zero keeps the absence visible where the comparison is made. That is a deliberate reporting choice about what the measurement discloses, not a statement about the format itself.

How should the precision-versus-recall distinction be kept separate from numerical precision?

They share a word and nothing else. Precision-and-recall are classification quality metrics computed on a model’s predictions; numerical precision is a bit-allocation choice about how values are represented, described here as the 1+8+23 split of FP32 and its lower-bit alternatives. A change in numerical precision may move a classification metric — which is exactly why a per-precision result is paired with a correctness check — but the two concepts sit on different axes and should never be substituted for one another in a performance claim.

FP32 remains the safety net for gradient stability

Master weights remain in FP32 during mixed-precision training because FP16 accumulators lose enough precision that convergence often stalls within a few thousand iterations. Because half-precision accumulation errors compound rapidly enough to freeze convergence after mere thousands of iterations. If any of those differ, are you still looking at a comparison, or two unrelated observations?