On Stochastic Rounding with Few Random Bits

Andrew Fitzgibbon; Stephen Felix

REVIEW 2 major objections 5 minor 14 references

On Stochastic Rounding with Few Random Bits

T0 review · 2 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that few-bit stochastic rounding implementations carry a systematic bias unless corrected, and that a two-step scheme—round to intermediate precision $P+N$, then apply SRFF—restores exact unbiasedness under the paper's…

desk verdict A correct and useful analysis of few-bit stochastic rounding bias, with a simple fix; the practical transfer rests on a prior the paper states clearly. read the letter →

arxiv 2504.20634 v1 pith:HRS325GM submitted 2025-04-29 math.NA cs.AIcs.LGcs.MScs.NA

Andrew Fitzgibbon , Stephen Felix This is my paper · ORCID

classification math.NAcs.AIcs.LGcs.MScs.NA MSC 65G5065Y04

keywords stochasticroundingfew-bitfloating-pointarithmeticbiasquantization-awaretraininglow-precisionmachinelearningrandombitsmixedprecision

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Stochastic rounding rounds a high-precision value up or down randomly, and this paper examines what happens when the supply of random bits is small. The paper proves that if the number $N$ of random bits is less than the precision gap $D$ between the source and target formats, the most natural implementation (SRFF) is systematically biased: with exact real inputs its mean error is $-2^{-(N+1)}$, and with inputs at finite precision its bias is exactly $(2^{-D}-2^{-N})/2$ for $N\le D$. A common variant (SRF) is unbiased for exact inputs but retains a bias of at most $2^{-(D+1)}$ for finite-precision inputs. The paper's correction, SRC, first rounds the input to an intermediate precision $P+N$ and then applies SRFF, which makes the expected error exactly zero under the paper's uniform-between-floats criterion. The practical stake is that biased few-bit rounding can make quantized language-model training diverge where an unbiased scheme converges.

What carries the argument

The load-bearing object is the rounding predicate $R(\delta,n)$, the indicator that $\delta$ plus a random noise term is at least 1; here $\delta$ is the fractional position of the input between two adjacent target floats and $n$ is a uniformly random integer with $0\le n<2^N$. SRFF uses noise $n2^{-N}$, SRF uses $(n+\tfrac12)2^{-N}$, and SRC applies SRFF after deterministically rounding the input to $P+N$ bits. The analysis reduces unbiasedness to the discrete identity that the average over $n$ of the integral of $R(\delta,n)$ over $\delta\in[0,1]$ equals $1/2$; the paper evaluates that average exactly for each scheme, producing the bias formulas and the tight bounds for $N\le D$. SRC works because rounding the input to $P+N$ bits makes the effective precision gap equal to $N$, the case in which SRFF's bias is zero.

What would settle it

Generate a large sample of inputs uniformly distributed between two adjacent floats of a target format, round them with SRFF using $N<D$ random bits, and compare the average of $x-\text{round}(x)$ with the predicted values: $-2^{-(N+1)}$ for exact real inputs, and $(2^{-D}-2^{-N})/2$ for inputs that are $P+D$-bit floats when $N\le D$. Matching those values to sampling accuracy confirms the bias calculation; a systematic mismatch would show the analysis depends on the assumed input distribution.

Watch

Extended reading notes

Core claim

Under the paper's uniform-between-floats unbiasedness criterion, the central discovery is that finite randomness, not finite input precision, is what breaks unbiasedness in natural stochastic rounding. SRFF, which rounds when $\delta + n2^{-N}\ge 1$, has non-positive pointwise bias: for every fractional position $\delta$, the expected rounded value is at most $\delta$. Integrated over a binade, this gives bias $-2^{-(N+1)}$ for infinite-precision inputs and exactly $(2^{-D}-2^{-N})/2$ for $P+D$-bit inputs when $N\le D$. SRF uses the shifted noise $(n+\tfrac12)2^{-N}$, which cancels the infinite-precision bias but leaves a finite-precision bias of $2^{-(D+1)}$. The proposed SRC scheme removes the bias structurally: deterministic rounding to $P+N$ bits reduces the problem to the $N=D$ case, for which SRFF is exactly unbiased. The paper demonstrates the practical effect in low-precision language-model training, where SRFF's bias leads to divergence while SRF and SRC converge.

Load-bearing premise

The load-bearing premise is that 'unbiased' means zero mean error under the uniform distribution between each adjacent pair of floats in the target format; if real input distributions are not uniform between adjacent floats, the numeric bias values and the exact unbiasedness of SRC do not transfer directly.

Editorial extensions

If this is right

Designers of low-precision hardware can use the formulas as a design rule: for SRFF, setting the number of random bits equal to the precision gap $N=D$ eliminates the bias, and for $N<D$ the bias is known in advance.
Training runs that quantize weights with biased few-bit rounding can show divergence that is a rounding artifact rather than an optimization failure; comparing SRFF against SRF or SRC in the same setup isolates the effect.
Unbiased few-bit rounding is achievable at any $N$ by adding one deterministic rounding step to $P+N$ bits before applying SRFF, so the cost of unbiasedness is a rounding operation rather than more random bits.
Because SRFF's bias is downward at every fractional position, its effect over a training run is a systematic drift that accumulates across billions of roundings, making it a concern in large-scale training even when the per-rounding bias is tiny.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A consequence the authors leave implicit: because SRFF's pointwise bias is non-positive on every interval, the direction of its drift does not depend on the input distribution even though the exact magnitude does; any training setup that rounds magnitudes with SRFF should expect a systematic downward pull on weight magnitudes.
The same interval-wise accounting could be applied to other rounding rules, such as ties-to-even stochastic rounding, block-floating-point formats, or non-uniform noise distributions, to test whether the 'round to $P+N$ first' construction remains exactly unbiased in those settings.
A larger-scale test of the thesis would compare SRFF and SRC in a training run while logging per-step sums of weight updates; the bias should appear as a systematic difference between the two update sums that grows with the number of roundings.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The short version: this paper is a solid, small analysis of a real implementation bug in few-bit stochastic rounding, with a simple fix. It deserves a serious referee and likely publication after minor cleanup.

The new things are: the finite-precision bias bound for SRF, the SRC two-step correction (round to P+N then apply SRFF), and a small QAT demonstration where biased SRFF diverges while SRF/SRC converge. The derivations in §III and Appendix A are correct; I checked the arithmetic and the formulas match the empirical mean values in Figs 1–2. The paper is honest that the infinite-precision SRFF bias was already in El Arar et al.; it frames its contribution as the finite-precision analysis and the mitigation, which is fair.

Soft spots are real but mostly minor. The unbiasedness definition is uniform-between-adjacent-floats; the stress-test is right that a fixed input can show pointwise bias for all three schemes, so the exact numbers don't transfer to arbitrary nonuniform ML tensors. But the paper states the criterion explicitly and doesn't hide it, and the ML experiment is a demonstration, not a claim of universal transfer. The experiments are single-seed, small scale, and the authors say so in the limitations. There are also two annoying inconsistencies: the code availability statement in the abstract says code is available while the conclusion says it will be made available, and Figure 2's caption says "2 bits" where the text says 3. These should be fixed but are not substantive.

The weakness that matters most: the SRC correction's practical value is only as strong as the uniform-interval criterion. If a hardware vendor cares about worst-case bias on adversarial inputs, this paper doesn't answer that. But that's a scope question, not a flaw in the math.

Who should read it: anyone implementing stochastic rounding in hardware or quantized training, and researchers working on low-precision numerics. I'd bring it to reading group. I'd cite it if I write about SR implementations.

For peer review: send it. The theory is checkable, the correction is new, and the negative result about SRFF is useful even if the ML evidence is modest.

Referee Report

2 major / 5 minor

Summary. The paper studies few-bit stochastic rounding (FBSR), in which N random bits are used to round a value from a source format with precision Q = P + D to a target format with precision P, with N < D. It analyzes two natural implementations: SRFF (adding n × 2^{-N}) and SRF (adding (n + 1/2) × 2^{-N}), under the interval-uniform unbiasedness criterion of Eq. (6). It derives exact bias expressions for SRFF and SRF for infinite-precision inputs and bounds for finite-precision inputs, and proposes a correction, SRC, which first rounds to precision P + N and then applies SRFF, claiming exact unbiasedness. Experiments with nanoGPT show that SRFF can diverge in quantized language-model training while SRF and SRC converge.

Significance. The derivations are correct and the experiments reproduce the predicted bias values, which gives the paper a solid core. The practical message is useful: with few random bits, natural stochastic-rounding implementations can carry a systematic bias large enough to affect low-precision training. The proposed SRC correction is simple and plausible. The code and symbolic-algebra derivations are valuable for reproducibility. However, the quantitative claims are tied to the uniform-interval prior of Eq. (6), and the paper would be substantially stronger if it clearly scoped that limitation and supplied a formal proof of SRC's unbiasedness.

major comments (2)

[§III, Eq. (6); §IV] The unbiasedness criterion in Eq. (6) is a chosen prior (uniform between adjacent target floats), not pointwise unbiasedness. For a fixed input the paper's statements do not hold: with N = 2 and fractional position δ = 0.4, SRFF has mean error -0.15, SRF +0.10, and SRC +0.10, and only after averaging δ over [0,1) do SRF and SRC become unbiased under this criterion. The language-model experiments use non-uniform weight and gradient distributions, so the theoretical bias values do not transfer directly and the claim that SRC corrects the bias in QAT is not established by the analysis. Please either state explicitly that the bias results are prior-dependent and not worst-case, or provide pointwise worst-case bias expressions or experiments with measured input distributions to support the ML conclusions.
[§III-F, Eq. (8); Appendix A] The exactness of SRC is asserted rather than proved, and the phrase 'Round is any unbiased rounding scheme' is too broad. For finite-precision source values on a uniform grid, round-to-nearest with ties-to-even or ties-to-odd is unbiased over the grid, but ties-away-from-zero is not. The paper should include a formal derivation of the unbiasedness of the first rounding in Eq. (8) over the discrete source grid (for example, by pairing the tie contributions in each block of 2^{D-N} source points) and restrict 'Round' to tie-breaking rules that satisfy this discrete unbiasedness.

minor comments (5)

[Figure 2 caption] The caption says '100,000 samples are rounded with 2 bits of randomness,' but the figure labels and the text use 3 bits; this is inconsistent.
[Figure 2, right panel] The panel label 'SR' is inconsistent with the caption and the text, which call this method SRC.
[Figure 2 caption] The caption contains the apparent debugging artifact 'P+srnumbits=6 =?= 8'; this should be removed.
[§III-E and Appendix A-D] The SRF finite-precision bound is tight for N < D, not for N ≤ D; for N = D the bias is zero, so the parenthetical in §III-E should be corrected.
[§III and Eq. (6)] The paper should explicitly note in the introduction or at Eq. (6) that its use of 'unbiased' differs from the common pointwise definition of unbiased stochastic rounding, in order to avoid misleading readers who may not notice the interval-uniform prior.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: bias calculations and the SRC correction follow from explicit definitions with no fitted inputs or load-bearing self-citation.

full rationale

The paper's central claims are derived directly from its stated definitions. The bias of SRFF and SRF is computed by explicit summation over the uniform-interval family P defined in Eq. (6), using the formal definitions of RSRFF and RSRF; no fitted constants, empirical calibration, or external results are used to force the formulas. The finite-precision bounds are obtained by analogous discrete sums over the 2^D input grid, also from first principles, with the no-binade-wrap caveat stated explicitly. The SRC correction is constructed by first rounding to precision P+N and then applying SRFF, and its unbiasedness is a direct consequence of the same definitional calculation, reducing to the N=D case already derived. The unbiasedness criterion is an openly stated modeling choice, not a hidden assumption introduced to manufacture the conclusion; the paper explicitly acknowledges that the expectation is over a chosen prior family. The experiments are demonstrations that the predicted bias affects training and do not feed fitted parameters back into the theory. The only self-citation, the gfloat library [13], is used as an implementation tool for experiments and is not load-bearing for any analytical result. No circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on four explicit domain assumptions (floating-point model, uniform-interval unbiasedness criterion, uniform discrete input model for finite-precision bounds, uniform random bits) and one standard mathematical fact (RTNE unbiasedness). No parameters are fitted to data; all bias values are derived symbolically. The assumptions are stated in the paper, so the ledger is transparent.

assumptions (5)

domain assumption Floating point format model of Eq. 1 (sign, exponent, significand, subnormals; NaNs, infinities, and negative values excluded from the rounding analysis).
The rounding analysis is restricted to strictly positive finite values; extension to negatives is symmetric by sign, and deterministic round-to-nearest is assumed for out-of-range values.
domain assumption Unbiasedness criterion in Eq. 6: family of test distributions P consists of uniform distributions between each adjacent pair of floats.
All bias values, bounds, and the SRC construction are judged against this criterion; it is a modeling choice the paper states explicitly.
domain assumption For finite-precision inputs, the 2^D fractional values in a binade are treated as equally likely (uniform) in the bias sums of Eq. 7 and Appendix A.
The bias bounds for SRFF and SRF in §III-E and §A-C/D are derived under this discrete uniform prior; real data distributions may differ.
domain assumption The random bits n are uniformly distributed over [0, 2^N-1] and independent of the input.
The paper states that RNG quality is orthogonal to its concerns; this assumption underlies the expectation over n.
standard math Round-to-nearest with ties to even (RTNE) is an unbiased deterministic rounding scheme for the SRC's preliminary step (Eq. 8).
SRC relies on the preliminary rounding being unbiased under the uniform-interval criterion; RTNE has zero mean error on each interval.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On Stochastic Rounding with Few Random Bits." pith.science (2026). https://pith.science/paper/HRS325GM

@misc{pith2026250420634,
  author       = {Pith},
  title        = {Pith review of: On Stochastic Rounding with Few Random Bits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HRS325GM}},
  note         = {Machine review of arXiv:2504.20634}
}

read the original abstract

Large-scale numerical computations make increasing use of low-precision (LP) floating point formats and mixed precision arithmetic, which can be enhanced by the technique of stochastic rounding (SR), that is, rounding an intermediate high-precision value up or down randomly as a function of the value's distance to the two rounding candidates. Stochastic rounding requires, in addition to the high-precision input value, a source of random bits. As the provision of high-quality random bits is an additional computational cost, it is of interest to require as few bits as possible while maintaining the desirable properties of SR in a given computation, or computational domain. This paper examines a number of possible implementations of few-bit stochastic rounding (FBSR), and shows how several natural implementations can introduce sometimes significant bias into the rounding process, which are not present in the case of infinite-bit, infinite-precision examinations of these implementations. The paper explores the impact of these biases in machine learning examples, and hence opens another class of configuration parameters of which practitioners should be aware when developing or adopting low-precision floating point. Code is available at http://github.com/graphcore-research/arith25-stochastic-rounding.

Figures

Figures reproduced from arXiv: 2504.20634 by the authors.

**Figure 2.** Bias in stochastic rounding with finite randomness, finite-precision inputs. For each [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Experiments on language model training (nanoGPT, small model). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Experiments on language model training (nanoGPT, GPT-2 (350M params)). Binary16 baseline: validation loss follows training loss (not shown) very [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

[1]

Dynamic point stochastic rounding algorithm for limited precision arithmetic in deep belief network training,

M. Essam, T. B. Tang, E. T. W. Ho, and H. Chen, “Dynamic point stochastic rounding algorithm for limited precision arithmetic in deep belief network training,” in 2017 8th International IEEE/EMBS Confer- ence on Neural Engineering (NER) . IEEE, 2017, pp. 629–632

work page 2017
[2]

8-bit numeri- cal formats for deep neural networks,

B. Noune, P. Jones, D. Justus, D. Masters, and C. Luschi, “8-bit numeri- cal formats for deep neural networks,” arXiv preprint arXiv:2206.02915, 2022

arXiv 2022
[3]

Climate modeling in low precision: Effects of both deterministic and stochastic rounding,

E. A. Paxton, M. Chantry, M. Kl ¨ower, L. Saffin, and T. Palmer, “Climate modeling in low precision: Effects of both deterministic and stochastic rounding,” Journal of Climate , vol. 35, no. 4, pp. 1215–1229, 2022

work page 2022
[4]

Fast: DNN training under variable precision block floating point with stochastic rounding,

S. Q. Zhang, B. McDanel, and H. Kung, “Fast: DNN training under variable precision block floating point with stochastic rounding,” in 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2022, pp. 846–860

work page 2022
[5]

Effects of round-to-nearest and stochastic rounding in the numerical solution of the heat equation in low precision,

M. Croci and M. B. Giles, “Effects of round-to-nearest and stochastic rounding in the numerical solution of the heat equation in low precision,” IMA Journal of Numerical Analysis, vol. 43, no. 3, pp. 1358–1390, 2023

work page 2023
[6]

Stochastic rounding: implementation, error analysis and applications,

M. Croci, M. Fasi, N. J. Higham, T. Mary, and M. Mikaitis, “Stochastic rounding: implementation, error analysis and applications,” Royal Soci- ety Open Science , vol. 9, no. 3, p. 211631, 2022

work page 2022
[7]

Stochastic rounding variance and probabilistic bounds: A new approach,

E.-M. El Arar, D. Sohier, P. de Oliveira Castro, and E. Petit, “Stochastic rounding variance and probabilistic bounds: A new approach,” SIAM Journal on Scientific Computing , vol. 45, no. 5, pp. C255–C275, 2023

work page 2023
[8]

Probabilistic error analysis of limited-precision stochastic rounding

E.-M. El Arar, M. Fasi, S.-I. Filip, and M. Mikaitis, “Probabilistic error analysis of limited-precision stochastic rounding,” arXiv preprint arXiv:2408.03069, 2024

work page Pith review arXiv 2024

Show all 14 references

[9]

Improved stochastic rounding,

L. Xia, M. Anthonissen, M. Hochstenbach, and B. Koren, “Improved stochastic rounding,” arXiv preprint arXiv:2006.00489 , 2020

2006 arXiv
[10]

You already have it: A generator-free low- precision DNN training framework using stochastic rounding,

G. Yuan, S.-E. Chang, Q. Jin, A. Lu, Y . Li, Y . Wu, Z. Kong, Y . Xie, P. Dong, M. Qin et al. , “You already have it: A generator-free low- precision DNN training framework using stochastic rounding,” in Euro- pean Conference on Computer Vision . Springer, 2022, pp. 34–51

2022
[11]

Quantization-aware training for large language models with PyTorch,

A. Or, J. Zhang, E. Smothers, K. Khandelwal, and S. Rao, “Quantization-aware training for large language models with PyTorch,” The PyTorch Foundation, 2024. [Online]. Available: https://pytorch.org/ blog/quantization-aware-training

2024
[12]

NanoGPT,

A. Karpathy, “NanoGPT,” https://github.com/karpathy/nanoGPT, 2024

2024
[13]

GFloat: Generic floating point formats in Python,

A. Fitzgibbon, “GFloat: Generic floating point formats in Python,” https: //gfloat.readthedocs.io/, 2024

2024
[14]

Interim report on binary floating-point formats for machine learning,

IEEE SA P3109 Working Group, “Interim report on binary floating-point formats for machine learning,” The Institute of Electrical and Electronics Engineers, Inc., 2024. [Online]. Available: https://github.com/P3109/Public

2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Dynamic point stochastic rounding algorithm for limited precision arithmetic in deep belief network training,

M. Essam, T. B. Tang, E. T. W. Ho, and H. Chen, “Dynamic point stochastic rounding algorithm for limited precision arithmetic in deep belief network training,” in 2017 8th International IEEE/EMBS Confer- ence on Neural Engineering (NER) . IEEE, 2017, pp. 629–632

work page 2017

[2] [2]

8-bit numeri- cal formats for deep neural networks,

B. Noune, P. Jones, D. Justus, D. Masters, and C. Luschi, “8-bit numeri- cal formats for deep neural networks,” arXiv preprint arXiv:2206.02915, 2022

arXiv 2022

[3] [3]

Climate modeling in low precision: Effects of both deterministic and stochastic rounding,

E. A. Paxton, M. Chantry, M. Kl ¨ower, L. Saffin, and T. Palmer, “Climate modeling in low precision: Effects of both deterministic and stochastic rounding,” Journal of Climate , vol. 35, no. 4, pp. 1215–1229, 2022

work page 2022

[4] [4]

Fast: DNN training under variable precision block floating point with stochastic rounding,

S. Q. Zhang, B. McDanel, and H. Kung, “Fast: DNN training under variable precision block floating point with stochastic rounding,” in 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2022, pp. 846–860

work page 2022

[5] [5]

Effects of round-to-nearest and stochastic rounding in the numerical solution of the heat equation in low precision,

M. Croci and M. B. Giles, “Effects of round-to-nearest and stochastic rounding in the numerical solution of the heat equation in low precision,” IMA Journal of Numerical Analysis, vol. 43, no. 3, pp. 1358–1390, 2023

work page 2023

[6] [6]

Stochastic rounding: implementation, error analysis and applications,

M. Croci, M. Fasi, N. J. Higham, T. Mary, and M. Mikaitis, “Stochastic rounding: implementation, error analysis and applications,” Royal Soci- ety Open Science , vol. 9, no. 3, p. 211631, 2022

work page 2022

[7] [7]

Stochastic rounding variance and probabilistic bounds: A new approach,

E.-M. El Arar, D. Sohier, P. de Oliveira Castro, and E. Petit, “Stochastic rounding variance and probabilistic bounds: A new approach,” SIAM Journal on Scientific Computing , vol. 45, no. 5, pp. C255–C275, 2023

work page 2023

[8] [8]

Probabilistic error analysis of limited-precision stochastic rounding

E.-M. El Arar, M. Fasi, S.-I. Filip, and M. Mikaitis, “Probabilistic error analysis of limited-precision stochastic rounding,” arXiv preprint arXiv:2408.03069, 2024

work page Pith review arXiv 2024