REVIEW 4 major objections 5 minor 1 cited by
Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper demonstrates that Adam-style optimizers can keep their state in as few as 2-3 bits per value without meaningful accuracy loss, by fixing two EMA-specific quantization failures: signal swamping in unsigned states and inflated…
desk verdict SOLO's 4/2-bit optimizer is a solid, well-tested contribution; the 2-bit claim leans on a heuristic momentum rule that the paper itself doesn't fully support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the quantized EMA update: $\tilde{x}_t = Q^\dagger(q_t)$, then $\hat{x}_{t+1} = \beta\tilde{x}_t + (1-\beta)z_{t+1}$, then $q_{t+1}=Q(\hat{x}_{t+1})$, where $z_{t+1}$ is the gradient (signed) or squared gradient (unsigned). Two results extracted from this loop do the work. Theorem 3.1 states that the quantized level stays fixed whenever the level radius satisfies $r \ge (1-\beta)|\frac{z_{t+1}}{\Delta_{t+1}} - y_{q_t}| + |\frac{\Delta_t}{\Delta_{t+1}}-1|$, which identifies why standard quantizers fail at low precision and large tensor size. Theorem 3.6 proves that signed-state quantization inflates gradient variance by at most $\big(\frac{\beta}{1-\beta} r_{\max}(b)\Delta_t\big)^2$, which motivates the $\beta'$ adjustment rule of Eq. (15) using $r_{\mathrm{median}}$. The closing piece is the logarithmic quantizer of Eq. (10), $Q(x)=\mathrm{Clip}(\lfloor \log_\alpha(x/\Delta)+\xi\rceil;0,2^b-1)$ with $\alpha=(x_p/\Delta)^{1/(2^b-1)}$ and $\xi\sim U[-0.5,0.5]$, which prevents swamping and matches EMA decay.
What would settle it
Run the recommended 2-bit settings ($\beta'=0.527$ for fine-tuning, lower for scratch) on a large-scale pretraining run the paper leaves untested, e.g., a 175B model; if loss diverges or the accuracy gap exceeds the fine-tuning margins, the variance-regulation hypothesis fails. A cheaper check: directly measure gradient variance under 2-bit signed quantization and compare it with the bound in Eq. (14) and the claimed $r_{\mathrm{median}}/r_{\max}$ ratio similarity.
Extended reading notes
Core claim
The paper's central claim is that ultra-low-bit quantization of Adam is possible when the EMA state update is treated as a dequantize-EMA-quantize loop, and two mechanisms are added. For the unsigned second moment, Theorem 3.1 gives a precise signal-swamping condition: if the quantization radius around the current level is large enough, the state stays constant even when the incoming signal differs sharply; the paper shows this occurs at momentum values as low as $\beta \approx 0.833$ for 2-bit linear quantization. Its remedy is a logarithmic quantizer with levels clustered near zero, base $\alpha = (x_p/\Delta)^{1/(2^b-1)}$, and stochastic rounding, which also reproduces the exact expected decay time under consecutive zero signals. For the signed first moment, Theorem 3.6 bounds the added gradient variance by $\big(\frac{\beta}{1-\beta} r_{\max}(b)\Delta_t\big)^2$, and SOLO chooses a lower momentum $\beta'$ so that $\frac{\beta'}{1-\beta'} r_{\mathrm{median}}(b') \le \frac{\beta}{1-\beta} r_{\mathrm{median}}(b)$, giving $\beta' \le 0.820$ for 4-bit fine-tuning and $\beta' \le 0.527$ for 2-bit. The resulting 4/2-bit AdamW matches 32-bit AdamW within statistical noise on the evaluated tasks.
Load-bearing premise
The central claim depends on the hypothesis that bringing the quantization-induced gradient-variance upper bound under control is enough to keep convergence acceptable: the paper states this as a hypothesis rather than a theorem, and the recommended $\beta'$ values are chosen from empirical observation on the evaluated tasks, not from a convergence proof.
Editorial extensions
If this is right
- Optimizer memory for Adam drops to about one-eighth (4/2-bit) or one-sixteenth (2-bit) of the 32-bit state size, letting practitioners fit larger models or batches on the same hardware.
- Existing learning rates and weight decay remain unchanged; only the first-moment momentum needs lowering, to 0.8 for 4/2-bit and 0.5 for 2-bit fine-tuning, and to 0.3/0.1 for training from scratch.
- The quantization scheme is model- and task-agnostic in the tested regimes: it transfers across Swin-T, Transformer-Base, RoBERTa-L, DCN, HSTU, LLaMA-7B/13B/33B, and LLaVA-1.5, and also covers the AdaBelief optimizer.
- The logarithmic quantizer preserves second-moment dynamics when the block size is increased from 128 to 2048, unlike linear or dynamic-exponent quantizers that collapse, so per-block scale-factor overhead can be nearly eliminated.
Reading between the lines
- This suggests the momentum-reduction recipe could carry over to other EMA-based optimizers such as Lion or Adafactor, since the variance bound comes from the EMA loop rather than from Adam's specifics.
- The signal-swamping condition implies a general design rule for low-bit states that feed a denominator: allocate quantization levels near zero and use stochastic rounding; this could inform low-bit batch-normalization statistics or embedding-table scalers.
- The p-quantile base selection appears transferable to fully quantized training of weights and activations, where small-magnitude values dominate, but this would require independent validation.
- The reported equivalence holds on only five MMLU-style benchmarks; extending the comparison to a larger suite with repeated seeds would test whether the "no significant difference" claim generalizes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SOLO, a low-bit optimizer-state quantization method that aims to maintain Adam-style optimizer states at 3-bit or even 2-bit precision with minimal accuracy loss. The authors identify two problems with naive low-bit quantization: signal swamping in unsigned EMA updates of the second moment, and increased gradient variance in signed EMA updates of the first moment. They address swamping with a logarithmic quantization scheme whose base is set by a p-quantile and which admits a simple stochastic-rounding implementation with an exact exponential-decay property. They address the gradient-variance issue by proposing a reduced first-moment momentum β' selected through an inequality (Eq. 15) that compares quantization radii at different precisions. The method is evaluated across CV, NLP, recommender systems, LLM instruction tuning, and LVM visual instruction tuning, with the 4/2-bit variant and a fully 2-bit variant compared against 32-bit, 16-bit, 8-bit, and 4-bit AdamW baselines. The paper also reports a robustness study of block-size choice and p-quantile sensitivity.
Significance. If the results hold, SOLO is a significant advance: it would reduce optimizer memory by roughly 10x relative to 32-bit AdamW while keeping performance close to full precision, and it is orthogonal to model architecture and distributed training. The unsigned logarithmic quantization with the p-quantile base is a clean and useful contribution, particularly because it preserves state dynamics under sparse-gradient and zero-signal conditions. The paper is transparent about its limitations, explicitly labels the momentum-reduction rule as a hypothesis, releases code, and reports broad experiments including LLaMA-13B/33B and LLaVA-1.5. The main weakness is that the load-bearing recipe for β'—the signed-momentum adjustment that makes 2-bit states workable—is a heuristic whose stated justification contains a numerically false 'nearly identical' ratio claim and whose variance analysis omits the EMA estimator's own variance increase when β is reduced. These issues are fixable but require substantive revision.
major comments (4)
- [Section 3.3, Eq. (15), Fact A.1, Figure 4] The assertion that the ratio r*(b)/r*(b') is 'nearly identical' for the median and maximum radius is contradicted by the paper's own Fact A.1 for signed DE quantization. For example, using the reported values, r_med(5)/r_med(2)=0.034/0.275≈0.124, while r_max(5)/r_max(2)=0.056/0.275≈0.204, a difference of roughly 1.65x. Since Theorem 3.6 bounds the quantization-induced variance in terms of r_max, replacing r_max by r_median in Eq. (15) is not a cosmetic change; the 'upper bounds' shown in Figure 4 are not consequences of the theorem. The authors should either re-derive the inequality with r_max and adjust the recommended β' values, or explicitly present Eq. (15) as an empirical heuristic and remove the 'nearly identical' claim.
- [Section 3.3, Eq. (14), Figure 6] The variance analysis in Eq. (14) accounts only for the quantization-induced noise term and omits the effect of reducing β on the EMA estimator's own variance. For a stationary gradient with variance σ², the first-moment estimate has variance proportional to (1-β)/(1+β); reducing β from 0.9 to 0.5 increases this component by roughly a factor of 6, which can offset the reduction in the quantization-noise term. The empirical optimum at β'≈0.5 in Figure 6 therefore does not follow from Eq. (14). The authors should provide a complete variance tradeoff for the momentum choice, including the estimator-variance term, or explicitly rephrase the Section 3.3 rationale as empirically motivated rather than derived.
- [Section 3.3, Theorem 3.6] Theorem 3.6's unbiasedness result, Eq. (13), relies on stochastic rounding, as the proof notes, but the manuscript does not state whether the signed DE quantization used in the experiments employs stochastic rounding. If nearest rounding is used, E[\tilde{g}] ≠ E[g] and the variance bound in Eq. (14) is not directly applicable. Please state the rounding scheme used for signed states; if it is not stochastic rounding, extend the analysis to cover the resulting bias or provide empirical evidence that the bias is negligible for the reported settings.
- [Tables 1-2, Section 4.1] Several 2-bit results show visible drops relative to 32-bit AdamW that are not discussed: HSTU HR 32.73→32.03, LLaVA ScienceQA 70.81→69.51, and LLaMA SIQA 48.50→47.21. The Wilcoxon signed-rank tests are based on only five metrics and are underpowered, so 'no statistically significant difference' (p=0.125 and p=0.285) should not be presented as evidence of equivalence. Please discuss these exceptions explicitly and soften the 'minimal accuracy loss' phrasing for the 2-bit setting, or provide additional runs and confidence intervals to support the claim.
minor comments (5)
- [Table 2 captions] The phrase 'ap-vlue' should read 'p-value'.
- [Eq. (10)] The notation ⌊·⌉ for convergent rounding is used without definition in the main text; please define it where Eq. (10) is introduced.
- [Figure 3 caption] The caption and subfigure text are dense and difficult to parse; consider splitting the figure into two panels with clearer labels for the quantization levels and the p-quantile mechanism.
- [Appendix C.1, Table 9] The word 'utlize' should be 'utilize'.
- [Section 3.3, Figure 4] If Eq. (15) is a heuristic rather than a proven upper bound, the figure caption should state this explicitly to avoid overstating the theoretical grounding.
Circularity Check
No significant circularity: SOLO's design and recommended beta' are either derived from explicit quantization-geometry arguments or openly labeled as empirical hypotheses, not as predictions forced by fitted inputs.
full rationale
The paper's two central design choices are not circular. The unsigned logarithmic quantizer follows from the signal-swamping bound (Theorem 3.1, Corollary 3.2) and the variance analysis of stochastic rounding near zero (Proposition 3.4); the choice of base alpha=(x_p/Delta)^(1/(2^b-1)) is a separate quantile-adaptive construction that is tested for sensitivity (Figure 8, Table 11), not derived from the final accuracy numbers. The signed-momentum adjustment is openly presented as a hypothesis: Section 3.3 states 'we hypothesize that an acceptable convergence rate could be achieved once the upper bound is somewhat regulated' and Eq. (15) is a regulation criterion, not a theorem; the paper then tests beta' values empirically (Figure 6, Table 12). No self-citations are load-bearing—the citations to Higham, Oh, Xia, Dettmers, Li, etc. are external and do not supply the paper's own claims. The hyperparameters p=0.1 and the default beta' values are disclosed as empirical settings ('based on Figure 4 and empirical observations', 'for simplicity'), so they are not secretly fitted quantities relabeled as predictions. The paper also honestly flags an important limitation: large-scale pretraining 'remains unverified due to prohibitive computational costs', which is a scope restriction rather than a circular move. The r_median-vs-r_max ratio statement in Section 3.3 is not fully supported by Fact A.1 (2-bit signed DE gives r_med ratio approximately 0.0145 versus r_max ratio approximately 0.0255), but this is an internal-consistency/correctness concern, not a circularity: Eq. (15) would still be a non-tautological heuristic. Overall, the derivation chain is self-contained and empirically anchored; there are no steps in which an output quantity equals an input by construction.
Assumptions & free parameters
free parameters (2)
- p-quantile p =
0.1 (fixed); effective range reported as [0.05, 0.3]
- first-moment momentum β' =
4/2-bit: 0.8 (fine-tune), 0.3 (scratch); 2-bit: 0.5 (fine-tune), 0.1 (scratch)
assumptions (4)
- standard math Stochastic rounding has zero-mean quantization error
- domain assumption Reducing gradient variance improves convergence
- ad hoc to paper Variance-upper-bound regulation is sufficient for acceptable convergence
- ad hoc to paper r_median approximates r_max for the ratio in Eq. (15)
Cite this review
Pith. "Pith review of Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics." pith.science (2026). https://pith.science/paper/ZPNXSEK2
@misc{pith2026250500347,
author = {Pith},
title = {Pith review of: Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZPNXSEK2}},
note = {Machine review of arXiv:2505.00347}
}
read the original abstract
The rapid scaling of models has led to prohibitively high training and fine-tuning costs. A major factor accounting for memory consumption is the widespread use of stateful optimizers (e.g., Adam), which maintain auxiliary information of even 2x the model size in order to achieve optimal convergence. We therefore present SOLO in this work to spawn a novel type of optimizer that requires an extremely light memory footprint. While previous efforts have achieved certain success in 8-bit or 4-bit cases, SOLO enables Adam-style optimizers to maintain quantized states with precision as low as 3 bits, or even 2 bits. This immense progress is due to the identification and resolution of two key challenges: the signal swamping problem in unsigned quantization that results in unchanged state dynamics, and the increased gradient variance in signed quantization that leads to incorrect descent directions. The theoretical analysis suggests a tailored logarithmic quantization for the former and a precision-specific momentum hyperparameter for the latter. SOLO can thus be seamlessly applied to Adam-style optimizers, leading to substantial memory savings with minimal accuracy loss.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 1 Pith paper
-
MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
MuonQ achieves stable 4-bit quantization of Muon optimizer states via pre-quantization normalization, singular component decomposition with power iteration, and μ-law companding, matching full-precision loss and accur...
Reference graph
Works this paper leans on
-
[6]
1-bit fqt: Pushing the limit of fully quantized training to 1-bit
Gao, C., Chen, J., Zhao, K., Wang, J., and Jing, L. 1-bit fqt: Pushing the limit of fully quantized training to 1-bit. arXiv preprint arXiv:2408.14267,
-
[7]
Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,
GenAI, M. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,
-
[8]
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,
-
[9]
Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A com- prehensive survey.arXiv preprint arXiv:2403.14608,
-
[10]
Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,
arXiv 2009
-
[11]
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024a. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical repor...
-
[12]
Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 364,
Liu, Y . Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 364,
arXiv 1907
-
[15]
Socialiqa: Commonsense reasoning about social interac- tions.arXiv preprint arXiv:1904.09728,
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y . Socialiqa: Commonsense reasoning about social interac- tions.arXiv preprint arXiv:1904.09728,
arXiv 1904
Show all 28 references
-
[16]
1-bit stochas- tic gradient descent and its application to data-parallel distributed training of speech dnns
Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. 1-bit stochas- tic gradient descent and its application to data-parallel distributed training of speech dnns. InInterspeech, vol- ume 2014, pp. 1058–1062. Singapore,
2014
-
[17]
Mkq- bert: Quantized bert with 4-bits weights and activations
Tang, H., Zhang, X., Liu, K., Zhu, J., and Kang, Z. Mkq- bert: Quantized bert with 4-bits weights and activations. arXiv preprint arXiv:2203.13483,
-
[18]
Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,
-
[19]
GLUE: A multi-task benchmark and analy- sis platform for natural language understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE: A multi-task benchmark and analy- sis platform for natural language understanding. InPro- ceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353–355...
2018 arXiv
-
[20]
Improved stochastic rounding.arXiv preprint arXiv:2006.00489,
Xia, L., Anthonissen, M., Hochstenbach, M., and Ko- ren, B. Improved stochastic rounding.arXiv preprint arXiv:2006.00489,
2006 arXiv
-
[21]
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115,
-
[22]
D., et al
Zhang, B., Luo, L., Chen, Y ., Nie, J., Liu, X., Li, S., Zhao, Y ., Hao, Y ., Yao, Y ., Wen, E. D., et al. Wukong: Towards a scaling law for large-scale recommendation. InInter- national Conference on Machine Learning (ICML), pp. 59421–59434, 2024a. Zhang, Y ., Chen, C., Li, Z...
-
[23]
16 SOLO -0.571 -0.429 -0.286 -0.143 0.1430 0.286 0.7140.429 0.571 0.857 1-1 -0.857 -0.714 -0.571 -0.429 -0.286 -0.143 0.1430 0.286 0.7140.429 0.571 0.857 1-1 -0.857 -0.7140.3330 0.667 10.3330 0.667 1 -0.571 -0.429 -0.286 -0.143 0.1430 0.286 0.7140.429 0.571 0.857 1-1 -0.857 -0...
2023
-
[24]
c−1 c , (βc)k+k′+1, w.p
can be represented as follows: qt+1 = (βc)k+k′ , w.p. c−1 c , (βc)k+k′+1, w.p. 1 c. (26) Thus, transitioning from the state value of (βc)k to (βc)k+s needs exact s independent successes. Denoted by N the counting of updates required to achieve exacts successes, this random var...
2020
-
[25]
LLaMA-7B/13B/33B ( ) on Alpaca.LLM fine-tuning is one of the most compelling applications, but still faces challenges due to its high memory demands
Hyperparameter LLaMA-7B LLaMA-13B LLaMA-33B Batch Size 120 120 128 LR 2e-5 1e-5 1e-5 Weight Decay 0 0 0 β2 0.999 0.999 0.999 Warmup Ratio 0.03 0.03 0.03 Epochs 3 5 5 Table 9.The hyperparameters of LLaMA fine-tuning on Alpaca. LLaMA-7B/13B/33B ( ) on Alpaca.LLM fine-tuning is o...
2023
-
[26]
and standard common sense reasoning benchmarks: ARC easy and challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and Social Interaction QA (Sap et al., 2019), we utlize OpenCompass (Contributors, 2023). Hyperparameter Value Batch Size 120 LR 2e-5 Weight Decay 0...
2018
-
[27]
on a mixed instruction tuning dataset comprising COCO (Caesar et al., 2018), GQA (Hudson & Manning, 2019), OCR-VQA (Mishra et al., 2019), TextVQA (Singh et al., 2019), and VisualGenome (Krishna et al., 2017). Due to 7https://github.com/MTandHJ/RecBoard/tree/master/DCN 8https:/...
2018
-
[28]
In accordance with the official evaluation protocols, we assess the fine-tuned models on several benchmarks, including ScienceQA (Lu et al., 2022), TextVQA (Singh et al., 2019), POPE (Li et al., 2023c), and MME (Fu et al., 2023). Specifically, we report accuracy for ScienceQA ...
2022
-
[2018]
Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789,
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789,
-
[2019]
M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report.arXiv preprint arXiv:2305.10403,
-
[2020]
Gptq: Accurate post-training quantization for generative pre- trained transformers.arXiv preprint arXiv:2210.17323,
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers.arXiv preprint arXiv:2210.17323,
-
[2021]
Scaling neu- ral machine translation.arXiv preprint arXiv:1806.00187,
Ott, M., Edunov, S., Grangier, D., and Auli, M. Scaling neu- ral machine translation.arXiv preprint arXiv:1806.00187,
-
[2022]
MME: A comprehensive evaluation benchmark for mul- timodal large language models.CoRR, abs/2306.13394,
Fu, C., Chen, P., Shen, Y ., Qin, Y ., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., Li, K., Sun, X., and Ji, R. MME: A comprehensive evaluation benchmark for mul- timodal large language models.CoRR, abs/2306.13394,
-
[2023]
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O
URL https://lmsys.org/blog/ 2023-03-30-vicuna/. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,
2023 arXiv
-
[2024]
Qwen technical report.arXiv preprint arXiv:2309.16609,
Bai, J., Bai, S., Chu, Y ., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y ., Huang, F., et al. Qwen technical report.arXiv preprint arXiv:2309.16609,
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.