{"id":"8f1d14a1-d13d-4978-9241-cad4875a8236","arxiv_id":"2505.00347","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SOLO compresses Adam optimizer states to 2 to 3 effective bits using p-quantile-based logarithmic quantization for second moments and momentum reduction for first moments, preserving accuracy on most tested benchmarks.","lead":"SOLO is a quantization scheme that stores Adam-like optimizer states in as few as 2 bits instead of 32, cutting optimizer memory by roughly 10 to 16 times. On image, text, recommendation, and LLM fine-tuning benchmarks, the 4/2-bit variant matches full-precision accuracy closely, while the 2-bit variant shows smaller but visible losses on some tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2-bit claim rests on an unproven momentum-adjustment heuristic: Eq. 15's r_median 'nearly identical' ratio is contradicted by the paper's own Fact A.1 at b'=2, and no convergence argument covers the reduced-β EMA regime.","rationale":"The paper makes a credible case for the unsigned log-quantization fix: synthetic experiments, Proposition 3.5, and block-size robustness in Figure 5 support the swamping analysis, and the 4/2-bit results are broadly consistent across CV/NLP/RS/LLM/LVM with released code. The single weakest link is the signed-momentum adjustment, which is the only mechanism enabling 2-bit signed states. It is explicitly a hypothesis, the r_median substitution is contradicted by the paper's own radius tables at 2 bits, and the chosen β' values are selected empirically (Figure 6, Appendix D.2). The concern is not that the empirical results are fabricated or that 2-bit optimization is impossible; it is that the central claim generalizes beyond the measured fine-tuning/from-scratch settings on the authority of an unproven variance-regulation conjecture. A targeted experiment can settle whether the heuristic is robust or regime-limited. Since the reader already identified this as the weakest assumption and assigned CONDITIONAL, my read does not change the verdict.","tokens_in":27165,"tokens_out":18519,"duration_ms":181445,"concrete_test":"Run a controlled LLaMA-7B Alpaca fine-tuning comparison under 2-bit SOLO with β' = 0.5, β' = 0.647 (the r_max-derived threshold), and β' = 0.9, with at least three seeds each; then repeat with a small from-scratch pretraining run (e.g., ~1B tokens on a small LM) at β' = 0.5 vs 0.9. If β' = 0.9 matches β' = 0.5, Eq. 15's adjustment is not load-bearing, and if the reduction only helps fine-tuning, the 'seamless' claim is conditional on training regime.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3's signed-momentum fix is the load-bearing step for the 2-bit and from-scratch claims. Theorem 3.6 only upper-bounds the quantization-induced variance of an effective gradient; Eq. 15 then asserts that keeping β'/(1−β')·r_median(b') ≤ β/(1−β)·r_median(b) preserves convergence. The paper states this as a hypothesis ('we hypothesize that an acceptable convergence rate could be achieved once the upper bound is somewhat regulated'), and it is not derived from any optimizer convergence theorem. The stated justification for using r_median instead of r_max is also internally contradicted by the paper's own Fact A.1: for signed DE, r_med(8)/r_med(2) = 0.004/0.275 ≈ 0.0145, whereas r_max(8)/r_max(2) = 0.007/0.275 ≈ 0.0255, a 1.76× difference, so the 'nearly identical ratio' claim fails precisely in the 2-bit regime. In addition, reducing β from 0.9 to 0.5 shortens the EMA window and increases the variance of the moment estimate itself, an effect absent from Eq. 14; Figure 6 shows 0.5 is an empirical optimum rather than a consequence of Eq. 15. If this variance-regulation hypothesis fails in settings beyond those tested (Section 5 explicitly leaves large-scale pretraining unverified), the seamless 2-bit claim collapses. The empirical coverage is broad and the code is released, but these do not supply the missing convergence argument for the β' recipe.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SOLO, a low-bit optimizer-state quantization method that aims to maintain Adam-style optimizer states at 3-bit or even 2-bit precision with minimal accuracy loss. The authors identify two problems with naive low-bit quantization: signal swamping in unsigned EMA updates of the second moment, and increased gradient variance in signed EMA updates of the first moment. They address swamping with a logarithmic quantization scheme whose base is set by a p-quantile and which admits a simple stochastic-rounding implementation with an exact exponential-decay property. They address the gradient-variance issue by proposing a reduced first-moment momentum β' selected through an inequality (Eq. 15) that compares quantization radii at different precisions. The method is evaluated across CV, NLP, recommender systems, LLM instruction tuning, and LVM visual instruction tuning, with the 4/2-bit variant and a fully 2-bit variant compared against 32-bit, 16-bit, 8-bit, and 4-bit AdamW baselines. The paper also reports a robustness study of block-size choice and p-quantile sensitivity.","tokens_in":27575,"tokens_out":9233,"duration_ms":87997,"significance":"If the results hold, SOLO is a significant advance: it would reduce optimizer memory by roughly 10x relative to 32-bit AdamW while keeping performance close to full precision, and it is orthogonal to model architecture and distributed training. The unsigned logarithmic quantization with the p-quantile base is a clean and useful contribution, particularly because it preserves state dynamics under sparse-gradient and zero-signal conditions. The paper is transparent about its limitations, explicitly labels the momentum-reduction rule as a hypothesis, releases code, and reports broad experiments including LLaMA-13B/33B and LLaVA-1.5. The main weakness is that the load-bearing recipe for β'—the signed-momentum adjustment that makes 2-bit states workable—is a heuristic whose stated justification contains a numerically false 'nearly identical' ratio claim and whose variance analysis omits the EMA estimator's own variance increase when β is reduced. These issues are fixable but require substantive revision.","major_comments":[{"comment":"The assertion that the ratio r*(b)/r*(b') is 'nearly identical' for the median and maximum radius is contradicted by the paper's own Fact A.1 for signed DE quantization. For example, using the reported values, r_med(5)/r_med(2)=0.034/0.275≈0.124, while r_max(5)/r_max(2)=0.056/0.275≈0.204, a difference of roughly 1.65x. Since Theorem 3.6 bounds the quantization-induced variance in terms of r_max, replacing r_max by r_median in Eq. (15) is not a cosmetic change; the 'upper bounds' shown in Figure 4 are not consequences of the theorem. The authors should either re-derive the inequality with r_max and adjust the recommended β' values, or explicitly present Eq. (15) as an empirical heuristic and remove the 'nearly identical' claim.","section":"Section 3.3, Eq. (15), Fact A.1, Figure 4"},{"comment":"The variance analysis in Eq. (14) accounts only for the quantization-induced noise term and omits the effect of reducing β on the EMA estimator's own variance. For a stationary gradient with variance σ², the first-moment estimate has variance proportional to (1-β)/(1+β); reducing β from 0.9 to 0.5 increases this component by roughly a factor of 6, which can offset the reduction in the quantization-noise term. The empirical optimum at β'≈0.5 in Figure 6 therefore does not follow from Eq. (14). The authors should provide a complete variance tradeoff for the momentum choice, including the estimator-variance term, or explicitly rephrase the Section 3.3 rationale as empirically motivated rather than derived.","section":"Section 3.3, Eq. (14), Figure 6"},{"comment":"Theorem 3.6's unbiasedness result, Eq. (13), relies on stochastic rounding, as the proof notes, but the manuscript does not state whether the signed DE quantization used in the experiments employs stochastic rounding. If nearest rounding is used, E[\\tilde{g}] ≠ E[g] and the variance bound in Eq. (14) is not directly applicable. Please state the rounding scheme used for signed states; if it is not stochastic rounding, extend the analysis to cover the resulting bias or provide empirical evidence that the bias is negligible for the reported settings.","section":"Section 3.3, Theorem 3.6"},{"comment":"Several 2-bit results show visible drops relative to 32-bit AdamW that are not discussed: HSTU HR 32.73→32.03, LLaVA ScienceQA 70.81→69.51, and LLaMA SIQA 48.50→47.21. The Wilcoxon signed-rank tests are based on only five metrics and are underpowered, so 'no statistically significant difference' (p=0.125 and p=0.285) should not be presented as evidence of equivalence. Please discuss these exceptions explicitly and soften the 'minimal accuracy loss' phrasing for the 2-bit setting, or provide additional runs and confidence intervals to support the claim.","section":"Tables 1-2, Section 4.1"}],"minor_comments":[{"comment":"The phrase 'ap-vlue' should read 'p-value'.","section":"Table 2 captions"},{"comment":"The notation ⌊·⌉ for convergent rounding is used without definition in the main text; please define it where Eq. (10) is introduced.","section":"Eq. (10)"},{"comment":"The caption and subfigure text are dense and difficult to parse; consider splitting the figure into two panels with clearer labels for the quantization levels and the p-quantile mechanism.","section":"Figure 3 caption"},{"comment":"The word 'utlize' should be 'utilize'.","section":"Appendix C.1, Table 9"},{"comment":"If Eq. (15) is a heuristic rather than a proven upper bound, the figure caption should state this explicitly to avoid overstating the theoretical grounding.","section":"Section 3.3, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The empirical breadth and released code make this paper a strong candidate for a machine-learning journal, and the unsigned logarithmic quantization is a genuine contribution. The main obstacle is the signed-momentum recipe: it is central to the 2-bit claim, but its stated theoretical justification contains a false ratio assertion and an incomplete variance tradeoff. The revision should fix or re-scope Eq. (15), add the missing estimator-variance analysis, and ideally test the 2-bit claim on a small pretraining run (e.g., a 1B-scale model) to address the acknowledged gap beyond fine-tuning. With those changes the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for the 4/2-bit scheme. The analysis of why unsigned EMA states get swamped and why signed states blow up variance is genuinely new, and the log-scale quantization with data-dependent base is a credible fix. The empirical spread is wide: ImageNet, WMT, GLUE, Criteo, MovieLens, LLaMA fine-tuning, LLaVA. The 4/2-bit variant matches 32-bit within noise, and the block-size robustness result is convincing. Code is released. The theory in Section 3 is internally coherent as far as I checked; the appendix proofs are standard and go through.\n\nThe soft spots are mostly around the 2-bit story. The signed-momentum reduction rule, Eq. 15, is explicitly labeled a hypothesis, and the paper admits no convergence theorem for the reduced-β regime. That would be fine if the empirical case were airtight, but it isn't: HSTU HR drops 32.73→32.03, LLaVA ScienceQA 70.81→69.51, LLaMA SIQA 48.50→47.21. Some of these are single runs, and none carry error bars. The stress-test note is right that the 'nearly identical ratio' justification for r_median vs r_max is contradicted by Fact A.1 in the 2-bit signed DE case: r_med ratio is 0.0145 vs r_max 0.0255, a 1.76x difference. That doesn't kill the 4/2-bit result, but it means the 2-bit recommendation rests on a heuristic that the paper's own numbers undermine. The p-quantile and β' choices are tuned; the sensitivity analysis shows a forgiving range for p, but β'=0.5 for 2-bit is an empirical peak, not a derivation.\n\nNone of this is fatal for the main contribution. The 4/2-bit memory savings (roughly 10x on optimizer state) are real and broadly demonstrated. The 2-bit variant is a promising direction, but the 'minimal accuracy loss' claim is over-stated for the current evidence.\n\nFor a referee: yes, this deserves serious review. The right revision path is to either (a) prove the convergence argument for the β' rule, or (b) downgrade the 2-bit claims to preliminary and let the 4/2-bit carry the paper. I'd cite the 4/2-bit result if I were doing efficient fine-tuning.","headline":"SOLO's 4/2-bit optimizer is a solid, well-tested contribution; the 2-bit claim leans on a heuristic momentum rule that the paper itself doesn't fully support.","tokens_in":28096,"tokens_out":1625,"would_cite":true,"duration_ms":15206,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper demonstrates that Adam-style optimizers can keep their state in as few as 2-3 bits per value without meaningful accuracy loss, by fixing two EMA-specific quantization failures: signal swamping in unsigned states and inflated…","keywords":["low-bit optimizers","Adam","quantization","EMA dynamics","signal swamping","gradient variance","stochastic rounding","memory-efficient training"],"falsifier":"Run the recommended 2-bit settings ($\\beta'=0.527$ for fine-tuning, lower for scratch) on a large-scale pretraining run the paper leaves untested, e.g., a 175B model; if loss diverges or the accuracy gap exceeds the fine-tuning margins, the variance-regulation hypothesis fails. A cheaper check: directly measure gradient variance under 2-bit signed quantization and compare it with the bound in Eq. (14) and the claimed $r_{\\mathrm{median}}/r_{\\max}$ ratio similarity.","tokens_in":26978,"feed_emoji":"💾","tokens_out":13776,"duration_ms":116318,"temperature":0.7,"pith_summary":"SOLO claims that Adam-style optimizers can store their state in as few as 3 or even 2 bits per value instead of 32, with only marginal accuracy loss. The reason previous 4-bit attempts degraded is that they ignored the exponential moving average (EMA) dynamics: nearest-rounding quantization freezes the unsigned second-moment state (\"signal swamping\"), and signed first-moment quantization inflates gradient variance enough to reverse descent. The fix is a logarithmic quantizer with stochastic rounding for the unsigned state, plus a smaller momentum hyperparameter for the signed state, chosen from a variance bound. On benchmarks spanning computer vision, translation, recommendation, and LLM/LVM fine-tuning, the 4/2-bit variant is statistically indistinguishable from full-precision Adam (Wilcoxon p=0.125 on MMLU-style tasks), and a fully 2-bit version remains competitive. If correct, this cuts optimizer memory by roughly an order of magnitude, making large-model training feasible on much smaller GPU budgets.","feed_headline":"Adam optimizer runs at 2-bit precision with minimal accuracy loss","feed_subtitle":"SOLO slims optimizer memory to a tenth, freeing GPUs for bigger models and batches.","key_machinery":"The load-bearing object is the quantized EMA update: $\\tilde{x}_t = Q^\\dagger(q_t)$, then $\\hat{x}_{t+1} = \\beta\\tilde{x}_t + (1-\\beta)z_{t+1}$, then $q_{t+1}=Q(\\hat{x}_{t+1})$, where $z_{t+1}$ is the gradient (signed) or squared gradient (unsigned). Two results extracted from this loop do the work. Theorem 3.1 states that the quantized level stays fixed whenever the level radius satisfies $r \\ge (1-\\beta)|\\frac{z_{t+1}}{\\Delta_{t+1}} - y_{q_t}| + |\\frac{\\Delta_t}{\\Delta_{t+1}}-1|$, which identifies why standard quantizers fail at low precision and large tensor size. Theorem 3.6 proves that signed-state quantization inflates gradient variance by at most $\\big(\\frac{\\beta}{1-\\beta} r_{\\max}(b)\\Delta_t\\big)^2$, which motivates the $\\beta'$ adjustment rule of Eq. (15) using $r_{\\mathrm{median}}$. The closing piece is the logarithmic quantizer of Eq. (10), $Q(x)=\\mathrm{Clip}(\\lfloor \\log_\\alpha(x/\\Delta)+\\xi\\rceil;0,2^b-1)$ with $\\alpha=(x_p/\\Delta)^{1/(2^b-1)}$ and $\\xi\\sim U[-0.5,0.5]$, which prevents swamping and matches EMA decay.","core_discovery":"The paper's central claim is that ultra-low-bit quantization of Adam is possible when the EMA state update is treated as a dequantize-EMA-quantize loop, and two mechanisms are added. For the unsigned second moment, Theorem 3.1 gives a precise signal-swamping condition: if the quantization radius around the current level is large enough, the state stays constant even when the incoming signal differs sharply; the paper shows this occurs at momentum values as low as $\\beta \\approx 0.833$ for 2-bit linear quantization. Its remedy is a logarithmic quantizer with levels clustered near zero, base $\\alpha = (x_p/\\Delta)^{1/(2^b-1)}$, and stochastic rounding, which also reproduces the exact expected decay time under consecutive zero signals. For the signed first moment, Theorem 3.6 bounds the added gradient variance by $\\big(\\frac{\\beta}{1-\\beta} r_{\\max}(b)\\Delta_t\\big)^2$, and SOLO chooses a lower momentum $\\beta'$ so that $\\frac{\\beta'}{1-\\beta'} r_{\\mathrm{median}}(b') \\le \\frac{\\beta}{1-\\beta} r_{\\mathrm{median}}(b)$, giving $\\beta' \\le 0.820$ for 4-bit fine-tuning and $\\beta' \\le 0.527$ for 2-bit. The resulting 4/2-bit AdamW matches 32-bit AdamW within statistical noise on the evaluated tasks.","pith_inferences":["This suggests the momentum-reduction recipe could carry over to other EMA-based optimizers such as Lion or Adafactor, since the variance bound comes from the EMA loop rather than from Adam's specifics.","The signal-swamping condition implies a general design rule for low-bit states that feed a denominator: allocate quantization levels near zero and use stochastic rounding; this could inform low-bit batch-normalization statistics or embedding-table scalers.","The p-quantile base selection appears transferable to fully quantized training of weights and activations, where small-magnitude values dominate, but this would require independent validation.","The reported equivalence holds on only five MMLU-style benchmarks; extending the comparison to a larger suite with repeated seeds would test whether the \"no significant difference\" claim generalizes."],"forward_implications":["Optimizer memory for Adam drops to about one-eighth (4/2-bit) or one-sixteenth (2-bit) of the 32-bit state size, letting practitioners fit larger models or batches on the same hardware.","Existing learning rates and weight decay remain unchanged; only the first-moment momentum needs lowering, to 0.8 for 4/2-bit and 0.5 for 2-bit fine-tuning, and to 0.3/0.1 for training from scratch.","The quantization scheme is model- and task-agnostic in the tested regimes: it transfers across Swin-T, Transformer-Base, RoBERTa-L, DCN, HSTU, LLaMA-7B/13B/33B, and LLaVA-1.5, and also covers the AdaBelief optimizer.","The logarithmic quantizer preserves second-moment dynamics when the block size is increased from 128 to 2048, unlike linear or dynamic-exponent quantizers that collapse, so per-block scale-factor overhead can be nearly eliminated."],"supporting_citations":[{"why":"Defines Adam, the optimizer whose states SOLO quantizes; the EMA update rule is the object of analysis.","marker":"Kingma & Ba, 2015"},{"why":"Defines AdamW, the base optimizer for the 4/2-bit and 2-bit variants used in all experiments.","marker":"Loshchilov & Hutter, 2017"},{"why":"Establishes 8-bit block-wise optimizers, the prior state of the art that SOLO extends to 2-bit.","marker":"Dettmers et al., 2022"},{"why":"Proposes 4-bit optimizers with linear/DE quantization; SOLO identifies their failure to handle EMA dynamics.","marker":"Li et al., 2023a"},{"why":"Provides stochastic rounding, which SOLO adapts to escape signal swamping in unsigned states.","marker":"Xia et al., 2020"},{"why":"Introduces logarithmic quantization that SOLO tailors with a p-quantile base for 2nd-moment states.","marker":"Oh et al., 2021"},{"why":"Formalizes the swamping problem in floating-point summation, which the paper generalizes to quantized EMA updates.","marker":"Higham, 1993"},{"why":"Shows reducing gradient variance aids convergence; cited to justify the momentum-reduction fix.","marker":"Li et al., 2023b"},{"why":"Analyzes convergence of Adam under non-uniform smoothness, supporting the claim that variance control preserves convergence.","marker":"Wang et al., 2024"}],"fun_headline_variants":["2-bit Adam: SOLO solves EMA dynamics","SOLO: Adam at 2 bits with minimal loss","Ultra-low-bit Adam: SOLO's logarithmic fix","Adam in 2 bits: SOLO tames the EMA","SOLO: memory-light Adam at 2-bit precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the hypothesis that bringing the quantization-induced gradient-variance upper bound under control is enough to keep convergence acceptable: the paper states this as a hypothesis rather than a theorem, and the recommended $\\beta'$ values are chosen from empirical observation on the evaluated tasks, not from a convergence proof.","fun_headline_variants_meta":{"raw":{"variants":["2-bit Adam: SOLO solves EMA dynamics","SOLO: Adam at 2 bits with minimal loss","Ultra-low-bit Adam: SOLO's logarithmic fix","Adam in 2 bits: SOLO tames the EMA","SOLO: memory-light Adam at 2-bit precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1888,"prompt_tokens":1032,"completion_tokens":856,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":776}},"tokens_in":648,"tokens_out":856,"duration_ms":8408,"temperature":1.0,"reasoning_tokens":776,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:44:38.665944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the recommended 2-bit settings ($\\beta'=0.527$ for fine-tuning, lower for scratch) on a large-scale pretraining run the paper leaves untested, e.g., a 175B model; if loss diverges or the accuracy gap exceeds the fine-tuning margins, the variance-regulation hypothesis fails. A cheaper check: directly measure gradient variance under 2-bit signed quantization and compare it with the bound in Eq. (14) and the claimed $r_{\\mathrm{median}}/r_{\\max}$ ratio similarity.","supporting_citations":[],"review_version":1}