{"id":"f4fabc62-22e3-4a87-9aa1-3037bfecb149","arxiv_id":"2412.02153","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Initializing Adam's second-order moment estimate to a non-zero value, rather than the default zero, reduces early training instability and improves generalization across several deep learning tasks.","lead":"This paper finds that a tiny detail in Adam, initializing its second-moment estimate to zero, makes the first update behave like a sign-only step and can destabilize training. The authors show that simply starting this estimate at a non-zero value, based on gradient statistics or random values, makes Adam train more stably and often improves final accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sec 2.3's drift analysis is contradicted by the recommended random initialization: the inequality guaranteeing reduced drift requires λ<2(ḡ²+σ²), but σ=100 yields λ≈0.1 which fails for realistic gradient scales, so the stated theory cannot explain the observed improvements.","rationale":"The paper's central claim has two parts: an empirical observation (v0 non-zero improves stability/performance) and a proposed mechanism (drift reduction / avoiding sign descent). The empirical part is supported by broad experiments. The mechanism, however, is the load-bearing link that lets the authors 'show' v0=0 is a significant factor. Section 2.3's drift analysis is the only theoretical support, and it contains an internal inconsistency: the inequality that guarantees improved stability for random initialization is violated by the σ=100 configuration used in all experiments. Unless the gradient second moment at initialization is implausibly large (≈0.05 per coordinate), the paper's own criterion predicts v0,rnd should be worse than vanilla Adam. The fact that it is consistently better means either the drift metric is not a meaningful stability measure or the improvement is driven by a different mechanism (e.g., lowering early effective step size). Both possibilities undermine the stated causal explanation. The reader's stationarity concern is related: it also targets Sec 2.3, but it questions the constancy of gradient statistics; the concern raised here is sharper because it does not depend on non-stationarity and shows a contradiction even in the idealized model. I would keep the CONDITIONAL verdict: the empirical contributions are valuable and reproducible, but the paper should either correct the drift analysis to account for the actual λ and gradient scales, or clearly reframe the theoretical section as a heuristic and attribute the gains to early-step size reduction. A direct computation of λ vs 2μ on the released code would settle which of these is the case.","tokens_in":17416,"tokens_out":13642,"duration_ms":136736,"concrete_test":"Using the released code, instantiate the CIFAR-10 ResNet-34 setup, compute per-coordinate μ=E[g²] at initialization (e.g., with 5,000 samples as described), and for every weight tensor used in v0,rnd evaluate whether σ/(fan_in+fan_out) is less than 2μ. If the inequality fails for the large majority of tensors while v0,rnd still improves test accuracy over v0=0, the Sec 2.3 drift argument is contradicted by the recommended hyperparameters. Report the fraction of tensors satisfying the inequality and the resulting drift comparison.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2.3 defines drift_vt(v0)=||E[v∞]-E[v0]|| and claims that for random initialization v0=λI, the drift is smaller than for v0=0 exactly when 0<λ<2(ḡ²+σ²). The paper then recommends v0,rnd ~ χ²₁·σ/(fan_in+fan_out) with σ=100 (Eq. 11). For any typical weight tensor, fan_in+fan_out≥512, so the mean of v0 is λ≈0.1–0.2 (and typical samples are the same order). The targeted regime is one of small gradients ('gradients are correspondingly small', Sec 2.2); even a generous per-coordinate second moment of 1e-2 would require λ<0.02, which the recommended λ≈0.1 violates. Thus the paper's own drift inequality predicts that v0,rnd should be less stable than vanilla Adam, yet Tables 1–5 show consistent improvements. The same issue applies to v0,data if the initial gradient scale is small. This is not a minor approximation: the theoretical mechanism advertised in Sec 2.3 cannot explain the empirical success of the very configurations used in the experiments. Either the drift metric is not a valid predictor of stability, or the benefit arises from a different effect (e.g., a sustained reduction of early effective step size). Either way, the central theoretical justification is internally inconsistent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that Adam's default initialization of the second-order moment estimate, v0 = 0, causes the first update to degenerate into pure sign descent with step magnitude equal to the learning rate, and that this destabilizes training and degrades final performance in deep networks. It proposes two non-zero initialization schemes for v0: a data-driven one based on empirical gradient statistics (Eq. 10) and a random one based on a scaled chi-squared distribution (Eq. 11). The authors support the proposal with a linear-loss Gaussian-oracle model and a drift metric (Sec. 2.3), and report improvements across CIFAR-10, ImageNet, Penn Treebank, IWSLT'14, and CIFAR-10 GAN experiments for Adam, AdamW, AdaBound, RAdam, and AdaBelief.","tokens_in":17745,"tokens_out":10271,"duration_ms":104576,"significance":"If the empirical findings hold, this is a simple, computationally cheap modification with broad applicability: it requires changing only the initialization of the optimizer state and improves stability and final performance across multiple architectures and optimizers. The paper's strengths are its extensive empirical evaluation (five seeds, several tasks, several optimizers, an ablation over the scale parameter) and the release of code. The theoretical analysis, however, is heuristic and contains an internal inconsistency: the recommended random-initialization scale appears to violate the paper's own drift-reduction condition in the small-gradient regime it describes. The empirical results are valuable, but the theoretical framing needs substantial correction before the paper is publishable in its current form.","major_comments":[{"comment":"The recommended random initialization contradicts the drift-reduction condition stated just above it. For v0 = λI, the paper derives that drift(v0 = λI) < drift(v0 = 0) iff 0 < λ < 2(gbar^2 + σ^2), where gbar^2 + σ^2 is the per-coordinate steady-state second moment of the gradient. With Eq. (11) and σ = 100, the mean of v0 is λ = 100/(fan_in + fan_out); for a 512×512 weight matrix this is about 0.098. But Sec. 2.2 motivates the problem precisely by saying the initial Transformer landscape is flat and 'gradients are correspondingly small,' implying that the per-coordinate gbar^2 + σ^2 is far below 0.05. Under the paper's own inequality, v0_rnd would then have a larger drift than zero initialization, not a smaller one, which is the opposite of the results in Tables 1–5. The ablation in Table 6 strengthens the contradiction: σ = 1000 (λ ≈ 1) still improves over σ = 0. This is not a minor approximation; either the drift metric is not the operative mechanism, or the recommended scale is in the wrong regime. The authors should measure the actual per-coordinate gradient second-moment scale and reconcile Eq. (11) with the inequality, or explicitly reclassify the drift analysis as heuristic.","section":"Sec. 2.3, Eq. (11)"},{"comment":"The steady-state argument assumes the stochastic gradient is generated by a stationary oracle with constant mean gbar and covariance σ^2I. Deep-network training is non-stationary: the weights change, and the distribution of g_t changes with them. Consequently, the claim that v0_data 'keeps E[vt] constant' and 'eliminates the need for adjustments' from the start is not established for the networks in Sec. 3; it holds only if the initial gradient statistics remain representative of the whole trajectory, which is not shown. This is load-bearing because the theoretical justification for both proposals rests on this transfer. I recommend either tracking per-coordinate second-moment statistics through training (e.g., comparing v0 to the running v_t) or presenting Eqs. (6)–(9) explicitly as intuition for a simplified model, not as a proof for deep networks.","section":"Sec. 2.3, Eqs. (6)–(7)"},{"comment":"The drift metric is introduced as 'reflecting how much the optimizer must adapt' and 'a smaller drift term indicates better stability,' but no derivation or empirical correlation links drift to update-step variance, loss spikes, or final performance. Since v0 = gbar^2 + σ^2I makes drift exactly zero by construction, the superiority of the data-driven rule in this metric is built in; as a theoretical justification it is circular. The experiments verify the proposal independently, but they do not test the drift metric itself. I would like to see either a direct test (e.g., compare drift values across v0 settings against measured instability) or a modest reformulation of the contribution as an empirical finding with a heuristic explanation.","section":"Sec. 2.3, Eq. (8)"}],"minor_comments":[{"comment":"The statement that the standard deviation of each coordinate of vt is of scale O((1−β2)σ^2) misses a square-root factor: for an exponential moving average with smoothing (1−β2), the fluctuation scale is O(sqrt(1−β2)·sd(g_t^2)), not O((1−β2)σ^2). The concentration conclusion still holds for β2 close to 1, but the stated scale is wrong.","section":"Sec. 2.3 (after Eq. (8))"},{"comment":"The symbol σ is overloaded: it denotes the gradient noise covariance scale in Eq. (7) and a user-set multiplier in Eqs. (10) and (11). This makes the condition 0 < λ < 2(gbar^2 + σ^2) appear to involve the same σ as in Eq. (11), which it does not.","section":"Secs. 2.3 and 3"},{"comment":"The pseudocode divides accumulated gradients by num_samples = len(dataloader.dataset), which is inconsistent with the text's statement that only 5,000 random samples are used. Moreover, if the dataloader uses batches of size B, param.grad is the batch-averaged gradient, so the pseudocode estimates (1/B)·E[g^2], not E[g^2] as claimed in Eq. (10).","section":"Appendix B, Algorithm 2"},{"comment":"There are several typos: 'IWSTL' in the caption of Table 4, 'dataset dataset' in the caption of Table 5, and 'are are non linearly connected' in Appendix C.5.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution appears solid and within scope, but the theoretical section currently overclaims and contains a specific internal contradiction between the recommended random-initialization scale and the stated drift-reduction condition. I would be comfortable with acceptance after a revision that (a) fixes or explicitly downgrades the drift analysis, (b) reconciles the random-initialization scale with the stated condition or removes that condition, and (c) clarifies the data-driven implementation in the pseudocode."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — here's the short version: this paper reports a cheap, broadly effective trick — initialize Adam's second-moment estimate v0 to a non-zero data-driven or random value — and backs it with consistent gains across five optimizers and four tasks. The first-step sign-descent observation (Eq. 5) is correct and already partly in Balles & Hennig and Kunstner et al.; the new bit is the systematic evaluation of non-zero v0 initialization and the two concrete schemes. Credit where due: the experiments are honest, five seeds, reasonable baselines, and the gains are real (e.g., Adam on IWSLT BLEU from 30.1 to 33.7; consistent but smaller gains elsewhere). The ablation on σ shows robustness from 1 to 1000, so the trick doesn't require careful tuning.\n\nThe soft spots are in the theory, not the data. The drift analysis in Sec 2.3 is ad hoc: defining drift as ||E[v∞]-E[v0]|| makes the ideal v0=E[g²] optimal by construction, which is circular. More seriously, the random initialization recommended with σ=100 gives mean v0 ≈ σ/(fan_in+fan_out) ≈ 0.1 for typical layers. The paper's own inequality says random init reduces drift only if 0 < λ < 2(ḡ²+σ²), and the paper also says gradients at initialization are small. For ḡ²+σ² ~ 1e-2 (generous), the bound is λ < 0.02, so the recommended λ≈0.1 violates it. That means the advertised theoretical mechanism cannot explain the empirical success of the very configuration tested. The same issue affects the standard-deviation claim: they state the std of vt is O((1-β2)σ²), missing a sqrt factor; it's actually O(σ²√(1-β2)). These are fixable — the likely real explanation is that a non-zero v0 simply reduces the early effective step size, which is exactly the sign-descent observation, and the drift metric is unnecessary.\n\nVerdict: the paper is worth refereeing. The empirical finding is useful and easy to check, and the proposed init could reduce dependence on warmup. But the authors should either fix Sec 2.3 or cut the drift theory and present the step-size argument alone. I'd recommend a conditional accept after revision, not a desk reject.","headline":"A simple, empirically solid v0-initialization trick for Adam with a theory section that doesn't hold up — worth refereeing for the experiments, but the drift analysis needs to go.","tokens_in":18243,"tokens_out":5074,"would_cite":true,"duration_ms":49201,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adam's default zero initialization of the second-order moment makes its first update a pure sign-descent step; replacing that zero with a data-driven or random positive value stabilizes training and improves final performance.","keywords":["Adam optimizer","second-order moment initialization","sign descent","adaptive gradient methods","training stability","Transformer optimization","learning rate warmup"],"falsifier":"Ignoring the $\\epsilon$ term, the mechanism predicts that with $v_0=0$ every first-step coordinate update has magnitude $\\alpha$ (Eq. 5), while with non-zero $v_0$ every first-step coordinate satisfies $|\\Delta\\theta_1|<\\alpha$ (Eq. 12); finding a single coordinate with $|\\Delta\\theta_1|>\\alpha$, or observing identical first-step distributions under $v_0=0$ and $v_{0,\\mathrm{data}}$, would falsify the claimed mechanism.","tokens_in":17214,"feed_emoji":"⚙️","tokens_out":16639,"duration_ms":133818,"temperature":0.7,"pith_summary":"The paper argues that Adam's default initialization of the second-order moment estimate ($v_0=0$) is not harmless: it makes the first update a pure sign-descent step whose per-coordinate magnitude is the learning rate, which is disproportionately large when early gradients are small. The proposed fix is to set $v_0$ to a positive value, either computed from gradient statistics on a small data sample ($v_{0,\\mathrm{data}}$) or drawn from a scaled chi-squared distribution ($v_{0,\\mathrm{rnd}}$). Across image classification, language modeling, machine translation, and GAN training, this change stabilizes the early phase and improves final accuracy, perplexity, BLEU, and FID for Adam and several adaptive optimizers. The practical consequence is that one default can be replaced by a cheap initialization, often matching or beating recent Adam variants and learning-rate warmup.","feed_headline":"Adam's first step is a blind sign jump; changing one default fixes it","feed_subtitle":"The default v0=0 makes Adam's first update a pure sign step; data-driven or random v0 removes the instability.","key_machinery":"The load-bearing object is the second-order moment estimate $v_t$ and its initial value $v_0$ in Adam's update (Eq. 2). The mechanism is the first-step collapse in Eq. 5: because $v_0=0$, the bias-corrected $\\hat v_1$ is just $g_1^2/(1-\\beta_2)$, so $\\Delta\\theta_1=-\\alpha\\,\\operatorname{sign}(g_1)$. The paper quantifies the disturbance with the drift term $\\mathrm{drift}_{v_t}(v_0)=\\|\\mathbb{E}[v_\\infty]-\\mathbb{E}[v_0]\\|$ (Eq. 8), which is large for $v_0=0$ and zero for the ideal $v_0=\\bar g^2+\\sigma^2 I$ in the linear-loss Gaussian oracle; non-zero $v_0$ removes the sign-descent degeneracy at step one and keeps $v_t$ at the right scale from the start.","core_discovery":"With $m_0=v_0=0$, the first Adam update collapses to $\\Delta\\theta_1=-\\alpha\\,\\operatorname{sign}(g_1)$ (Eq. 5): every coordinate is moved by the learning rate $\\alpha$ regardless of gradient magnitude. In deep networks, especially Transformers, early gradients are small, so this sign-descent step is overly large and pushes parameters out of the initial flat region. Initializing $v_0$ to the expected squared gradient plus variance (Eq. 10) or to a scaled chi-squared random value (Eq. 11) makes the first step depend on gradient magnitude, keeps $|\\Delta\\theta_1|<\\alpha$, and reduces the drift of $v_t$ toward its steady state. The paper shows empirically that this change stabilizes and improves Adam, AdamW, AdaBound, RAdam, and AdaBelief on CIFAR-10 and ImageNet classification, Penn Treebank LSTM language modeling, IWSLT'14 DE-EN Transformer translation, and CIFAR-10 DCGAN generation, and that it can replace or complement warmup.","pith_inferences":["The paper does not test it, but the same first-step argument suggests that initializing the first moment $m_0$ away from zero, or setting the random scale per layer from measured gradient variances, could close the remaining gap between warmup and non-warmup training.","Because $v_{0,\\mathrm{rnd}}$ injects random scale into each weight's update, it may act as a mild implicit regularizer; that could explain the flatter minima and the lack of linear mode connectivity, though the paper does not isolate this effect.","If the stationarity assumption holds even approximately, data-driven $v_0$ could be computed once and reused across runs or related tasks as a cheap calibration step; this is an extension beyond the paper's claims."],"forward_implications":["With non-zero $v_0$, Adam can train the IWSLT'14 Transformer without a warmup phase, whereas vanilla Adam without warmup fails to converge.","On ResNet-34/CIFAR-10, Adam with $v_{0,\\mathrm{data}}$ raises test accuracy from 95.25% to 96.02%, and on IWSLT'14 DE-EN it raises BLEU from 30.14 to 33.64; similar gains appear for AdamW, AdaBound, RAdam, and AdaBelief.","The scale factor $\\sigma$ is tuning-friendly: for both initialization strategies, values in $[1,1000]$ beat the zero baseline in the CIFAR-10 ablation.","Switching $v_0$ changes the solution basin: networks trained with $v_0=0$ and $v_{0,\\mathrm{rnd}}$ are not linearly connected, and the $v_{0,\\mathrm{rnd}}$ solution sits in a flatter loss landscape."],"supporting_citations":[{"why":"Supplies the Adam update rule and the default $m_0=v_0=0$ that the paper modifies.","marker":"[4]"},{"why":"Gives the sign/magnitude dissection of Adam used to derive the first-step sign-descent identity.","marker":"[18]"},{"why":"Provides prior evidence that sign descent, not noise, explains Adam's behavior on Transformers, motivating the instability analysis.","marker":"[19]"},{"why":"RAdam is both a related stabilization fix and a baseline optimizer that the proposed $v_0$ initialization improves.","marker":"[11]"},{"why":"AdaBelief is a recent adaptive optimizer used as a baseline and improved by non-zero $v_0$.","marker":"[13]"},{"why":"Supplies the Transformer architecture and the warmup strategy compared against; also the NMT task where zero $v_0$ fails without warmup.","marker":"[17]"},{"why":"Provides the Xavier initialization logic that motivates the scaled chi-squared random $v_0$.","marker":"[30]"},{"why":"ResNet is the backbone for the CIFAR-10 and ImageNet classification experiments.","marker":"[32]"}],"fun_headline_variants":["Zero v0 makes Adam's first step a blind sign jump; new init fixes it","Set v0 nonzero: a one-line change stabilizes Adam and matches recent variants","v0=0 causes oversized Adam update; data-driven init outperforms warmup","Adam's first update is a magnitude-blind sign step; changing v0 solves it","Don't start Adam's second moment at zero: it improves Transformer training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes the average size of the gradients stays about the same throughout training, so a $v_0$ chosen from early samples remains the right scale for the whole run.","fun_headline_variants_meta":{"raw":{"variants":["Zero v0 makes Adam's first step a blind sign jump; new init fixes it","Set v0 nonzero: a one-line change stabilizes Adam and matches recent variants","v0=0 causes oversized Adam update; data-driven init outperforms warmup","Adam's first update is a magnitude-blind sign step; changing v0 solves it","Don't start Adam's second moment at zero: it improves Transformer training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001302,"raw_usage":{"total_tokens":5306,"prompt_tokens":933,"completion_tokens":4373,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":4268}},"tokens_in":549,"tokens_out":4373,"duration_ms":31795,"temperature":1.0,"reasoning_tokens":4268,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:47:02.473251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ignoring the $\\epsilon$ term, the mechanism predicts that with $v_0=0$ every first-step coordinate update has magnitude $\\alpha$ (Eq. 5), while with non-zero $v_0$ every first-step coordinate satisfies $|\\Delta\\theta_1|<\\alpha$ (Eq. 12); finding a single coordinate with $|\\Delta\\theta_1|>\\alpha$, or observing identical first-step distributions under $v_0=0$ and $v_{0,\\mathrm{data}}$, would falsify the claimed mechanism.","supporting_citations":[{"cited_title":"Dissecting adam: The sign, magnitude and variance of stochastic gradients","cited_arxiv_id":null,"evidence_quote":"Gives the sign/magnitude dissection of Adam used to derive the first-step sign-descent identity."},{"cited_title":"Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be","cited_arxiv_id":null,"evidence_quote":"Provides prior evidence that sign descent, not noise, explains Adam's behavior on Transformers, motivating the instability analysis."},{"cited_title":"Onthevarianceoftheadaptivelearningrateandbeyond","cited_arxiv_id":null,"evidence_quote":"RAdam is both a related stabilization fix and a baseline optimizer that the proposed $v_0$ initialization improves."},{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"ResNet is the backbone for the CIFAR-10 and ImageNet classification experiments."}],"review_version":1}