{"id":"170ae091-83fb-44ca-a8e9-dcf94f8c8ce9","arxiv_id":"2506.17768","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LMD, a log-normal multiplicative-weight optimizer, trains ViT and GPT-2 from scratch and keeps accuracy under MXFP6 forward-pass quantization.","lead":"LMD is a new optimizer that trains neural networks by multiplying weights instead of adding to them, based on noisy multiplicative dynamics seen in biological synapses. In tests, it trained a Vision Transformer and GPT-2 from scratch and kept accuracy when forward computations were compressed to the MXFP6 low-precision format.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-2 low-precision comparison with AdamW is confounded by sequence length, so the claim that AdamW cannot train stably under MXFP6 forward passes is not established.","rationale":"The paper's central empirical claim — that a multiplicative optimizer can train ViT and GPT-2 from scratch and maintain performance under MXFP6 forward emulation — is supported by the within-LMD bf16-versus-MXFP6 comparisons in Table 1 and by Madam's failure on both architectures. I do not see grounds to reject the paper. The load-bearing weak point is the GPT-2 comparison with AdamW: Table 1 and the Figure 1 caption use different sequence lengths and batch sizes for AdamW versus LMD, and the text's tokens-per-step note does not control for the effect of 4096-token context on optimization stability. This does not falsify LMD's viability, but it weakens the comparative claim that AdamW is specifically unable to stably use MXFP6 forward passes. The hyperparameter-transfer issue raised by the reader is real but secondary, because the fixed sigma=0.125 and m_r settings still produced stable runs on both architectures; a sensitivity study would strengthen the usability claim. The overall verdict should remain conditional, so no change to the reader's verdict is needed.","tokens_in":14883,"tokens_out":16959,"duration_ms":177989,"concrete_test":"Run AdamW under the exact LMD GPT-2 recipe (sequence length 4096, batch size 16, 32 gradient accumulations, cosine schedule, gradient clipping at 10) with both bf16 and MXFP6 forward passes, n=3; include a sequence-length-1024/batch-size-16 AdamW arm to isolate sequence length from batch size. If AdamW at sequence length 4096 with MXFP6 trains stably within about 0.1 validation loss of its bf16 run, then LMD has no demonstrated low-precision advantage over AdamW at equal configuration; if AdamW diverges, the confound is resolved and the comparison stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 (GPT-2 rows) compares LMD and Madam at sequence length 4096 with batch size 16, while AdamW is run at sequence length 1024 with batch size 64. The authors note that the token count per step is the same, but sequence length and batch size are not equivalent controls: AdamW at sequence length 4096 in bf16 already gives an unstable validation loss of 4.790±2.017, so the comparison in the Figure 1 caption that \"AdamW cannot stably learn using low-precision forward passes\" conflates the optimizer effect with the training-configuration effect. In the low-precision contrast, LMD MXFP6 at sequence length 4096 achieves 2.927 vs 2.925 in bf16, whereas AdamW MXFP6 at sequence length 1024 achieves 3.015 vs 2.937 in bf16; because these are different sequence lengths, the degradation cannot be attributed to MXFP6 alone. Contribution 1's low-precision claim for GPT-2 therefore rests partly on a cross-configuration comparison unless this confound is controlled.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Log-Normal Multiplicative Dynamics (LMD), an optimizer that trains neural networks with multiplicative updates. LMD is derived from a Lie-group Bayesian learning rule with a fixed-variance log-normal posterior; the update multiplies positive/negative median parameters by exp(-eta(sign(nu_temp)+r)), where the regularizer r pulls each EG+- component toward a preset median m_r. The paper reports experiments on ViT/ImageNet and GPT-2/OpenWebText, including forward passes emulated in MXFP6 (and MXFP4 for ViT ablations), and claims stable from-scratch training with no degradation under low-precision forward passes, plus suppressed weight growth relative to Madam. It also includes ablations of multiplicative versus additive weight decay and of sampled versus mean training.","tokens_in":15085,"tokens_out":11932,"duration_ms":118204,"significance":"If the central claim holds, this would be the first demonstration that multiplicative optimizers can train large transformers from scratch and tolerate low-precision forward passes, which is relevant to energy-efficient hardware. The derivation of the multiplicative weight-decay step (Eqs. 8-10) is internally consistent, the code is public, the main table reports n=3 repeated runs, and the ablation structure cleanly separates regularization from noise. However, the GPT-2 low-precision claim currently rests on a sequence-length/batch-size confound, no Lion baseline isolates the effect of the sign-momentum component, and the new hyperparameters are not studied for sensitivity. The paper's own Limitations section correctly notes that backward passes remain bf16 and no real speedups were measured, so the abstract should be qualified accordingly.","major_comments":[{"comment":"The GPT-2 comparison confounds optimizer with sequence length and batch size: AdamW is evaluated at sequence length 1024 with batch size 64, while LMD and Madam use sequence length 4096 and batch size 16. The figure caption's statement that the token count per step is the same does not control for sequence-length-dependent optimization dynamics, and the table shows that AdamW at sequence length 4096 in bf16 is already unstable (4.790±2.017). Hence the claim that \"AdamW cannot stably learn using low-precision forward passes\" is not established: the degradation from 2.937 to 3.015 could be a sequence-length effect rather than an MXFP6 effect. This weakens the GPT-2 part of Contribution 1. Notably, at the same sequence length 1024 in bf16, AdamW (2.937±0.001) is slightly better than LMD (2.961±0.002), so aligning the configuration also changes the bf16 comparison. Please add an AdamW run at sequence length 4096 with MXFP6 forward passes, or an LMD run at sequence length 1024, before claiming low-precision stability for GPT-2.","section":"§4.1, Table 1 and Figure 1 caption"},{"comment":"The method's ease-of-use claim depends on the default hyperparameters sigma=0.125 and m_r=0.01*exp(sigma^2/2), and Section 3.2 explicitly calls the scale-parameter initialization \"heuristic.\" No sensitivity study is reported for either parameter, and the paper's \"drop-in replacement\" claim implies these values should transfer without per-model tuning. The fact that the same values work for ViT and GPT-2 is encouraging, but a small grid over sigma and m_r, or a statement of the working range, is needed before the transferability claim is supported.","section":"§3.2 and §4 (Experimental Settings)"},{"comment":"LMD's update rule is built from Lion's signed momentum and interpolation coefficients, yet no Lion baseline is run in Table 1. Because Lion is a strong and widely used transformer optimizer, the comparison with AdamW and Madam alone does not identify whether the reported gains come from the multiplicative dynamics or from the Lion-style sign momentum with decoupled multiplicative regularization. A Lion baseline, with and without the log-normal noise, would make the contribution of the multiplicative mechanism explicit.","section":"§3, Algorithm 1; §4.1"},{"comment":"The ViT comparison shows a 9-point accuracy gap between LMD (77.06±0.08) and AdamW (68.11±0.38). This is much larger than typical optimizer effects for ViT training, so the reader cannot tell whether the gap reflects the optimizer or an under-tuned AdamW baseline. Please provide a learning-rate sweep or reference values for the ViT setup to show that 0.001 is well chosen for AdamW in this configuration, or explicitly discuss the comparison as a fixed-configuration comparison rather than a tuned one.","section":"§4.1, Table 1 (ViT rows)"}],"minor_comments":[{"comment":"Please state how test-time predictions are produced for LMD: with the median weights m only, or with Monte Carlo sampling. If sampling is used at test time, the accuracy comparison would reflect ensemble averaging rather than training dynamics alone.","section":"§4.1"},{"comment":"The variance notation is inconsistent: Eq. (8) uses sigma_p^2 for the prior variance while the noise variance in Eq. (2) is sigma^2, and alpha=eta*gamma/sigma^2 introduces gamma without prior definition (it appears to be tau). Please align the notation.","section":"§3.1, Eqs. (8)-(9)"},{"comment":"The \"mean training\" ablation in Figure 4 is not defined in Algorithm 1. Please specify whether it sets epsilon=1, uses m directly, or replaces the sampled weights by their mean, and confirm that all other hyperparameters are identical to the sampled runs.","section":"§4.3, Figure 4"},{"comment":"There are several typos and formatting errors: \"begingJ= 8\" in the ViT settings, \"m_r = 0.01×exp(sigma^2/2)1\" with a stray superscript, and \"2.1×10 6 tokens per step\" should be typeset consistently.","section":"§4, ViT and GPT-2 settings"},{"comment":"The log-normal prior is imposed on the positive EG+- components theta+ and theta-, not on the effective network weights theta+-theta-; the statement that the penalty \"does not force weights to zero\" should be restricted to the components, because the effective weights are already centered at zero by construction.","section":"§3.1"},{"comment":"The abstract says \"low-precision inference and learning on future energy-efficient hardware,\" but the paper only evaluates low-precision forward passes, with backward passes in bf16 and no hardware speedups. Please align the abstract and contribution claims with Section 5's limitations.","section":"§5 (Limitations) and Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope for the journal, and I do not see an internal inconsistency in the derivation; the confounded GPT-2 comparison and the missing controls (Lion baseline, hyperparameter sensitivity) are the main blockers. The novelty relative to the authors' own Kiral et al. (2023) line of work should also be stated more crisply in the revised text, since LMD is presented as a specialization plus a substantially modified update recipe."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The new thing here is that a multiplicative update rule actually trains a ViT and a GPT-2 from scratch, and tolerates MXFP6 forward passes. That is a real first for this family of optimizers, and the ViT result is the cleanest evidence: 77% versus AdamW's 68%, with LMD running at a much smaller weight norm. The derivation from the Lie-group BLR framework is coherent, and the ablations back up the two mechanism claims—multiplicative weight decay suppresses weight growth more than additive, and the sampling matters under quantization. The code link is there. This is a legitimate optimizer paper, not a repackaging.\n\nThe soft spots are concentrated in the GPT-2 story. Table 1 runs AdamW at sequence length 1024 while LMD and Madam run at 4096, and the stress-test note has this right: the phrase 'AdamW cannot stably learn using low-precision forward passes' conflates the optimizer effect with the configuration effect. The AdamW runs at 4096 are unstable even in bf16, so the MXFP6 comparison is not apples-to-apples. That is fixable with a desequenced comparison, but as written the claim overstates the evidence. Second, the update borrows sign momentum from Lion, and Lion is never run as a baseline. That is a conspicuous omission for a paper claiming drop-in-easy Adam-level behavior. Third, sigma and m_r are fixed across both models and called heuristic in the text; there is no sensitivity study, so the 'stable and easy to use across architectures' claim rests on a pair of hand-set numbers.\n\nNone of these are load-bearing flaws for the ViT result, and the GPT-2 claim can be repaired by a fairer baseline. The citation pattern is fine—the self-citation to Kiral et al. is appropriate since the method is a specialization, and the prior work is clearly acknowledged. Who is this for? Anyone working on low-precision training and optimizer design. It deserves a serious referee. I would send it out with a request for a controlled AdamW comparison and a Lion baseline.","headline":"LMD is a genuinely new optimizer with a striking ViT result, but the GPT-2 low-precision claim rests on a confounded baseline comparison; worth refereeing.","tokens_in":15669,"tokens_out":2377,"would_cite":true,"duration_ms":22161,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A biologically inspired log-normal optimizer trains transformers from scratch with multiplicative updates.","keywords":["log-normal multiplicative dynamics","multiplicative weight updates","low-precision training","MXFP6","Vision Transformer","GPT-2","Bayesian learning rule","multiplicative noise injection"],"falsifier":"Train a large model such as BERT or a ResNet from scratch with LMD under the paper's default $\\sigma=0.125$ and $m_r=0.01\\exp(\\sigma^2/2)$ and an MXFP6 forward pass; if the run diverges, the weight norm explodes, or accuracy collapses relative to bfloat16, the claimed drop-in stability does not generalize. A second check: disable weight sampling by using only the mean $\\theta=m$ under MXFP6; the paper predicts the weight-norm dynamics become inconsistent and regularization weakens, so if mean training matches sampled training exactly, multiplicative noise is not the active stabilizing mechanism.","tokens_in":14637,"feed_emoji":"🧠","tokens_out":6547,"duration_ms":56907,"temperature":0.7,"pith_summary":"LMD (Log-Normal Multiplicative Dynamics) is a proposed optimizer that updates neural-network weights multiplicatively, sampling them from a log-normal distribution and applying weight decay multiplicatively in log space. The paper's central claim is that this biologically inspired rule achieves the first successful training-from-scratch of Vision Transformers and GPT-2 with multiplicative weight updates, and that it does so stably even when forward passes are computed in the MXFP6 low-precision format. This matters because multiplicative updates have long been considered unstable for large networks, since their weights tend to grow exponentially, while low-precision training is a leading route to energy-efficient hardware. The paper reports that LMD matches or beats AdamW on ViT/ImageNet and GPT-2/OpenWebText, keeps weight norms near their initial values, and needs only one extra state vector compared with AdamW.","feed_headline":"Multiplicative updates train ViT and GPT-2 from scratch","feed_subtitle":"A log-normal optimizer keeps weight norms in check under MXFP6 forward passes.","key_machinery":"The central object is the log-normal posterior over weights and the multiplicative-group update on its median. LMD maintains $m^+$ and $m^-$ (positive and negative copies of each weight), samples log-normal noise $\\varepsilon$, forms $\\theta = m \\odot \\varepsilon$, computes the gradient scaled by $\\theta$, and updates $m$ multiplicatively as $m \\leftarrow m \\odot \\exp(-\\eta(\\operatorname{sign}(\\nu_{\\mathrm{temp}}) + r))$. The regularizer $r = \\tau(\\log \\theta - \\log m_r)/\\sigma^2$ is chosen from a log-normal prior, so in expectation the update performs weight decay in log space, pulling $m$ toward $m_r$ instead of toward zero; this is what prevents the exponential weight growth that broke earlier multiplicative methods like Madam. Multiplicative noise injection serves as a regularizer and, because perturbations scale with weight magnitude, they are not wiped out when weights are quantized to MXFP6 or MXFP4, giving stability in low-precision forward passes.","core_discovery":"The discovery is that a multiplicative update rule can be made stable at scale by making both the noise and the regularization multiplicative, in direct analogy to the noisy multiplicative dynamics of biological synapses. Specifically, the paper derives LMD from the Lie-group Bayesian learning rule over log-normal posteriors: weights are sampled as $\\theta = m \\odot \\varepsilon$ with $\\varepsilon \\sim \\mathrm{LogN}(0, \\sigma^2 I)$, and the median $m$ is updated as $m \\leftarrow m \\odot \\exp(-\\eta(\\operatorname{sign}(\\nu_{\\mathrm{temp}}) + r))$, where $r = \\tau(\\log \\theta - \\log m_r)/\\sigma^2$ acts as weight decay in logarithmic space. In this scheme the sign of a weight is fixed by keeping separate positive and negative copies (the EG$\\pm$ trick), gradient scaling by the weight magnitude replaces additive updates, and multiplicative noise injection survives low-precision quantization because perturbations scale with weight size. On ViT/ImageNet and GPT-2/OpenWebText, LMD reportedly trains from scratch with no degradation under MXFP6 forward passes, reaching 77.06\\% test accuracy on ViT compared with 68.11\\% for AdamW, and reaching a 2.925 validation loss on GPT-2 at sequence length 4096.","pith_inferences":["The fixed hyperparameters across ViT and GPT-2 hint at a scale-free property of the log-space dynamics; a natural test is whether the same $\\sigma=0.125$ and $m_r=0.01\\exp(\\sigma^2/2)$ transfer to convolutional nets, encoder-only LLMs, or fine-tuning, none of which the paper examines.","If multiplicative noise's main role is preserving perturbations through quantization, LMD's sampling could replace or complement stochastic rounding in low-precision forward passes; a direct comparison would isolate which mechanism stabilizes MXFP6 training.","The log-space sign update resembles signSGD on log-weights, so convergence and generalization analyses from online learning and exponentiated-gradient theory may carry over to LMD, giving a theoretical handle the paper does not develop.","Extending LMD to low-precision backward passes, which the paper keeps in bfloat16, would determine whether multiplicative dynamics unlock fully low-precision training or only low-precision inference-side computation."],"forward_implications":["If LMD is correct, multiplicative weight updates are no longer confined to small networks: ViT and GPT-2 can be trained from scratch with them, so exponentiated-gradient optimizers re-enter the practical deep-learning toolbox.","MXFP6 forward passes without performance loss would let training run on hardware built around microscaling formats, reducing the memory and energy cost of forward matrix multiplications.","The log-space weight decay pulls weights toward $m_r$ instead of zero, so LMD changes how regularization and pruning interact: weights near $m_r$ are implicitly treated as emulated activation perturbations rather than dead parameters.","Because LMD keeps the weight norm close to its initial value, it may make training dynamics more predictable and remove the need for gradient-norm clipping (the paper uses none for ViT).","The method costs one extra state vector over AdamW ($4P$ parameters versus $3P$), which is small enough for a drop-in optimizer replacement in existing code."],"supporting_citations":[{"why":"Introduces multiplicative weight updates (Madam) for neural networks and documents the exponential weight growth and low-precision motivation that LMD must overcome.","marker":"Bernstein et al. (2020)"},{"why":"Supplies the Lie-group Bayesian learning rule and Algorithm 2 from which LMD's update is derived.","marker":"Kiral et al. (2023)"},{"why":"Provides the biological evidence that spine sizes follow a log-normal distribution arising from multiplicative dynamics, motivating the log-normal posterior.","marker":"Loewenstein et al. (2011)"},{"why":"The decoupled AdamW weight decay whose instability issues motivate LMD's separate treatment of gradient and regularizer.","marker":"Loshchilov and Hutter (2019)"},{"why":"Lion's signed momentum and two momentum coefficients are adopted in LMD's update.","marker":"Chen et al. (2023)"},{"why":"Defines the MX data formats and the MXFP6/MXFP4 settings used to emulate low-precision forward passes.","marker":"Rouhani et al. (2023c)"},{"why":"IVON provides the practical drop-in variational-optimizer template that LMD's implementation follows.","marker":"Shen et al. (2024)"},{"why":"Defines the Vision Transformer architecture used for the ImageNet experiments.","marker":"Dosovitskiy et al. (2021)"},{"why":"GPT-2 is one of the two architectures whose from-scratch training with multiplicative updates is the paper's headline result.","marker":"Radford et al. (2019)"}],"fun_headline_variants":["Multiplicative update rule from biology stabilizes low-precision training","Log-normal dynamics make low-precision training of ViT and GPT-2 stable","Synaptic-inspired LMD beats AdamW in MXFP6 training from scratch","Low-precision training stabilized by multiplicative noise and decay","LMD: log-normal multiplicative updates for stable low-precision learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-set defaults for the noise width, $\\sigma=0.125$, and the target weight scale, $m_r=0.01\\exp(\\sigma^2/2)$, transfer across architectures without per-model tuning; the paper itself calls the scale-parameter initialization heuristic and reports no sensitivity study.","fun_headline_variants_meta":{"raw":{"variants":["Multiplicative update rule from biology stabilizes low-precision training","Log-normal dynamics make low-precision training of ViT and GPT-2 stable","Synaptic-inspired LMD beats AdamW in MXFP6 training from scratch","Low-precision training stabilized by multiplicative noise and decay","LMD: log-normal multiplicative updates for stable low-precision learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1701,"prompt_tokens":969,"completion_tokens":732,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":639}},"tokens_in":585,"tokens_out":732,"duration_ms":7455,"temperature":1.0,"reasoning_tokens":639,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:02:22.922596+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a large model such as BERT or a ResNet from scratch with LMD under the paper's default $\\sigma=0.125$ and $m_r=0.01\\exp(\\sigma^2/2)$ and an MXFP6 forward pass; if the run diverges, the weight norm explodes, or accuracy collapses relative to bfloat16, the claimed drop-in stability does not generalize. A second check: disable weight sampling by using only the mean $\\theta=m$ under MXFP6; the paper predicts the weight-norm dynamics become inconsistent and regularization weakens, so if mean training matches sampled training exactly, multiplicative noise is not the active stabilizing mechanism.","supporting_citations":[{"cited_title":"Learning compositional functions via multiplicative weight updates","cited_arxiv_id":null,"evidence_quote":"Introduces multiplicative weight updates (Madam) for neural networks and documents the exponential weight growth and low-precision motivation that LMD must overcome."},{"cited_title":"The L ie- G roup B ayesian L earning R ule","cited_arxiv_id":null,"evidence_quote":"Supplies the Lie-group Bayesian learning rule and Algorithm 2 from which LMD's update is derived."},{"cited_title":"Multiplicative Dynamics Underlie the Emergence of the Log-Normal Distribution of Spine sizes in the Neocortex In Vivo","cited_arxiv_id":null,"evidence_quote":"Provides the biological evidence that spine sizes follow a log-normal distribution arising from multiplicative dynamics, motivating the log-normal posterior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Lion's signed momentum and two momentum coefficients are adopted in LMD's update."},{"cited_title":"An I mage is W orth 16x16 W ords: T ransformers for I mage R ecognition at S cale","cited_arxiv_id":null,"evidence_quote":"Defines the Vision Transformer architecture used for the ImageNet experiments."},{"cited_title":"Language Models are Unsupervised Multitask Learners","cited_arxiv_id":null,"evidence_quote":"GPT-2 is one of the two architectures whose from-scratch training with multiplicative updates is the paper's headline result."}],"review_version":2}