{"id":"098fb9cc-25db-45fd-a024-3addd7b79512","arxiv_id":"2506.19697","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training recipe combining Muon, single-scale RMSNorm, and a learnable embedding projection produces a 1.4B-parameter LLM with near-zero activation excess kurtosis and strong 4-bit quantization performance.","lead":"This paper shows that training a large language model with a specific recipe (Muon optimizer, single-scale normalization, and an embedding projection) prevents the formation of activation outliers, making the model surprisingly robust to 4-bit quantization. A 1.4B-parameter model trained on 1 trillion tokens keeps its accuracy under quantization, while an Adam-trained model collapses to near-random.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal case for Muon is not closed: without an Adam+SSNorm+EmbProj ablation, the near-zero kurtosis could be entirely due to the architectural components rather than to the optimizer.","rationale":"I read the paper as making two claims: an empirical one, that the OSP recipe produces an outlier-free, 4-bit-friendly 1.4B model, and a mechanistic one, that Muon eliminates privileged bases while the architectural components address remaining channel-wise scaling. The empirical claim is well supported by the controlled 100B-token ablation and the 1T-token run, modulo the usual single-run and reproducibility caveats. The mechanistic claim, however, is underdetermined by Table 2. The optimizer-only contrast shows Muon has a small effect at best, and the dramatic drop to 0.04 appears only when the architectural components are added. Without the Adam+architecture cell, the paper cannot rule out that the architecture alone is sufficient, which would collapse the privileged-bases story and change the claimed novelty from a three-part framework to a two-part architectural workaround. The reader's weakest assumption, transfer to 3B and 7B scales, is a legitimate generalization caveat, but it does not threaten the internal causal chain as directly as this missing control. A single feasible 100B-token experiment would resolve the ambiguity, and the released code and TPU infrastructure make it a reasonable condition to impose. I therefore keep the conditional verdict, with the additional condition that the missing ablation be run or the causal claims be softened accordingly.","tokens_in":26493,"tokens_out":24355,"duration_ms":248530,"concrete_test":"Train a 1.4B-parameter model on the same 100B-token subset with AdamW (LR 5e-3, trapezoidal schedule, weight decay 0.01) but with Single-Scale RMSNorm and EMBPROJ enabled, keeping all other settings identical to the OSP runs. Then compute the same excess-kurtosis metric at the same layer and token positions, and report 4-bit (W4A4) WikiText-2 perplexity with and without the Hadamard transform. If kurtosis is near zero (<1) and the 4-bit PPL is close to the Muon-OSP values (about 19.6 by RTN and 13.5 with the Hadamard transform), the optimizer is not the causal agent and the privileged-basis narrative must be revised. If kurtosis remains large (>100), the Muon component is validated as necessary.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central explanatory claim is that Adam's diagonal preconditioning creates privileged bases and that replacing Adam with Muon is a key part of the recipe (Abstract; §3.1). Table 2 is the only controlled optimizer comparison: with the standard architecture (no SSNorm, no EmbProj), Adam has excess kurtosis 1818.56 and Muon has 1575.12, a reduction of only about 13%. The near-zero kurtosis (0.04) appears only after adding SSNorm and EmbProj to Muon. No condition with Adam + SSNorm + EmbProj is reported, so the design cannot separate the optimizer's contribution from the architectural contribution. It remains possible that the bounded L2 normalization and the learned rotation are responsible for outlier suppression and that Muon is dispensable, or even that it hurts: Muon without Adam on embeddings has lower kurtosis (361.35) than Muon with Adam-trained embeddings (1575.12), indicating the embedding optimizer matters too. If Adam+SSNorm+EmbProj also yields near-zero kurtosis and comparable 4-bit perplexity, the privileged-basis mechanism is unsupported and the recipe collapses to two architectural components. The paper's own Figure 3 shows partial OSP components leave kurtosis elevated, so the interaction matters, but the missing cell in the ablation grid is exactly the one needed to attribute causality to the optimizer.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Outlier-Safe Pre-Training (OSP), a pre-training recipe intended to prevent activation outliers from forming in large language models rather than fixing them post-hoc. OSP combines three components: replacing Adam with the Muon optimizer, using Single-Scale RMSNorm, and adding a learnable embedding projection. The authors train a 1.4B-parameter LLaMA-style model on 1T tokens, reporting near-zero excess kurtosis (0.04) versus 1818.56 for a matched Adam baseline, and a 4-bit quantized benchmark average of 35.7 versus 26.5. They also report a 2% training overhead, release code and checkpoints, and provide a 100B-token ablation study at Table 2. Additional analyses cover complementarity with post-training quantization methods and the persistence of attention sinks in the absence of massive activations.","tokens_in":26721,"tokens_out":7620,"duration_ms":72759,"significance":"The contribution is substantial if the empirical claims hold. The paper ships source code and a pretrained checkpoint, includes a controlled Adam baseline trained on the same 1T corpus, and reports a dramatic difference in activation kurtosis (0.04 vs 1818.56) that is mechanistically meaningful for quantization. The 4-bit benchmark improvement over the matched baseline (35.7 vs 26.5) is large, and the modest 2% training overhead makes the recipe practical. The attention-sink analysis in Section 5.2 is a valuable falsifiable observation: sinks persist without massive activations, separating two previously conflated phenomena. The main limitations are the missing ablation cell that would separate optimizer and architectural effects, and under-specified evaluation protocols for the headline comparisons; these issues are fixable but affect the strength of the causal claims.","major_comments":[{"comment":"The central mechanistic attribution to the Muon optimizer is underdetermined because the ablation grid omits the Adam + SSNorm + EmbProj cell. In Table 2, replacing Adam with Muon in the standard architecture reduces excess kurtosis only from 1818.56 to 1575.12, while SSNorm and EmbProj individually reduce it to 66.69 and 703.23; the near-zero value 0.04 appears only in the full OSP row. Without a row in which Adam is combined with SSNorm and EmbProj, the design cannot distinguish the optimizer's contribution from the architectural components' contribution, and the Section 3.1 claim that Muon's removal of privileged bases is 'the foundation' of the framework is not supported. Please add this ablation at the 100B scale used for Table 2, reporting excess kurtosis and at least one 4-bit perplexity value, or, if this is infeasible, substantially soften the causal claims about Muon.","section":"§4.3, Table 2"},{"comment":"The headline comparison in Table 3 is not reproducible as reported because the quantization protocol is underspecified. The text says only '4-bit quantization' and does not state the bit widths for weights, activations, and KV cache independently; whether an online Hadamard rotation is applied; the calibration data and number of calibration samples; the group size; or whether the EmbProj matrices have been folded into the embedding and unembedding weights before evaluation. This matters because Table 2 shows that results depend strongly on these choices, with separate columns for 16-16-16, 4-8-16, 4-8-8, 4-4-16, and 4-4-4 configurations and a separate flag for Hadamard use. Please specify the full protocol used for Table 3, or report the 4-4-4 RTN and 4-4-4-plus-Hadamard results in separate columns.","section":"§4.4, Table 3"},{"comment":"The kurtosis measurement itself is underspecified. Equation (4) defines excess kurtosis, but the manuscript does not state over which activations (for example, input to MHSA, input to FFN, or all layers pooled), which layers, or how many tokens the reported values in Table 2 and Figures 3 and 7 are computed. Since the 'near-zero excess kurtosis (0.04)' is a headline claim and the comparison across configurations is the basis for the ablation conclusions, please define the exact extraction protocol and state whether it is identical for all configurations and checkpoints.","section":"§4.1, §4.3, Figures 3 and 7"}],"minor_comments":[{"comment":"The memory-usage expressions are garbled in the typeset text, e.g., 'O36LD2' and 'O 338 3 LD2'; please typeset the theoretical memory formulas in proper math notation.","section":"Table 1"},{"comment":"Notation is inconsistent: 'SSN ORM' appears in the running text, 'Ex. Kurt.' in Table 2, and 'EMBPROJ' is written in all caps in several places; choose one form, such as SSNorm and EmbProj, and use it consistently.","section":"Throughout"},{"comment":"The placement of EmbProj is ambiguous: the text says it is 'positioned after the embedding layer and before the unembedding layer,' which could mean one projection matrix or two; please clarify whether there are separate projections for the embedding and unembedding sides and how they are folded during inference.","section":"§3.3"},{"comment":"Several entries in the Tokens column are listed as '–' (Qwen 2.5 and LLAMA 3.2) even though public training configurations exist; either fill these in or explain why they are omitted.","section":"Table 3 and Appendix A.2"},{"comment":"The row 'Muon† (w/o Adam)' shows substantially lower kurtosis (361.35) than the row with decoupled Adam-trained embeddings (1575.12), but the main text does not discuss this comparison; because it bears on the embedding-optimizer interaction, it deserves an explanatory sentence in the ablation discussion.","section":"§3.3, Table 2"},{"comment":"The caption of Figure 6 does not define what 'Pos. Attn Logits' and 'Neg. Attn Logits' mean or how they are aggregated; please add definitions so the reader can interpret the distributional comparison.","section":"§5.2, Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper reports an unusually strong and clean empirical result, and the missing ablation cell is the key risk. If the authors can run Adam with SSNorm and EmbProj at the 100B scale and confirm that kurtosis remains high, the causal story for Muon is supported and the paper would be a strong accept candidate after protocol details are added. If that cell is not run, the authors should reframe the paper around the full OSP recipe rather than the optimizer-specific mechanism. I would also ask the editor to verify that the Table 3 quantization protocol is fully documented before publication, since the headline 35.7 vs 26.5 comparison depends on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real empirical result, worth refereeing, but the paper's mechanistic claim about Muon being the key outlier-prevention ingredient is not actually supported by its own ablation table. The missing cell is Adam + SSNorm + EmbProj.\n\nWhat is actually new: this is the first production-scale LLM (1.4B parameters, 1T tokens) trained without activation outliers. Excess kurtosis of 0.04 versus 1818.56 for the matched Adam baseline is a dramatic difference, and the 4-bit quantization gap (35.7 vs 26.5 average over 10 benchmarks) is large and practically meaningful. The attention-sink analysis is also a real contribution: sinks persist in the outlier-free model, which cuts against the earlier claim that sinks generate outliers.\n\nThe paper does good controlled work: a same-corpus Adam baseline at 1T, a 100B ablation study, and a release of code and checkpoints. That is reproducible evidence and should be credited.\n\nSoft spots, in order of importance. First, the ablation grid is missing the Adam + SSNorm + EmbProj condition. Table 2 shows Muon alone barely moves kurtosis (1575 vs 1818), and only the full combination reaches 0.04. Without the cross condition you cannot attribute the outcome to the optimizer, which is exactly what Section 3.1 and the abstract do. It is entirely possible that the two architectural components are doing the work and the privileged-basis story for Adam is wrong. The practical recipe survives, but the central mechanism is underdetermined. This is the one load-bearing gap.\n\nSecond, Table 3 does not specify the quantization protocol. Table 2 carefully distinguishes RTN and online Hadamard settings; Table 3's caption just says '4-bit quantization.' A reader cannot tell whether the 35.7 came from plain RTN or from a rotated variant. That needs spelling out.\n\nThird, single runs, no seeds, no error bars. At this training cost that is understandable, but the headline numbers are point estimates.\n\nFourth, the validation is at one scale (1.4B); 3B and 7B are untested. The authors say so in the limitations, which is honest, but it does limit the generality of the 'outliers are a training-strategy consequence' conclusion.\n\nI do not think the stress-test concern kills the paper. The package works; what is unsupported is the specific causal role of Muon. A serious revision should run the missing cross ablation at 100B scale, or substantially soften the mechanistic language. This paper is for people working on low-bit LLM inference and pre-training methodology. It deserves a serious referee, not a desk reject. I would send it to review with a request for that ablation and clearer quantization details.","headline":"A real and reproducible result—outlier-free pre-training at 1.4B/1T scale—but the paper's causal claim about Muon is underdetermined by its own ablation grid.","tokens_in":27327,"tokens_out":3964,"would_cite":true,"duration_ms":37421,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM activation outliers are caused by training choices, not by the architecture, and can be prevented by combining the Muon optimizer, Single-Scale RMSNorm, and a learnable embedding projection, yielding a 1.4B…","keywords":["outlier-free pre-training","activation outliers","4-bit quantization","Muon optimizer","Single-Scale RMSNorm","embedding projection","large language models","excess kurtosis"],"falsifier":"Train a 3B-parameter transformer with exactly the OSP recipe on, say, 500 billion tokens and track excess kurtosis of residual-stream activations every 50 billion tokens; if kurtosis rises well above 1 or the 4-bit average benchmark score approaches the Adam baseline (roughly 26-28), the claim that OSP prevents outliers at scale is falsified.","tokens_in":26223,"feed_emoji":"🧠","tokens_out":7226,"duration_ms":62865,"temperature":0.7,"pith_summary":"Outlier-Safe Pre-Training claims that the extreme activation outliers which wreck low-bit quantization of large language models are not an unavoidable property of the architecture but a side effect of how the model is trained. The paper combines three training choices—the Muon optimizer, Single-Scale RMSNorm, and a learnable embedding projection—so that outliers never form in the first place. On a 1.4B-parameter model trained for 1 trillion tokens, the recipe holds activation excess kurtosis (a measure of heavy-tailed outlier concentration) near 0.04 while an Adam-trained baseline reaches 1818.56, and under aggressive 4-bit weight-and-activation quantization the OSP model averages 35.7 across ten benchmarks versus 26.5. These numbers matter because they suggest deployment-friendly low-bit models can be produced at pre-training time with only about 2% added training cost.","feed_headline":"Outlier-free training keeps 4-bit LLMs accurate","feed_subtitle":"Muon, single-scale norm, and an embedding projection keep a 1.4B model's 4-bit benchmark score at 35.7 vs 26.5.","key_machinery":"The argument is carried by three interventions applied together. Muon replaces Adam's element-wise gradient scaling with momentum plus Newton–Schulz orthogonalization, removing the per-coordinate privileged basis; Single-Scale RMSNorm normalizes each vector by its Euclidean norm and multiplies by one shared scalar $\\gamma$, so no channel gets its own amplification factor; and the learnable embedding projection, a full-rank matrix after the embedding layer, redistributes any magnitudes that originate in the Adam-trained embeddings and can be absorbed into adjacent weights after training. The paper's measurements show that no single piece suffices: kurtosis stays low only when all three are present simultaneously.","core_discovery":"The paper's central claim is that outliers are consequences of training strategy: a transformer trained with per-parameter diagonal preconditioning (Adam) plus channel-wise normalization develops a privileged basis in which a few channels accumulate huge activations, while the same architecture trained under OSP does not. The authors report that the OSP model's excess kurtosis stays near zero across the entire 1-trillion-token run, that 4-bit quantization preserves most of its accuracy, and that combining OSP with existing post-training quantization methods still gives further gains. They also argue that attention sinks persist in the outlier-free model, so sinks are not the root cause of massive activations.","pith_inferences":["If the recipe transfers to 3B and 7B scales, on-device 4-bit inference could become a default for mobile-scale models; the paper does not test those sizes, so this is an extrapolation.","The shared scalar $\\gamma$ in Single-Scale RMSNorm is a new hyperparameter; at larger depth it may need a schedule or regularizer to avoid drift, which the authors do not address.","Because attention sinks persist without outliers, sink-targeting quantization fixes may be attacking the wrong mechanism in OSP-trained models; one could test this by applying such fixes and measuring the residual gain.","Combining OSP with quantization-aware training might push usable precision below 4 bits, but the paper does not experiment with QAT."],"forward_implications":["Four-bit weight-and-activation quantization no longer collapses benchmark scores; the OSP 1.4B model keeps a 35.7 average where Adam-trained models fall to roughly 26.5.","Outlier-free pre-training is compatible with existing PTQ methods: applying rotation-based or learned-rotation quantization to the OSP model improves perplexity further rather than being made redundant.","Attention-sink behavior is separable from outlier formation, so explanations that tie massive activations directly to sinks need revision.","Production-scale outlier-free training is affordable: about 2% training overhead and lower memory than standard Adam runs.","The release of an outlier-free checkpoint allows the community to study whether other emergent behaviors depend on outliers."],"supporting_citations":[{"why":"Supplies the Muon optimizer and the decoupled-embedding practice that the OSP optimizer component builds on.","marker":"(Jordan et al., 2024)"},{"why":"Identifies channel-wise scaling and optimizer-induced privileged bases as causes of outlier features, framing the problem OSP targets.","marker":"(He et al., 2024)"},{"why":"Proposes the attention-sink explanation and gating-based mitigation that OSP challenges by showing sinks persist without outliers.","marker":"(Bondarenko et al., 2023)"},{"why":"Provides Simple RMSNorm, the no-learnable-parameter baseline that Single-Scale RMSNorm modifies to avoid slow convergence.","marker":"(Qin et al., 2023)"},{"why":"Introduces rotation-based quantization and Hadamard incoherence that motivate the embedding projection and the evaluation protocol.","marker":"(Chee et al., 2023)"},{"why":"Gives QuaRot, a rotation-based PTQ method used as a comparison to show OSP is complementary to post-training mitigation.","marker":"(Ashkboos et al., 2024b)"},{"why":"Defines massive activations and distinguishes them from channel outliers, a distinction the paper's outlier-free analysis builds on.","marker":"(Sun et al., 2024)"}],"fun_headline_variants":["OSP training keeps 4-bit LLMs accurate, no post-hoc fixes","Muon optimizer + norm changes nix LLM outliers for 4-bit","1.4B model trains without outliers, 4-bit score jumps to 35.7","Outliers are training artifacts: new recipe enables robust 4-bit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recipe is validated at one scale and one architecture—a 1.4B LLaMA trained on 1 trillion tokens—so the load-bearing premise is that its outlier-preventing behavior and the stability of its new components carry over to 3B, 7B, and other transformer variants; the authors explicitly note these scales are untested.","fun_headline_variants_meta":{"raw":{"variants":["OSP training keeps 4-bit LLMs accurate, no post-hoc fixes","Muon optimizer + norm changes nix LLM outliers for 4-bit","1.4B model trains without outliers, 4-bit score jumps to 35.7","Outliers are training artifacts: new recipe enables robust 4-bit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1624,"prompt_tokens":966,"completion_tokens":658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":574}},"tokens_in":582,"tokens_out":658,"duration_ms":6659,"temperature":1.0,"reasoning_tokens":574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:27:44.838009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 3B-parameter transformer with exactly the OSP recipe on, say, 500 billion tokens and track excess kurtosis of residual-stream activations every 50 billion tokens; if kurtosis rises well above 1 or the 4-bit average benchmark score approaches the Adam baseline (roughly 26-28), the claim that OSP prevents outliers at scale is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proposes the attention-sink explanation and gating-based mitigation that OSP challenges by showing sinks persist without outliers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces rotation-based quantization and Hadamard incoherence that motivate the embedding projection and the evaluation protocol."}],"review_version":2}