Pith. sign in

REVIEW 3 major objections 6 minor 70 references

Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that LLM activation outliers are caused by training choices, not by the architecture, and can be prevented by combining the Muon optimizer, Single-Scale RMSNorm, and a learnable embedding projection, yielding a 1.4B…

desk verdict A real and reproducible result—outlier-free pre-training at 1.4B/1T scale—but the paper's causal claim about Muon is underdetermined by its own ablation grid. read the letter →

arxiv 2506.19697 v1 pith:TOA7DL4O submitted 2025-06-24 cs.LG cs.AIcs.CL

Jungwoo Park , Taewhoo Lee , Chanwoong Yoon , Hyeon Hwang , Jaewoo Kang This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords outlier-freepre-trainingactivationoutliers4-bitquantizationMuonoptimizerSingle-ScaleRMSNormembeddingprojectionlargelanguagemodelsexcesskurtosis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Outlier-Safe Pre-Training claims that the extreme activation outliers which wreck low-bit quantization of large language models are not an unavoidable property of the architecture but a side effect of how the model is trained. The paper combines three training choices—the Muon optimizer, Single-Scale RMSNorm, and a learnable embedding projection—so that outliers never form in the first place. On a 1.4B-parameter model trained for 1 trillion tokens, the recipe holds activation excess kurtosis (a measure of heavy-tailed outlier concentration) near 0.04 while an Adam-trained baseline reaches 1818.56, and under aggressive 4-bit weight-and-activation quantization the OSP model averages 35.7 across ten benchmarks versus 26.5. These numbers matter because they suggest deployment-friendly low-bit models can be produced at pre-training time with only about 2% added training cost.

What carries the argument

The argument is carried by three interventions applied together. Muon replaces Adam's element-wise gradient scaling with momentum plus Newton–Schulz orthogonalization, removing the per-coordinate privileged basis; Single-Scale RMSNorm normalizes each vector by its Euclidean norm and multiplies by one shared scalar $\gamma$, so no channel gets its own amplification factor; and the learnable embedding projection, a full-rank matrix after the embedding layer, redistributes any magnitudes that originate in the Adam-trained embeddings and can be absorbed into adjacent weights after training. The paper's measurements show that no single piece suffices: kurtosis stays low only when all three are present simultaneously.

What would settle it

Train a 3B-parameter transformer with exactly the OSP recipe on, say, 500 billion tokens and track excess kurtosis of residual-stream activations every 50 billion tokens; if kurtosis rises well above 1 or the 4-bit average benchmark score approaches the Adam baseline (roughly 26-28), the claim that OSP prevents outliers at scale is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that outliers are consequences of training strategy: a transformer trained with per-parameter diagonal preconditioning (Adam) plus channel-wise normalization develops a privileged basis in which a few channels accumulate huge activations, while the same architecture trained under OSP does not. The authors report that the OSP model's excess kurtosis stays near zero across the entire 1-trillion-token run, that 4-bit quantization preserves most of its accuracy, and that combining OSP with existing post-training quantization methods still gives further gains. They also argue that attention sinks persist in the outlier-free model, so sinks are not the root cause of massive activations.

Load-bearing premise

The recipe is validated at one scale and one architecture—a 1.4B LLaMA trained on 1 trillion tokens—so the load-bearing premise is that its outlier-preventing behavior and the stability of its new components carry over to 3B, 7B, and other transformer variants; the authors explicitly note these scales are untested.

Editorial extensions

If this is right

  • Four-bit weight-and-activation quantization no longer collapses benchmark scores; the OSP 1.4B model keeps a 35.7 average where Adam-trained models fall to roughly 26.5.
  • Outlier-free pre-training is compatible with existing PTQ methods: applying rotation-based or learned-rotation quantization to the OSP model improves perplexity further rather than being made redundant.
  • Attention-sink behavior is separable from outlier formation, so explanations that tie massive activations directly to sinks need revision.
  • Production-scale outlier-free training is affordable: about 2% training overhead and lower memory than standard Adam runs.
  • The release of an outlier-free checkpoint allows the community to study whether other emergent behaviors depend on outliers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the recipe transfers to 3B and 7B scales, on-device 4-bit inference could become a default for mobile-scale models; the paper does not test those sizes, so this is an extrapolation.
  • The shared scalar $\gamma$ in Single-Scale RMSNorm is a new hyperparameter; at larger depth it may need a schedule or regularizer to avoid drift, which the authors do not address.
  • Because attention sinks persist without outliers, sink-targeting quantization fixes may be attacking the wrong mechanism in OSP-trained models; one could test this by applying such fixes and measuring the residual gain.
  • Combining OSP with quantization-aware training might push usable precision below 4 bits, but the paper does not experiment with QAT.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Outlier-Safe Pre-Training (OSP), a pre-training recipe intended to prevent activation outliers from forming in large language models rather than fixing them post-hoc. OSP combines three components: replacing Adam with the Muon optimizer, using Single-Scale RMSNorm, and adding a learnable embedding projection. The authors train a 1.4B-parameter LLaMA-style model on 1T tokens, reporting near-zero excess kurtosis (0.04) versus 1818.56 for a matched Adam baseline, and a 4-bit quantized benchmark average of 35.7 versus 26.5. They also report a 2% training overhead, release code and checkpoints, and provide a 100B-token ablation study at Table 2. Additional analyses cover complementarity with post-training quantization methods and the persistence of attention sinks in the absence of massive activations.

Significance. The contribution is substantial if the empirical claims hold. The paper ships source code and a pretrained checkpoint, includes a controlled Adam baseline trained on the same 1T corpus, and reports a dramatic difference in activation kurtosis (0.04 vs 1818.56) that is mechanistically meaningful for quantization. The 4-bit benchmark improvement over the matched baseline (35.7 vs 26.5) is large, and the modest 2% training overhead makes the recipe practical. The attention-sink analysis in Section 5.2 is a valuable falsifiable observation: sinks persist without massive activations, separating two previously conflated phenomena. The main limitations are the missing ablation cell that would separate optimizer and architectural effects, and under-specified evaluation protocols for the headline comparisons; these issues are fixable but affect the strength of the causal claims.

major comments (3)
  1. [§4.3, Table 2] The central mechanistic attribution to the Muon optimizer is underdetermined because the ablation grid omits the Adam + SSNorm + EmbProj cell. In Table 2, replacing Adam with Muon in the standard architecture reduces excess kurtosis only from 1818.56 to 1575.12, while SSNorm and EmbProj individually reduce it to 66.69 and 703.23; the near-zero value 0.04 appears only in the full OSP row. Without a row in which Adam is combined with SSNorm and EmbProj, the design cannot distinguish the optimizer's contribution from the architectural components' contribution, and the Section 3.1 claim that Muon's removal of privileged bases is 'the foundation' of the framework is not supported. Please add this ablation at the 100B scale used for Table 2, reporting excess kurtosis and at least one 4-bit perplexity value, or, if this is infeasible, substantially soften the causal claims about Muon.
  2. [§4.4, Table 3] The headline comparison in Table 3 is not reproducible as reported because the quantization protocol is underspecified. The text says only '4-bit quantization' and does not state the bit widths for weights, activations, and KV cache independently; whether an online Hadamard rotation is applied; the calibration data and number of calibration samples; the group size; or whether the EmbProj matrices have been folded into the embedding and unembedding weights before evaluation. This matters because Table 2 shows that results depend strongly on these choices, with separate columns for 16-16-16, 4-8-16, 4-8-8, 4-4-16, and 4-4-4 configurations and a separate flag for Hadamard use. Please specify the full protocol used for Table 3, or report the 4-4-4 RTN and 4-4-4-plus-Hadamard results in separate columns.
  3. [§4.1, §4.3, Figures 3 and 7] The kurtosis measurement itself is underspecified. Equation (4) defines excess kurtosis, but the manuscript does not state over which activations (for example, input to MHSA, input to FFN, or all layers pooled), which layers, or how many tokens the reported values in Table 2 and Figures 3 and 7 are computed. Since the 'near-zero excess kurtosis (0.04)' is a headline claim and the comparison across configurations is the basis for the ablation conclusions, please define the exact extraction protocol and state whether it is identical for all configurations and checkpoints.
minor comments (6)
  1. [Table 1] The memory-usage expressions are garbled in the typeset text, e.g., 'O36LD2' and 'O 338 3 LD2'; please typeset the theoretical memory formulas in proper math notation.
  2. [Throughout] Notation is inconsistent: 'SSN ORM' appears in the running text, 'Ex. Kurt.' in Table 2, and 'EMBPROJ' is written in all caps in several places; choose one form, such as SSNorm and EmbProj, and use it consistently.
  3. [§3.3] The placement of EmbProj is ambiguous: the text says it is 'positioned after the embedding layer and before the unembedding layer,' which could mean one projection matrix or two; please clarify whether there are separate projections for the embedding and unembedding sides and how they are folded during inference.
  4. [Table 3 and Appendix A.2] Several entries in the Tokens column are listed as '–' (Qwen 2.5 and LLAMA 3.2) even though public training configurations exist; either fill these in or explain why they are omitted.
  5. [§3.3, Table 2] The row 'Muon† (w/o Adam)' shows substantially lower kurtosis (361.35) than the row with decoupled Adam-trained embeddings (1575.12), but the main text does not discuss this comparison; because it bears on the embedding-optimizer interaction, it deserves an explanatory sentence in the ablation discussion.
  6. [§5.2, Figure 6] The caption of Figure 6 does not define what 'Pos. Attn Logits' and 'Neg. Attn Logits' mean or how they are aggregated; please add definitions so the reader can interpret the distributional comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OSP's near-zero kurtosis and quantization gains are measured empirical outcomes, not consequences built into the definitions of its components.

full rationale

The paper's central claim is empirical: training a 1.4B model with Muon + Single-Scale RMSNorm + EmbProj yields excess kurtosis 0.04 and strong 4-bit quantization performance. This is not derived from an equation that equates the input to the output. SSNorm (Eq. 3) normalizes each activation vector to L2 norm gamma, but boundedness alone does not force low excess kurtosis; sparse spike distributions can have high kurtosis even under a norm bound. Empirically, Muon+SSNorm alone gives 66.69 and Muon+EmbProj gives 703.23, with the full combination giving 0.04, so the headline value is not guaranteed by any single component. No parameter is fitted to the quantization benchmarks: Table 2 varies fixed components, Table 3 evaluates a separately trained Adam baseline under identical conditions, and all quantization scores are measured post hoc. The Muon/Shampoo convergence citations (Jordan et al., Bernstein & Newhouse, Duvvuri et al.) are external works, not self-citations, and they are not the load-bearing support for the outlier-free claim, which rests on the training dynamics and ablations in Figures 2-3 and Table 2. The Limitations section candidly notes untested 3B/7B scales and limited second-order optimizer comparisons; this is a generalizability gap, not circularity. The absence of an Adam+SSNorm+EmbProj cell is an ablation confound for causal attribution to the optimizer, but a confounded comparison is not a definitional reduction. Under the required standard, no circular step can be exhibited with quoted evidence, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central claim rests on an empirical recipe rather than a derivation. Key hand-chosen hyperparameters are listed. The causal mechanism (privileged bases) is adopted from prior work as a domain assumption. SSNorm and EmbProj are new components, both falsifiable through ablations and downstream quantization tests.

free parameters (4)
  • Muon learning rate = 5e-4
    Chosen by hand (Appendix A.1); affects whether OSP components produce stable training and low kurtosis.
  • Adam learning rate = 5e-3
    Chosen by hand (Appendix A.1) for the Adam baseline and for embedding layers.
  • Batch size = 4M tokens
    Chosen for throughput (Appendix A.1); not central to the outlier-prevention claim.
  • Weight decay = 0.01
    Standard setting (Appendix A.1).
assumptions (4)
  • domain assumption Excess kurtosis of activations is a valid proxy for quantization difficulty.
    Used throughout (Eq. 4) to measure outlier presence; not proven but widely used in prior work.
  • domain assumption Muon's orthogonalized updates eliminate privileged bases and hence outlier formation.
    Taken from Jordan et al. 2024; the paper provides no new proof, only empirical confirmation.
  • domain assumption The single-scale RMSNorm with learned scalar gamma preserves training stability while preventing channel-wise amplification.
    Proposed and empirically validated here, but no theoretical guarantee.
  • ad hoc to paper The learnable embedding projection redistributes outliers without changing the function.
    Introduced for OSP; relies on the ability to absorb matrices into adjacent embeddings (SliceGPT).
invented entities (2)
  • Single-Scale RMSNorm (SSNorm) independent evidence
    purpose: Normalization that uses a single scalar gamma, eliminating channel-wise scale factors.
    A new architectural component; testable by ablation and in other models.
  • Embedding projection (EmbProj) independent evidence
    purpose: Full-rank learnable projection after embedding to redistribute outlier magnitudes.
    New component inspired by PTQ rotations; can be evaluated independently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models." pith.science (2026). https://pith.science/paper/TOA7DL4O

@misc{pith2026250619697,
  author       = {Pith},
  title        = {Pith review of: Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TOA7DL4O}},
  note         = {Machine review of arXiv:2506.19697}
}
read the original abstract

Extreme activation outliers in Large Language Models (LLMs) critically degrade quantization performance, hindering efficient on-device deployment. While channel-wise operations and adaptive gradient scaling are recognized causes, practical mitigation remains challenging. We introduce Outlier-Safe Pre-Training (OSP), a practical guideline that proactively prevents outlier formation rather than relying on post-hoc mitigation. OSP combines three key innovations: (1) the Muon optimizer, eliminating privileged bases while maintaining training efficiency; (2) Single-Scale RMSNorm, preventing channel-wise amplification; and (3) a learnable embedding projection, redistributing activation magnitudes originating from embedding matrices. We validate OSP by training a 1.4B-parameter model on 1 trillion tokens, which is the first production-scale LLM trained without such outliers. Under aggressive 4-bit quantization, our OSP model achieves a 35.7 average score across 10 benchmarks (compared to 26.5 for an Adam-trained model), with only a 2% training overhead. Remarkably, OSP models exhibit near-zero excess kurtosis (0.04) compared to extreme values (1818.56) in standard models, fundamentally altering LLM quantization behavior. Our work demonstrates that outliers are not inherent to LLMs but are consequences of training strategies, paving the way for more efficient LLM deployment. The source code and pretrained checkpoints are available at https://github.com/dmis-lab/Outlier-Safe-Pre-Training.

Figures

Figures reproduced from arXiv: 2506.19697 by the authors.

Figure 1
Figure 1. Comparison of performance degradation pat [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Activation distribution analysis from the 20th layer input to Multi-Head Self-Attention (MHSA) and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Training dynamics comparison showing loss (left) and excess kurtosis evolution (right) across 100 billion [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: WikiText-2 perplexity under varying weight [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Activation magnitudes of query and key to [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Attention logit distributions at sink token posi [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Training dynamics over 1 trillion tokens demonstrating production-scale viability of our framework. The [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Activation distribution visualization of models trained with Adam optimizer across 1 trillion training [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Activation distribution visualization of models trained with [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Weight distribution visualization of models trained with Adam optimizer across 1 trillion training tokens. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Weight distribution visualization of model trained with [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 20 canonical work pages

  1. [1]

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werr...

    arXiv 2025

  2. [2]

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. 2024. Smollm - blazingly fast and remarkably powerful

    2024

  3. [3]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024 a . https://openreview.net/forum?id=vXxardq6db Slice GPT : Compress large language models by deleting rows and columns . In The Twelfth International Conference on Learning Representations

    work page 2024

  4. [4]

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024 b . https://proceedings.neurips.cc/paper_files/paper/2024/file/b5b939436789f76f08b9d0da5e81af7c-Paper-Conference.pdf Quarot: Outlier-free 4-bit inference in rotated llms . In Advances in Neural Information ...

    work page 2024

  5. [5]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  6. [6]

    Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veli c kovi \'c , and Razvan Pascanu. 2025. Why do llms attend to the first token? arXiv preprint arXiv:2504.02732

    arXiv 2025

  7. [7]

    Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. 2024. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834

    arXiv 2024

  8. [8]

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. https://huggingface.co/datasets/HuggingFaceTB/cosmopedia Cosmopedia

    2024

Show all 70 references
  1. [9]

    Jeremy Bernstein and Laker Newhouse. 2024 a . Modular duality in deep learning. arXiv preprint arXiv:2410.21265

    2024 arXiv

  2. [10]

    Jeremy Bernstein and Laker Newhouse. 2024 b . Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325

    2024 arXiv

  3. [11]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. https://proceedings.mlr.press/v202/...

    2023

  4. [12]

    Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. https://doi.org/10.1609/aaai.v34i05.6239 Piqa: Reasoning about physical commonsense in natural language . Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432--7439

    2020 doi

  5. [13]

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.627 Understanding and overcoming the challenges of efficient transformer quantization . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pr...

    2021 doi

  6. [14]

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/edbcb7583fd8921dad78adecfe06a99b-Paper-Conference.pdf Quantizable transformers: Removing outliers by helping attention heads do nothing . In Advances in N...

    2023

  7. [15]

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake Vander P las, Skye Wanderman- M ilne, and Qiao Zhang. 2018. http://github.com/jax-ml/jax JAX : composable transformations of P ython+ N um P y programs

    2018

  8. [16]

    Diego Caples and Rob Neuhaus. 2024. https://www.lesswrong.com/posts/yrhu6MeFddnGRSLtQ/adam-optimizer-causes-privileged-basis-in-transformer-lm Adam optimizer causes privileged basis in transformer lm residual stream . LessWrong

    2024

  9. [17]

    Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/0df38cd13520747e1e64e5b123a78ef8-Paper-Conference.pdf Quip: 2-bit quantization of large language models with guarantees . In Advances in Neural...

    2023

  10. [18]

    Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. 2024. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062

    2024 arXiv

  11. [19]

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. 2023. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886

    2023 arXiv

  12. [20]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    2018 arXiv

  13. [21]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  14. [22]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/c3ba4962c05c49636d4c6206a97e9c8a-Paper-Conference.pdf Gpt3.int8(): 8-bit matrix multiplication for transformers at scale . In Advances in Neural Inf...

    2022

  15. [23]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  16. [24]

    Sai Surya Duvvuri, Fnu Devvrit, Rohan Anil, Cho-Jui Hsieh, and Inderjit S Dhillon. 2024. https://openreview.net/forum?id=8j9hz8DVi8 Combining axes preconditioners through kronecker approximation for deep learning . In The Twelfth International Conference on Learning Representations

    2024

  17. [25]

    Nelson Elhage, Robert Lasenby, and Christopher Olah. 2023. https://transformer-circuits.pub/2023/privileged-basis/index.html Privileged bases in the transformer residual stream . Transformer Circuits Thread

    2023

  18. [26]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. https://openreview.net/forum?id=tcbBPnfwxS OPTQ : Accurate quantization for generative pre-trained transformers . In The Eleventh International Conference on Learning Representations

    2023

  19. [27]

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tusha...

    2024

  20. [28]

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. 2025. https://openreview.net/forum?id=78Nn4QJTEN When attention sink emerges in language models: An empirical view . In The Thirteenth International Conference on Learning Representations

    2025

  21. [29]

    Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I Jordan, and Song Mei. 2024. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms. arXiv preprint arXiv:2410.13835

    2024 arXiv

  22. [30]

    Vineet Gupta, Tomer Koren, and Yoram Singer. 2018. https://proceedings.mlr.press/v80/gupta18a.html Shampoo: Preconditioned stochastic tensor optimization . In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Res...

    2018

  23. [31]

    Alex H\" a gele, Elie Bakouch, Atli Kosson, Loubna Ben allal, Leandro Von Werra, and Martin Jaggi. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/8b970e15a89bf5d12542810df8eae8fc-Paper-Conference.pdf Scaling laws and compute-optimal training beyond fixed trai...

    2024

  24. [32]

    Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/986292a930c3692168b177a770025ab3-Paper-Conference.pdf Understanding and minimising outlier features in transformer training . In Advanc...

    2024

  25. [33]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  26. [34]

    NJ Higham. 2008. Functions of matrices: Theory and computation

    2008

  27. [35]

    Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. 2024. https://kellerjordan.github.io/posts/muon/ Muon: An optimizer for hidden layers in neural networks

    2024

  28. [36]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  29. [37]

    Mahoney, and Kurt Keutzer

    Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. 2024. https://proceedings.mlr.press/v235/kim24f.html S queeze LLM : Dense-and-sparse quantization . In Proceedings of the 41st International Confere...

    2024

  30. [38]

    Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. https://doi.org/10.18653/v1/2021.findings-acl.300 BERT busters: Outlier dimensions that disrupt transformers . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3392-...

    2021 doi

  31. [39]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023 arXiv

  32. [40]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. https://proceedings.mlsys.org/paper_files/paper/2024/file/42a452cbafa9dd64e9ba4aa95cc1ef21-Paper-Conference.pdf Awq: Activation-aware weig...

    2024

  33. [41]

    Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.460 I ntact KV : Improving large language model quantization by keeping pivot tokens intact . In Findings of the Associ...

    2024 doi

  34. [42]

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.26 LLM - QAT : Data-free quantization aware training for large language models...

    2024 doi

  35. [43]

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024 c . Spinquant--llm quantization with learned rotations. arXiv preprint arXiv:2405.16406

    2024 arXiv

  36. [44]

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. https://doi.org/10.57967/hf/2497 Fineweb-edu: the finest collection of educational content

    2024 doi

  37. [45]

    James Martens and Roger Grosse. 2015. https://proceedings.mlr.press/v37/martens15.html Optimizing neural networks with kronecker-factored approximate curvature . In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learni...

    2015

  38. [46]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843

    2016 arXiv

  39. [47]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    2018 doi

  40. [48]

    Aniruddha Nrusimha, Mayank Mishra, Naigang Wang, Dan Alistarh, Rameswar Panda, and Yoon Kim. 2024. Mitigating the impact of outlier channels for language model quantization with activation regularization. arXiv preprint arXiv:2404.03605

    2024 arXiv

  41. [49]

    Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Fei Yuan, Xiao Luo, et al. 2023. Scaling transnormer to 175 billion parameters. arXiv preprint arXiv:2307.14995

    2023 arXiv

  42. [50]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. https://doi.org/10.1609/aaai.v34i05.6399 Winogrande: An adversarial winograd schema challenge at scale . Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732--8740

    2020 doi

  43. [51]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. https://doi.org/10.18653/v1/D19-1454 Social IQ a: Commonsense reasoning about social interactions . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9...

    2019 doi

  44. [52]

    u nther Schulz. 1933. Iterative berechung der reziproken matrix. ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift f \

    G \"u nther Schulz. 1933. Iterative berechung der reziproken matrix. ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift f \"u r Angewandte Mathematik und Mechanik , 13(1):57--59

    1933

  45. [53]

    Seungwoo Son, Wonpyo Park, Woohyun Han, Kyuyeun Kim, and Jaeho Lee. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.134 Prefixing attention sinks can mitigate activation outliers for large language model quantization . In Proceedings of the 2024 Conference on Empirical Metho...

    2024 doi

  46. [54]

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762

    2024 arXiv

  47. [55]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  48. [56]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  49. [57]

    Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. https://proceedings.mlr.press/v235/tseng24a.html Q u IP \# : Even better LLM quantization with hadamard incoherence and lattice codebooks . In Proceedings of the 41st International Conferen...

    2024

  50. [58]

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. 2024. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321

    2024 arXiv

  51. [59]

    Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/6f6db140de9c9f111b12ef8a216320a9-Paper-Conference.pdf Outlier suppression: Pushing the limit of l...

    2022

  52. [60]

    Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. 2024. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective. arXiv preprint arXiv:2410.05192

    2024 arXiv

  53. [61]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. https://proceedings.mlr.press/v202/xiao23c.html S mooth Q uant: Accurate and efficient post-training quantization for large language models . In Proceedings of the 40th International Conference...

    2023

  54. [62]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. https://openreview.net/forum?id=NG7sS51zVF Efficient streaming language models with attention sinks . In The Twelfth International Conference on Learning Representations

    2024

  55. [63]

    Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Hongjun Choi, Blake Hechtman, and Shibo Wang. 2020. Automatic cross-replica sharding of weight update in data-parallel training. arXiv preprint arXiv:2004.13336

    2020 arXiv

  56. [64]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  57. [65]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  58. [66]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  59. [67]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. https://arxiv.org/abs/2401.02385 Tinyllama: An open-source small language model . Preprint, arXiv:2401.02385

    2024 arXiv

  60. [68]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  61. [69]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  62. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.