Pith. sign in

REVIEW 4 major objections 5 minor 50 references

Diffract: Spectral View of LLM Domain Adaptation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Continual pre-training adapts large language models by rotating singular vectors rather than changing singular value spectra, and selectively rewinding low-importance attention heads to their pre-trained state can improve benchmark…

desk verdict Solid spectral observations; the headline rewind gain needs a stronger baseline and error bars. read the letter →

arxiv 2608.10850 v1 pith:C63TLBZ2 submitted 2026-08-11 cs.LG

Nikita Borodin , Maria Krylova , Artem Zabolotnyi , Dmitry Aspisov , Egor Shikov , Nikita Tyuplyaev , Oleg Travkin , Roman Alferov
show 1 more author
Dmitry Vinichenko
This is my paper · ORCID
classification cs.LG
keywords continualpre-trainingsingularvaluedecompositionspectralanalysisdomainadaptationattentionheadsmodelmerginglinearinterpolationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies continual pre-training (CPT), the stage where a language model trained on general text is further trained on specialized data like math or code. Using singular value decomposition of weight matrices, it argues that CPT barely changes the singular value spectra — the relative magnitudes of a matrix's components — but instead rotates the singular vectors, meaning the directions along which the model's computations move. This leads to a practical claim: attention heads differ in how much their updates matter, and rewinding the least important heads back to their pre-trained values improves math benchmark accuracy by up to 4%, while up to 60% of head updates can be removed with no measurable loss. The paper also reports domain connectivity: linear interpolation between CPT checkpoints on different domains gives smooth quality interpolation. If the paper is right, domain adaptation is primarily a directional adjustment of a model's weight matrices, which would make cheaper, more targeted interventions possible.

What carries the argument

The central objects are the singular value decomposition of each weight matrix, written $W = U Σ V^T$, and the singular vector agreement measured by the cosine similarity between left singular vectors of a pre-trained and a CPT checkpoint. The paper's key identity is the head importance criterion $s_{l,h} = scale_{[0,1]}(‖W^{domain}_{l,h} − W^{pre-train}_{l,h}‖_F) − scale_{[0,1]}(‖W^{reference}_{l,h} − W^{pre-train}_{l,h}‖_F)$, where text CPT serves as the reference domain; this scalar ranks attention heads by how much extra they change for the target domain relative to the generic change. Head-wise rewinding replaces the four projection matrices of the lowest-ranked heads with their pre-trained values, and SVD truncation of the CPT delta tests how many small singular values can be dropped. Domain connectivity is quantified by linear interpolation $W_{interp}(ω) = (1−ω)W_{dom1} + ω W_{dom2}$ evaluated across many $ω$ values.

What would settle it

Train a new CPT run on a domain not studied here (e.g., biomedical text), apply the same difference-in-scaled-Frobenius-norms criterion with text CPT as reference, and test whether rewinding the selected heads reproduces the quality gain; if the gain disappears or accuracy drops, the claim that these heads are domain-specific, low-importance components would be refuted. A second check would transplant pre-train singular value spectra into CPT checkpoints across multiple domains and random seeds and verify that quality truly remains unchanged.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that continual pre-training of an LLM to a specialized domain does not reshape the singular value spectra of its weight matrices; instead, the adaptation is carried by changes in the singular vectors, with the strongest vector rotations concentrated near peaks in the spectra. The paper demonstrates this by transplanting the pre-train spectra into CPT checkpoints and observing no measurable quality change. It further shows that attention heads behave heterogeneously during CPT: some heads move in the same way across all domains, others are domain-specific, and this differentiation grows with pre-train token budget. Ordering heads by the difference between their scaled Frobenius-norm updates in a target domain versus a text-reference domain yields a ranking by which rewinding low-importance heads to their pre-trained state improves GSM8K accuracy by up to 4% for a 7B model and permits rewinding up to 60% of heads without significant loss. Finally, linear interpolation between checkpoints adapted to different domains becomes concave at small pre-train budgets, roughly linear at 400B tokens, and convex at 4T tokens and larger scales, a phenomenon the paper names domain connectivity.

Load-bearing premise

The head-importance ranking assumes that text continual pre-training is a neutral baseline; if text CPT is not a stable or task-agnostic reference, the difference metric that ranks heads, and with it the reported rewinding gains, loses its foundation.

Editorial extensions

If this is right

  • Continual pre-training can be compressed: up to 60% of attention-head updates and up to 50% of the smallest singular values of the CPT delta are redundant, with redundancy increasing with model scale (13B tolerates 70% singular-value truncation).
  • Selective head rewind is a viable post-CPT intervention: rewinding low-importance heads to their pre-trained state improves math accuracy by up to 4% on a 7B model, and similar gains appear on released OLMo 2 13B and 32B CPT checkpoints (up to 6.3% and 3.6% respectively).
  • Domain connectivity holds across scales: linear interpolation between CPT checkpoints on math, text, instruction, and code shows smooth quality interpolation, and interpolant quality improves with both pre-train token budget and model size.
  • The complex, multi-peaked spectral structure of attention heads that develops during pre-training appears to be a prerequisite for efficient domain adaptation, while MLP spectra stay close to heavy-tailed self-regularization.
  • Simple task-vector-style merging is limited for CPT: linear interpolation underperforms training directly on a data mixture, so methods built on task arithmetic would face significant challenges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If adaptation truly lives in singular vectors rather than spectra, then interventions that rotate subspaces or project orthogonally should be more effective for CPT than additive, low-rank updates like LoRA; this is a testable design implication the paper does not pursue.
  • The paper's choice of text CPT as the reference domain is one of many; replacing it with a different neutral reference (e.g., code CPT or a random-token CPT) could change the head ranking, so the +4% gain may be contingent on whether text CPT is genuinely a generic baseline.
  • The concave-to-convex transition in interpolation quality with pre-train scale suggests a phase-change-like property of the loss landscape; if it holds for other architectures, model-merging methods could exploit it by operating in the convex regime.
  • The rewind criterion's benefit may be domain-dependent: the paper's own code experiments show only a small HumanEval gain after CPT, so the head-importance signal might be weaker for domains where CPT itself produces little improvement.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a spectral (SVD-based) analysis of continual pre-training (CPT) in LLMs, using OLMo 2 models at 1B, 7B, 13B, and 32B scales across math, instruction, code, and text domains. It reports three main claims: CPT leaves singular value spectra largely invariant while changing singular vectors; attention heads exhibit domain-dependent heterogeneity, enabling a head-importance criterion that supports selective rewinding with up to 4% accuracy gains and removal of up to 60% of head updates; and linear interpolation between CPT checkpoints from different domains ('domain connectivity') yields smooth quality trade-offs. The paper also releases the Diffract toolkit and training/evaluation code.

Significance. If established, the paper would provide a mechanistic account of CPT as a directional rather than magnitude-based weight change, with practical interventions such as head rewinding and CPT-delta truncation. The open-source Diffract package, the multi-scale evidence across the OLMo family, and the singular-value transplantation experiment in Appendix C.2 are genuine strengths and give the work a reproducible core. However, the headline rewind claims rest on a reference-domain heuristic whose validity is not demonstrated, and the paper's own Table 4 contradicts the claim that the proposed heuristic outperforms greedy ordering. The contribution is potentially valuable, but the central intervention claims need either rigorous support or substantial qualification before publication.

major comments (4)
  1. [Section 4.4, Table 4] The main text states that the proposed difference-in-scaled-Frobenius-norms criterion 'outperforms not only standard spectral heuristics, but also the greedy ranking strategy (see Appendix Table 4)'. Table 4 does not support this. For the 1B model, the proposed metric has AUC-diff 11 versus greedy 19.1, i.e., it is strictly worse; for the 7B model the values are 6 versus 5, a difference that is small relative to the reported evaluation variability (GSM8K std 0.005 in Table 5) and for which no uncertainty is reported. The superiority claim should be removed or replaced with a properly powered comparison that includes confidence intervals on the AUC-diff values.
  2. [Section 4.4, Eq. (1), Table 2] The head-importance criterion subtracts min-max-scaled per-head Frobenius-norm deltas of the target CPT from those of a TEXT reference CPT, treating text CPT as a neutral, task-agnostic baseline. This assumption is not validated. TEXT is itself a 20B-token CPT run on a specific DCLM-heavy mixture (Table 2, with 8.5B FLAN tokens), so its per-head changes can reflect replay-distribution effects and optimizer noise rather than a generic baseline. Since the +4% rewind gain and the claim that up to 60% of heads can be rewound both depend on this head ordering, the reference assumption needs a direct test, for example by repeating the rewind analysis with a second TEXT run under a different seed or data split, or with another candidate reference domain, and showing that the head ordering and resulting gains are stable.
  3. [Section 4.4, Fig. 4, Appendix D] The headline +4% result is selected post hoc: the heuristic and the rewind fraction (around 15%) are chosen after inspecting rewind curves, and Figure 4 does not show the head-selection variability that Appendix D says was measured (only the random baseline in panel (a) has a standard-deviation band). To rule out selection bias, the authors should report mean and standard deviation of the rewind curves over the five head-selection seeds for the proposed criterion and the greedy baseline, and should state how many heuristics and rewind fractions were examined. Without this, the +4% gain cannot be distinguished from an artifact of multiple comparisons.
  4. [Section 4.4, Fig. 5, Section 6] The redundancy claims ('up to 60% of head updates removed without measurable quality loss' and 'up to 50% of the smallest singular values dropped') rely on an unspecified quality-loss tolerance. The phrase 'without significant quality drop' is never quantified, and Figure 5 plots relative accuracy decrease without marking a threshold. The authors should define the tolerance explicitly (for example, within 1% relative on the target metric), apply it consistently to both head-rewind and SVD-truncation results, and report the threshold-sensitivity per model scale, since Figure 5 shows the truncation tolerance is strongly scale-dependent.
minor comments (5)
  1. [Section 4.4, Eq. (1)] The scope of the min-max normalization is ambiguous: it should be stated whether scale[0,1] is applied per head, per layer, per matrix family, or globally across all heads.
  2. [Appendix A.3] The main text refers to 'row-maximum singular vector agreement', but Appendix A.3 only defines per-vector diagonal and row-maximum agreements informally; the row-maximum variant should be defined explicitly next to Eq. (7).
  3. [Section 4.4, Table 4] The table is referred to as 'Appendix Table 4' in the main text, but within the appendix it is unnumbered; use a consistent cross-reference.
  4. [Appendix C.2.1] The singular-value permutation ablation in Appendix C.2.1 is informative but is not referenced from the main text where the spectral-invariance claim is made; a citation there would strengthen the argument.
  5. [Throughout] The abbreviation 'MATH' is used both for the math CPT data mix and for the MATH-500 benchmark, which is confusing in places; consider distinguishing them notationally.

Circularity Check

1 steps flagged · score 3.0 of 10

The rewind gain (up to +4%) is the in-sample maximum of a head-ordering curve whose fraction and metric were selected on the same benchmarks; the spectral-invariance claim is independently supported, so the circularity is partial, not total.

  1. fitted input called prediction [Section 4.4, Eq. (1), Fig. 4; Appendix Table 4.]
    "Motivated by the head heterogeneity concept introduced in the previous section (Fig. 3), we propose a novel head ordering criterion that allows us (i) to achieve a quality increase—of up to +4% for math CPT of a 7B model upon rewinding around 15% of heads, and (ii) to rewind up to 60% of heads without a significant quality drop."

    The headline rewind result is read off the same curves used to choose the method. Eq. (1) is introduced right after stating that the metric 'performs better' for 7B and 13B models, and the '+4% ... around 15% of heads' is the maximum of the Fig. 4(b) curve at a fraction selected post hoc; no pre-specified fraction or held-out evaluation is used before reporting the gain. The prediction is therefore a summary of the fitted curve rather than an independent test. The paper's own Table 4 confirms the criterion is not robust: on the 1B model its AUC-diff (11) is below greedy (19.1), and on the 7B model it beats greedy by only one point (6 vs 5).

full rationale

The spectral-invariance claim is supported by a direct counterfactual experiment: transplanting pre-train singular values into CPT singular vectors (Eq. 14, Fig. 15) leaves GSM8K accuracy unchanged, which tests the claim rather than assuming it. Domain connectivity is a measured interpolation curve with endpoint evaluations, and the SVD-truncation redundancy is an empirical ablation on external benchmarks. There is no load-bearing self-citation: the paper's self-references are limited to its own released Diffract toolkit, and the prior-work citations are to external results. The main circularity concern is confined to the head-rewind headline, where the 15% fraction and the Eq. (1) reference-text metric were selected on the same evaluation curves that produce the +4% figure, and Appendix Table 4 shows inconsistent dominance over the greedy ground truth (worse on 1B, marginally better on 7B). The authors' own limitation statement acknowledges that conclusions are restricted to the OLMo family and benchmark coverage is limited, which is a correctness caveat rather than a circular reduction. Because the spectral findings and the transfer to externally released 13B and 32B checkpoints provide independent content, the paper is only partially affected, and the score is moderate rather than high.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claims do not introduce fitted constants or new physical entities. They rest on analytic choices: using SVD as the lens, treating text CPT as a reference domain, treating attention heads as independent units, and assuming same-initialization interpolation is meaningful. These choices are domain assumptions rather than free parameters, with the exceptions noted above.

free parameters (2)
  • Head rewind fraction for headline gain = around 15% for 7B math CPT, observed optimum
    The up to +4% improvement is reported at the optimum of the rewind curve; the fraction is selected post hoc from the same evaluations rather than fixed a priori or chosen on a held-out set.
  • Quality-loss tolerance for redundancy claims = unspecified; interpreted as no measurable drop
    The statements that up to 60% of heads or 50% to 70% of singular values can be removed without quality loss depend on an informal tolerance for what counts as measurable, which is not formalized or pre-registered.
assumptions (4)
  • domain assumption SVD decomposition and singular vector agreement capture adaptation-relevant structure in weights.
    The entire analysis in Sections 2, 4.2, and 4.4 assumes that spectral components of weight matrices are meaningful carriers of learned information.
  • ad hoc to paper Text CPT is a neutral reference for isolating domain-specific head changes.
    Eq. (1) subtracts text-CPT Frobenius norms from target-domain norms; the paper does not validate that text CPT changes are task-agnostic rather than domain-specific in their own right.
  • domain assumption Attention heads can be treated as independent units for rewinding.
    Section C.3 rewinds Q, K, V, and O blocks per head and assumes the intervention does not break cross-head interactions in a way that invalidates benchmark results.
  • domain assumption Checkpoints trained from the same initialization can be linearly interpolated without permutation alignment.
    Section 4.5 forms W_interp = (1-w) W_dom1 + w W_dom2 for models sharing the same W_pre-train; this presumes linear mode connectivity holds in CPT weight space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diffract: Spectral View of LLM Domain Adaptation." pith.science (2026). https://pith.science/paper/C63TLBZ2

@misc{pith2026260810850,
  author       = {Pith},
  title        = {Pith review of: Diffract: Spectral View of LLM Domain Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C63TLBZ2}},
  note         = {Machine review of arXiv:2608.10850}
}
read the original abstract

We study continual pre-training (CPT) as a mechanism for adapting general-purpose large language models to specialized domains: mathematics, instruction, code, and natural text. Using singular value decomposition of weight matrices, we find that CPT leaves singular value spectra largely invariant, with adaptation driven mainly by changes in singular vectors. An analysis of attention-head projection matrices reveals strong, domain-dependent head heterogeneity, which we exploit to define a head importance criterion: up to 60% of head updates can be removed without measurable quality loss. Selectively rewinding low-importance heads to their pre-trained state improves benchmark accuracy by up to 4% versus the fully trained baseline. Finally, we identify domain connectivity - linear interpolation between CPT checkpoints yields smooth domain-quality interpolation without notable degradation on either domain - and release Diffract, an open-source toolkit for scalable spectral analysis of billion-parameter models.

Figures

Figures reproduced from arXiv: 2608.10850 by the authors.

Figure 1
Figure 1. Math–language quality trade-off with continual pre-training on the OLMo 2 1B backbone. The y-axis reports the average accuracy on WinoGrande, ARC-Easy, and HellaSwag. Hollow circles denote pre-train checkpoints; filled circles denote continual pre-training runs initialized from the corresponding pre-train checkpoints. Marker size encodes the CPT token budget (10B, 20B, 50B) on a mixture of math and text data; marker… view at source ↗
Figure 2
Figure 2. Spectral shape and metrics evolution during pre-train. a) Rows (top to bottom) correspond to WQ , WV , and WFC2 weight matrices, while columns (left to right) report the Spectral norm and the Effective rank. Each subplot overlays jittered per-matrix values with box plots summarizing the distributions for models pre-trained on 20B, 100B, 400B, and 4T tokens. Points denote individual matrices and are color-coded by th… view at source ↗
Figure 3
Figure 3. Head heterogeneity—effect of the CPT domain on vector agreement with the pre-trained model vs. pre-train budget: 4T tokens (top) and 20B (bottom). a) Heatmaps show mean CPT–pre-train vector agreement for individual heads of WQ across domains (math, instruct, text). Lower agreement indicates larger weight rotations from pre-train. b) Agreement difference (math minus text): blue/red indicate larger changes under math/… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: CPT delta head-wise rewind results. a) GSM8K accuracy versus fraction of heads rewound for the 1B model for different head ordering schemes: average singular vector agreement between CPT and pre-train (solid red), PL-KS distance (blue), and random (red dashed line with…
Figure 5
Figure 5. Figure 5: CPT delta SVD redundancy analysis. GSM8K relative accuracy as a function of the fraction of singular values retained after SVD truncation of the CPT delta. Curves show models with different scales and pre-train budgets: 13B/5T tokens (pink), 7B/4T tokens (blue), 1B/4T …
Figure 6
Figure 6. Figure 6: Domain connectivity demonstration. Quality of linear interpolation between pairs of checkpoints on different domains, step size ω = 0.1. a) DROP F1 score vs. GSM8K accuracy for different pre-train token budgets and model sizes. The purple pentagon marker denotes models…
Figure 7
Figure 7. Figure 7: Analysis of CPT configurations varying in token budgets and math proportions. Panels (a)-(c) -show results for CPT initialized from the 20B, 100B and 400B pre-train checkpoints respectively. Performance is evaluated as a function of the mathematical data proportion in …
Figure 8
Figure 8. Figure 8: Comparison of spectral properties of 1B, 7B, and 13B parameter models. a) Layer-wise spectral norm and stable rank comparison for WQ , WV , and WFC2 matrices. The x-axis represents the relative layer depth, normalized as a percentage of total layers to account for the …
Figure 9
Figure 9. Figure 9: Comparison of spectral properties of 13B and 32B parameter models of OLMo 2 original CPT checkpoint. a) Layer-wise spectral norm and stable rank comparison for WQ , WV , and WFC2 matrices. The x-axis represents the relative layer depth, normalized as a percentage of to…
Figure 10
Figure 10. Figure 10: The effect of CPT data domain on vector agreement with pre-train. Violin plots (with jittered points colored by the left singular vector agreement between CPT and pre-train) display the WGATE per layer for CPT a) on text, b) a math-text mix, c) or pure math (all pre-t…
Figure 11
Figure 11. Figure 11: Goodness-of-fit—layer-wise Kolmogorov–Smirnov distance between the weight matrix ESD and Marchenko–Pastur model (left) and power law (right) model for WQ , WV , and WFC2 matrices. The x-axis indexes layers; solid lines show the mean across attention heads, dashed line…
Figure 12
Figure 12. Figure 12: Effect of CPT data domain on vector agreement. a) Heatmaps show the average vector agreement between CPT and pre-train for individual heads of WQ across different CPT domains (left to right: math, instruct) on 7B model. Lower vector agreement values correspond to larg…
Figure 13
Figure 13. Figure 13: Effect of CPT data domain on vector agreement. a) Heatmaps show the average vector agreement between CPT and pre-train for individual heads of WQ across different CPT domains (left to right: math, instruct) on 13B model. Lower vector agreement values correspond to lar…
Figure 14
Figure 14. Figure 14: Layer-wise Frobenius norm of CPT deltas for WQ and WFC2 across (a) CPT domains and pre-train budgets for 1B model and (b) across CPT domains for 7B and 13B models. Statistics are aggregated per head (mean: solid line, median: dashed line; shaded region: mean ± std). a…
Figure 15
Figure 15. Figure 15: Singular value transplantation results. Red circles denote CPT on DolminoMath, while star–diamond markers denote CPT on DolminoMath with singular values transplanted from the pre-train model. The results indicate that singular value transplantation does not affect GSM…
Figure 16
Figure 16. Figure 16: CPT delta head-wise rewind results. a) DROP F1 score versus the fraction of heads rewound for a 7B-parameter model with a 4T-token pre-train budget and a 20B-token instruct domain. Rewinding heads according to any heuristic has only a minor effect on CPT quality in th…
Figure 17
Figure 17. Figure 17: CPT delta head-wise rewind results for 13B model with 5T token pre-train for 10B token CPT. a) MATH-500 accuracy versus the fraction of heads rewound. Rewinding heads gains 2.2%. b) SQuAD F1 score versus the fraction of heads rewound. Head rewinding according to any h…
Figure 18
Figure 18. Figure 18: CPT delta head-wise rewind results for 13B and 32B OLMo 2 original checkpoints. a) GSM8K accuracy versus the fraction of heads rewound for 13B original OLMo 2 CPT checkpoint. Rewinding heads gains 6.3%. b) GSM8K accuracy versus the fraction of heads rewound for 32B or…
Figure 19
Figure 19. Figure 19: Head-wise rewind heatmap (per-layer, per-head impact) for 20B Math CPT based on 4T pre-train. Cells contain GSM8K accuracy change (%) [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Linear interpolation and CPT delta redundancy analysis on instruct domain. a) Quality of linear interpolation, step size ω = 0.1. Average language-task accuracy vs. DROP F1 score. As the pre-train token budget and model size increase, interpolation quality improves. b…
Figure 21
Figure 21. Figure 21: a) Quality of linear interpolation between 1B/4T tokens Instruct CPT and 1B/4T tokens Code CPT, step size ω = 0.1. b) Quality of linear interpolation between 1B/4T tokens Math CPT and 1B/4T tokens Code CPT, step size ω = 0.1. c) Truncation of 1B/4T tokens Code CPT del…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 23 canonical work pages

  1. [1]

    The Twelfth International Conference on Learning Representations , year=

    Let's Verify Step by Step , author=. The Twelfth International Conference on Learning Representations , year=

  2. [2]

    Conference on Empirical Methods in Natural Language Processing , year=

    SQuAD: 100,000+ Questions for Machine Comprehension of Text , author=. Conference on Empirical Methods in Natural Language Processing , year=

    work page

  3. [3]

    Clauset, Aaron and Shalizi, Cosma Rohilla and Newman, M. E. J. , year=. Power-Law Distributions in Empirical Data , volume=. SIAM Review , publisher=

    work page

  4. [4]

    The Twelfth International Conference on Learning Representations , year=

    The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction , author=. The Twelfth International Conference on Learning Representations , year=

    work page

  5. [5]

    2023 , eprint=

    Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data , author=. 2023 , eprint=

    work page 2023

  6. [6]

    Yin, Lu and Jaiswal, Ajay Kumar and Liu, Shiwei and Kundu, Souvik and Wang, Zhangyang , booktitle =. Junk. 2024 , editor =

    work page 2024

  7. [7]

    Golub, G. H. and Reinsch, C. , title =. Numer. Math. , month = apr, pages =. 1970 , issue_date =

    work page 1970

  8. [8]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Linear Mode Connectivity and the Lottery Ticket Hypothesis , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    work page 2020

Show all 50 references
  1. [9]

    Journal of Machine Learning Research , volume=

    Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning , author=. Journal of Machine Learning Research , volume=

  2. [10]

    2019 , cdate=

    Michael Mahoney and Charles Martin , title=. 2019 , cdate=

    2019

  3. [11]

    High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning , year=

    Random matrix theory analysis of neural network weight matrices , author=. High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning , year=

    2024

  4. [12]

    Transactions on Machine Learning Research , year=

    From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks , author=. Transactions on Machine Learning Research , year=

  5. [13]

    arXiv preprint arXiv:2410.17770 , year =

    Locating Information in Large Language Models via Random Matrix Theory , author =. arXiv preprint arXiv:2410.17770 , year =

  6. [14]

    International Conference on Machine Learning , pages=

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  7. [15]

    Advances in Neural Information Processing Systems , volume=

    Diverse weight averaging for out-of-distribution generalization , author=. Advances in Neural Information Processing Systems , volume=

  8. [16]

    The Eleventh International Conference on Learning Representations , year=

    Git Re-Basin: Merging Models modulo Permutation Symmetries , author=. The Eleventh International Conference on Learning Representations , year=

  9. [17]

    The Eleventh International Conference on Learning Representations , year=

    Editing models with task arithmetic , author=. The Eleventh International Conference on Learning Representations , year=

  10. [18]

    Advances in Neural Information Processing Systems , volume=

    Ties-merging: Resolving interference when merging models , author=. Advances in Neural Information Processing Systems , volume=

  11. [19]

    Workshop on Efficient Systems for Foundation Models @ ICML2023 , year=

    Continual Pre-Training of Large Language Models: How to re-warm your model? , author=. Workshop on Efficient Systems for Foundation Models @ ICML2023 , year=

  12. [20]

    arXiv preprint arXiv:2505.12512 , year=

    Scalable Strategies for Continual Learning with Replay , author=. arXiv preprint arXiv:2505.12512 , year=

    arXiv

  13. [21]

    Transactions on Machine Learning Research , year=

    Simple and Scalable Strategies to Continually Pre-train Large Language Models , author=. Transactions on Machine Learning Research , year=

  14. [22]

    IEEE Trans

    Liyuan Wang and Xingxing Zhang and Hang Su and Jun Zhu , title=. IEEE Trans. Pattern Anal. Mach. Intell. , volume=. 2024 , month=

    2024

  15. [23]

    Efficient Continual Pre-training for Building Domain Specific Large Language Models

    Xie, Yong and Aggarwal, Karan and Ahmad, Aitzaz. Efficient Continual Pre-training for Building Domain Specific Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024. 2024

    2024

  16. [24]

    Xing and Sham M

    Zhenting Qi and Fan Nie and Alexandre Alahi and James Zou and Himabindu Lakkaraju and Yilun Du and Eric P. Xing and Sham M. Kakade and Hanlin Zhang , booktitle=. Evo

  17. [25]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Task singular vectors: Reducing task interference in model merging , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  18. [26]

    International Conference on Machine Learning , year=

    Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch , author=. International Conference on Machine Learning , year=

  19. [27]

    Transactions on Machine Learning Research , year=

    Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic , author=. Transactions on Machine Learning Research , year=

  20. [28]

    Evan Pete Walsh and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Shane Arora and Akshita Bhagia and Yuling Gu and Shengyi Huang and Matt Jordan and Nathan Lambert and others , booktitle=. 2

  21. [29]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  22. [30]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  23. [31]

    First Conference on Language Modeling , year=

    Does your data spark joy? Performance gains from domain upsampling at the end of training , author=. First Conference on Language Modeling , year=

  24. [32]

    arXiv preprint arXiv:2308.12950 , year=

    Baptiste Rozière and Jonas Gehring and Fabian Gloeckle and Sten Sootla and Itai Gat and Xiaoqing Ellen Tan and Yossi Adi and Jingyu Liu and Tal Remez and Jérémy Rapin and others , title=. arXiv preprint arXiv:2308.12950 , year=

    arXiv

  25. [33]

    Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo , title=. 2024 , cdate=

    2024

  26. [34]

    Advances in Neural Information Processing Systems , volume=

    Datacomp-lm: In search of the next generation of training sets for language models , author=. Advances in Neural Information Processing Systems , volume=

  27. [35]

    arXiv preprint arXiv:2408.11804 , year=

    Approaching deep learning through the spectral dynamics of weights , author=. arXiv preprint arXiv:2408.11804 , year=

    arXiv

  28. [36]

    Fanxu Meng and Zhaohui Wang and Muhan Zhang , booktitle=. Pi

  29. [37]

    OLMES : A Standard for Language Model Evaluations

    Gu, Yuling and Tafjord, Oyvind and Kuehl, Bailey and Haddad, Dany and Dodge, Jesse and Hajishirzi, Hannaneh. OLMES : A Standard for Language Model Evaluations. Findings of the Association for Computational Linguistics: NAACL 2025. 2025

    2025

  30. [38]

    arXiv preprint arXiv:1803.05457 , year =

    Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord , title =. arXiv preprint arXiv:1803.05457 , year =

    arXiv

  31. [39]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year=

    HellaSwag: Can a Machine Really Finish Your Sentence? , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year=

  32. [40]

    2020 , cdate=

    Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi , title=. 2020 , cdate=

    2020

  33. [41]

    Edward J Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle=. Lo

  34. [42]

    Advances in Neural Information Processing Systems , volume=

    D-cpt law: Domain-specific continual pre-training scaling law for large language models , author=. Advances in Neural Information Processing Systems , volume=

  35. [43]

    International Conference on Learning Representations , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  36. [44]

    Attention is All you Need , volume =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , volume =

  37. [45]

    Transactions on Machine Learning Research , year=

    StarCoder: may the source be with you! , author=. Transactions on Machine Learning Research , year=

  38. [46]

    International Conference on Learning Representations , year=

    Finetuned Language Models are Zero-Shot Learners , author=. International Conference on Learning Representations , year=

  39. [47]

    North American Chapter of the Association for Computational Linguistics , year=

    DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs , author=. North American Chapter of the Association for Computational Linguistics , year=

  40. [48]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  41. [49]

    Mathematics of the USSR-Sbornik , volume=

    Distribution of eigenvalues for some sets of random matrices , author=. Mathematics of the USSR-Sbornik , volume=. 1967 , publisher=

    1967

  42. [50]

    2026 , eprint=

    Olmo 3 , author=. 2026 , eprint=

    2026

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.