{"id":"e2a47689-cc79-461a-bc4e-b73107865a94","arxiv_id":"2608.10850","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"During continual pre-training of OLMo 2 models, singular value spectra remain largely fixed while singular vectors change; selectively rewinding low-importance attention heads improves math accuracy by up to 4%.","lead":"Scientists studying how large language models learn specialized skills found that the models' weight spectra stay stable during domain adaptation; instead, the models change specific directional components, concentrated in particular attention heads. This opens a practical lever: rewinding selected attention heads to their pre-trained values can improve benchmarks, and the authors release a spectral analysis toolkit.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Head-importance criterion for the +4% rewind claim rests on an unvalidated text-CPT reference and is contradicted by Table 4 on the 1B model; this needs a direct test.","rationale":"The reader's weakest_assumption identifies the same load-bearing issue: the head-importance criterion treats text CPT as a neutral reference, and this is not independently validated. I agree with that assessment. The stress-test adds two concrete symptoms that make the concern sharper: (i) Eq. (1)'s min-max normalization makes the ranking scale-dependent, so the subtraction can be dominated by outlier heads in either domain; and (ii) the paper's own Table 4 shows the criterion does not consistently outperform the greedy baseline, contradicting the language in Sec. 4.4 for the 1B model. This matters because the headline rewind result is only as credible as the ordering it uses. The paper does have independent support for other parts of the central claim: the singular-value transplantation experiment in Appendix C.2/Fig. 15 is a genuine, falsifiable check that CPT quality does not depend much on the singular values, and the multi-scale consistency across 1B/7B/13B/32B checkpoints is real evidence. The concern is therefore not that the phenomena are fabricated, but that the practical rewind claim—the most novel and useful part—is conditional on an unvalidated reference-domain choice and on a post hoc selected rewound fraction. If the proposed test shows the ranking is stable under alternative references, the conditional verdict can be upgraded; if not, the +4% claim should be treated as an uncontrolled observation rather than a validated intervention. The correct verdict remains CONDITIONAL, with the same required revisions the reader already listed: error bars on the focal curves, a validation protocol for the rewind fraction, and corrected language about the greedy comparison.","tokens_in":23749,"tokens_out":8001,"duration_ms":86681,"concrete_test":"Re-run the 7B math head-rewind experiment replacing the TEXT reference in Eq. (1) with CODE CPT (the authors' 20B StarCoder run) and with a second independent TEXT CPT seed; for each reference, compute the top-15% rewound-head set and the full GSM8K rewind curve. If the Jaccard overlap of the top-15% sets is below about 0.5 or the +4% peak disappears under either alternative reference, the criterion is reference-dependent and the rewind claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central actionable claim—that rewinding low-importance heads improves math accuracy by up to 4% and that up to 60% of head updates can be removed—stands or falls with the head-importance criterion in Sec. 4.4, Eq. (1). That criterion subtracts min-max-scaled Frobenius-norm deltas of the target CPT from those of a TEXT reference CPT, assuming text CPT is a neutral, task-agnostic baseline. This assumption is not validated: TEXT is itself a 20B-token CPT run on a specific mixture (Table 2), with its own optimizer noise and distribution-specific head rotations, and the per-head min-max normalization makes the ranking sensitive to outlier heads. The paper's own Table 4 shows the proposed criterion does not beat greedy on the 1B model (AUC-diff 11 vs 19.1), and its 7B margin over greedy (6 vs 5) is within the reported evaluation noise (GSM8K std 0.005). Because the rewound fraction (~15%) is selected post hoc and the focal heuristic's rewind curves in Fig. 4 lack confidence bands, the +4% result has not been shown to be a property of domain adaptation rather than a reference-domain artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a spectral (SVD-based) analysis of continual pre-training (CPT) in LLMs, using OLMo 2 models at 1B, 7B, 13B, and 32B scales across math, instruction, code, and text domains. It reports three main claims: CPT leaves singular value spectra largely invariant while changing singular vectors; attention heads exhibit domain-dependent heterogeneity, enabling a head-importance criterion that supports selective rewinding with up to 4% accuracy gains and removal of up to 60% of head updates; and linear interpolation between CPT checkpoints from different domains ('domain connectivity') yields smooth quality trade-offs. The paper also releases the Diffract toolkit and training/evaluation code.","tokens_in":23936,"tokens_out":5995,"duration_ms":62191,"significance":"If established, the paper would provide a mechanistic account of CPT as a directional rather than magnitude-based weight change, with practical interventions such as head rewinding and CPT-delta truncation. The open-source Diffract package, the multi-scale evidence across the OLMo family, and the singular-value transplantation experiment in Appendix C.2 are genuine strengths and give the work a reproducible core. However, the headline rewind claims rest on a reference-domain heuristic whose validity is not demonstrated, and the paper's own Table 4 contradicts the claim that the proposed heuristic outperforms greedy ordering. The contribution is potentially valuable, but the central intervention claims need either rigorous support or substantial qualification before publication.","major_comments":[{"comment":"The main text states that the proposed difference-in-scaled-Frobenius-norms criterion 'outperforms not only standard spectral heuristics, but also the greedy ranking strategy (see Appendix Table 4)'. Table 4 does not support this. For the 1B model, the proposed metric has AUC-diff 11 versus greedy 19.1, i.e., it is strictly worse; for the 7B model the values are 6 versus 5, a difference that is small relative to the reported evaluation variability (GSM8K std 0.005 in Table 5) and for which no uncertainty is reported. The superiority claim should be removed or replaced with a properly powered comparison that includes confidence intervals on the AUC-diff values.","section":"Section 4.4, Table 4"},{"comment":"The head-importance criterion subtracts min-max-scaled per-head Frobenius-norm deltas of the target CPT from those of a TEXT reference CPT, treating text CPT as a neutral, task-agnostic baseline. This assumption is not validated. TEXT is itself a 20B-token CPT run on a specific DCLM-heavy mixture (Table 2, with 8.5B FLAN tokens), so its per-head changes can reflect replay-distribution effects and optimizer noise rather than a generic baseline. Since the +4% rewind gain and the claim that up to 60% of heads can be rewound both depend on this head ordering, the reference assumption needs a direct test, for example by repeating the rewind analysis with a second TEXT run under a different seed or data split, or with another candidate reference domain, and showing that the head ordering and resulting gains are stable.","section":"Section 4.4, Eq. (1), Table 2"},{"comment":"The headline +4% result is selected post hoc: the heuristic and the rewind fraction (around 15%) are chosen after inspecting rewind curves, and Figure 4 does not show the head-selection variability that Appendix D says was measured (only the random baseline in panel (a) has a standard-deviation band). To rule out selection bias, the authors should report mean and standard deviation of the rewind curves over the five head-selection seeds for the proposed criterion and the greedy baseline, and should state how many heuristics and rewind fractions were examined. Without this, the +4% gain cannot be distinguished from an artifact of multiple comparisons.","section":"Section 4.4, Fig. 4, Appendix D"},{"comment":"The redundancy claims ('up to 60% of head updates removed without measurable quality loss' and 'up to 50% of the smallest singular values dropped') rely on an unspecified quality-loss tolerance. The phrase 'without significant quality drop' is never quantified, and Figure 5 plots relative accuracy decrease without marking a threshold. The authors should define the tolerance explicitly (for example, within 1% relative on the target metric), apply it consistently to both head-rewind and SVD-truncation results, and report the threshold-sensitivity per model scale, since Figure 5 shows the truncation tolerance is strongly scale-dependent.","section":"Section 4.4, Fig. 5, Section 6"}],"minor_comments":[{"comment":"The scope of the min-max normalization is ambiguous: it should be stated whether scale[0,1] is applied per head, per layer, per matrix family, or globally across all heads.","section":"Section 4.4, Eq. (1)"},{"comment":"The main text refers to 'row-maximum singular vector agreement', but Appendix A.3 only defines per-vector diagonal and row-maximum agreements informally; the row-maximum variant should be defined explicitly next to Eq. (7).","section":"Appendix A.3"},{"comment":"The table is referred to as 'Appendix Table 4' in the main text, but within the appendix it is unnumbered; use a consistent cross-reference.","section":"Section 4.4, Table 4"},{"comment":"The singular-value permutation ablation in Appendix C.2.1 is informative but is not referenced from the main text where the spectral-invariance claim is made; a citation there would strengthen the argument.","section":"Appendix C.2.1"},{"comment":"The abbreviation 'MATH' is used both for the math CPT data mix and for the MATH-500 benchmark, which is confusing in places; consider distinguishing them notationally.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"For the editor: this paper is likely to attract attention because of its practical-sounding claims about rewinding heads and reducing CPT updates. The internal inconsistency between Section 4.4 and Table 4, and the unvalidated text-CPT reference assumption, are the key blockers; both are fixable with additional experiments and revised claims. I do not see grounds for rejection if the authors either validate the reference-domain heuristic or substantially downgrade the associated claims, and if they add the missing uncertainty quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper's central observational claims are well supported. The SVD transplantation experiment is a clean control: swapping in pre-train singular values into a CPT model doesn't hurt math accuracy, directly backing the claim that CPT mostly rotates singular vectors rather than changing spectra. That consistency across 1B, 7B, 13B, 32B, including public OLMo checkpoints, is solid. The Diffract toolkit is a real contribution, and domain connectivity is a useful empirical addition, even if it leans on known model-soup ideas.\n\nThe soft spot is exactly where the stress-test note lands. The +4% rewind gain and the \"up to 60% heads\" claim rest on the head-importance criterion in Eq. (1), which treats text CPT as a neutral reference. That assumption is never validated. Text CPT is itself a 20B-token run with its own distribution-specific rotations, and the min-max scaling makes the ranking sensitive to outliers. Worse, Table 4 shows the proposed criterion does not beat greedy on the 1B model (AUC-diff 11 vs 19.1), and the 7B margin (6 vs 5) is inside the reported GSM8K evaluation noise (std 0.005). So the stated advantage over greedy is not established. The rewind fraction (~15%) is selected post hoc, and Figure 4's orange curve lacks confidence bands. In other words, the strong actionable claim is plausible but not yet demonstrated. The phenomena are probably real; the criterion is a heuristic with weak support.\n\nCitation pattern looks honest; they engage the relevant spectral and merging literature without overselling novelty. The limitations section is fine, acknowledging the OLMo family and AdamW constraints.\n\nWho is this for? People studying CPT dynamics and model merging. The spectral observations and the toolkit are the durable parts; the rewind recipe is a lead to test, not a result to rely on. I'd send it to peer review with a clear ask: validate or replace the text-reference baseline, report error bars on the rewind curves, and soften the greedy comparison. As is, conditional accept with revisions.","headline":"Solid spectral observations; the headline rewind gain needs a stronger baseline and error bars.","tokens_in":24537,"tokens_out":2079,"would_cite":true,"duration_ms":21349,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Continual pre-training adapts large language models by rotating singular vectors rather than changing singular value spectra, and selectively rewinding low-importance attention heads to their pre-trained state can improve benchmark…","keywords":["continual pre-training","singular value decomposition","spectral analysis","domain adaptation","attention heads","model merging","linear interpolation","large language models"],"falsifier":"Train a new CPT run on a domain not studied here (e.g., biomedical text), apply the same difference-in-scaled-Frobenius-norms criterion with text CPT as reference, and test whether rewinding the selected heads reproduces the quality gain; if the gain disappears or accuracy drops, the claim that these heads are domain-specific, low-importance components would be refuted. A second check would transplant pre-train singular value spectra into CPT checkpoints across multiple domains and random seeds and verify that quality truly remains unchanged.","tokens_in":2074,"feed_emoji":"📊","tokens_out":11406,"duration_ms":138569,"temperature":0.7,"pith_summary":"The paper studies continual pre-training (CPT), the stage where a language model trained on general text is further trained on specialized data like math or code. Using singular value decomposition of weight matrices, it argues that CPT barely changes the singular value spectra — the relative magnitudes of a matrix's components — but instead rotates the singular vectors, meaning the directions along which the model's computations move. This leads to a practical claim: attention heads differ in how much their updates matter, and rewinding the least important heads back to their pre-trained values improves math benchmark accuracy by up to 4%, while up to 60% of head updates can be removed with no measurable loss. The paper also reports domain connectivity: linear interpolation between CPT checkpoints on different domains gives smooth quality interpolation. If the paper is right, domain adaptation is primarily a directional adjustment of a model's weight matrices, which would make cheaper, more targeted interventions possible.","feed_headline":"Rewinding selective attention heads lifts LLM accuracy by 4%","feed_subtitle":"SVD shows continual pre-training shifts singular vectors while leaving spectra intact, and partial head rewind beats full training.","key_machinery":"The central objects are the singular value decomposition of each weight matrix, written $W = U Σ V^T$, and the singular vector agreement measured by the cosine similarity between left singular vectors of a pre-trained and a CPT checkpoint. The paper's key identity is the head importance criterion $s_{l,h} = scale_{[0,1]}(‖W^{domain}_{l,h} − W^{pre-train}_{l,h}‖_F) − scale_{[0,1]}(‖W^{reference}_{l,h} − W^{pre-train}_{l,h}‖_F)$, where text CPT serves as the reference domain; this scalar ranks attention heads by how much extra they change for the target domain relative to the generic change. Head-wise rewinding replaces the four projection matrices of the lowest-ranked heads with their pre-trained values, and SVD truncation of the CPT delta tests how many small singular values can be dropped. Domain connectivity is quantified by linear interpolation $W_{interp}(ω) = (1−ω)W_{dom1} + ω W_{dom2}$ evaluated across many $ω$ values.","core_discovery":"On its own terms, the paper's central discovery is that continual pre-training of an LLM to a specialized domain does not reshape the singular value spectra of its weight matrices; instead, the adaptation is carried by changes in the singular vectors, with the strongest vector rotations concentrated near peaks in the spectra. The paper demonstrates this by transplanting the pre-train spectra into CPT checkpoints and observing no measurable quality change. It further shows that attention heads behave heterogeneously during CPT: some heads move in the same way across all domains, others are domain-specific, and this differentiation grows with pre-train token budget. Ordering heads by the difference between their scaled Frobenius-norm updates in a target domain versus a text-reference domain yields a ranking by which rewinding low-importance heads to their pre-trained state improves GSM8K accuracy by up to 4% for a 7B model and permits rewinding up to 60% of heads without significant loss. Finally, linear interpolation between checkpoints adapted to different domains becomes concave at small pre-train budgets, roughly linear at 400B tokens, and convex at 4T tokens and larger scales, a phenomenon the paper names domain connectivity.","pith_inferences":["If adaptation truly lives in singular vectors rather than spectra, then interventions that rotate subspaces or project orthogonally should be more effective for CPT than additive, low-rank updates like LoRA; this is a testable design implication the paper does not pursue.","The paper's choice of text CPT as the reference domain is one of many; replacing it with a different neutral reference (e.g., code CPT or a random-token CPT) could change the head ranking, so the +4% gain may be contingent on whether text CPT is genuinely a generic baseline.","The concave-to-convex transition in interpolation quality with pre-train scale suggests a phase-change-like property of the loss landscape; if it holds for other architectures, model-merging methods could exploit it by operating in the convex regime.","The rewind criterion's benefit may be domain-dependent: the paper's own code experiments show only a small HumanEval gain after CPT, so the head-importance signal might be weaker for domains where CPT itself produces little improvement."],"forward_implications":["Continual pre-training can be compressed: up to 60% of attention-head updates and up to 50% of the smallest singular values of the CPT delta are redundant, with redundancy increasing with model scale (13B tolerates 70% singular-value truncation).","Selective head rewind is a viable post-CPT intervention: rewinding low-importance heads to their pre-trained state improves math accuracy by up to 4% on a 7B model, and similar gains appear on released OLMo 2 13B and 32B CPT checkpoints (up to 6.3% and 3.6% respectively).","Domain connectivity holds across scales: linear interpolation between CPT checkpoints on math, text, instruction, and code shows smooth quality interpolation, and interpolant quality improves with both pre-train token budget and model size.","The complex, multi-peaked spectral structure of attention heads that develops during pre-training appears to be a prerequisite for efficient domain adaptation, while MLP spectra stay close to heavy-tailed self-regularization.","Simple task-vector-style merging is limited for CPT: linear interpolation underperforms training directly on a data mixture, so methods built on task arithmetic would face significant challenges."],"supporting_citations":[{"why":"Supplies the heavy-tailed self-regularization hypothesis that the paper's observed attention spectra deviate from.","marker":"(Mahoney & Martin, 2019)"},{"why":"Provides the HTSR model and power-law fitting used as the baseline for spectral shape analysis.","marker":"(Martin & Mahoney, 2021)"},{"why":"Earlier demonstration that singular vectors play a role in training, which the paper extends to continual pre-training.","marker":"(Yunis et al., 2024)"},{"why":"Introduces linear mode connectivity, the concept domain connectivity is patterned after.","marker":"(Frankle et al., 2020)"},{"why":"Task arithmetic framework that the paper shows underperforms for CPT domain interpolation.","marker":"(Ilharco et al., 2023)"},{"why":"Provides the OLMo 2 architecture, base checkpoints, and DolminoMath/Dolmino data mixtures used in all experiments.","marker":"(Walsh et al., 2025)"},{"why":"Power-law fitting and Kolmogorov-Smirnov distance that the paper uses to assess spectral fits.","marker":"(Clauset et al., 2009)"},{"why":"Supplies the DCLM pre-training corpus and its filtered variants used for pre-train and replay data.","marker":"(Li et al., 2024)"}],"fun_headline_variants":["Rewind low-importance heads, gain up to 4% accuracy","SVD reveals LLM adaptation in vectors, not spectra","Rewind 60% of heads, lift accuracy by 4%","Diffract: spectral toolkit reveals head rewinding gains","Domain connectivity: CPT checkpoints interpolate smoothly"],"cache_read_input_tokens":26624,"weakest_assumption_plain":"The head-importance ranking assumes that text continual pre-training is a neutral baseline; if text CPT is not a stable or task-agnostic reference, the difference metric that ranks heads, and with it the reported rewinding gains, loses its foundation.","fun_headline_variants_meta":{"raw":{"variants":["Rewind low-importance heads, gain up to 4% accuracy","SVD reveals LLM adaptation in vectors, not spectra","Rewind 60% of heads, lift accuracy by 4%","Diffract: spectral toolkit reveals head rewinding gains","Domain connectivity: CPT checkpoints interpolate smoothly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00157,"raw_usage":{"total_tokens":6257,"prompt_tokens":926,"completion_tokens":5331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":5247}},"tokens_in":542,"tokens_out":5331,"duration_ms":32723,"temperature":1.0,"reasoning_tokens":5247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:45:06.071225+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a new CPT run on a domain not studied here (e.g., biomedical text), apply the same difference-in-scaled-Frobenius-norms criterion with text CPT as reference, and test whether rewinding the selected heads reproduces the quality gain; if the gain disappears or accuracy drops, the claim that these heads are domain-specific, low-importance components would be refuted. A second check would transplant pre-train singular value spectra into CPT checkpoints across multiple domains and random seeds and verify that quality truly remains unchanged.","supporting_citations":[{"cited_title":"2019 , cdate=","cited_arxiv_id":null,"evidence_quote":"Supplies the heavy-tailed self-regularization hypothesis that the paper's observed attention spectra deviate from."}],"review_version":1}