{"id":"84394bc4-f4b0-4a86-a536-1edab23653e9","arxiv_id":"2608.13515","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Training data influence measured by progress toward final parameters shows literature data dominate early pretraining and STEM data dominate later.","lead":"This paper defines a training example's influence as how much its gradient update reduces the distance to the model's final parameters, estimated from saved checkpoints without retraining. Applied to 18 Pythia and PolyPythia runs, it finds literature data matter most early in pretraining and STEM data matter most later.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The literature-to-STEM crossover rests on an SGD per-example decomposition applied to Adam-trained models; §5.1 validates only the checkpoint approximation, not this assumption, so the decomposition may be driving the result.","rationale":"I agree with the reader's weakest assumption. The paper's positive contributions (exact mini-batch influence, parameter-free scoring, public checkpoints, and a transparent checkpoint approximation) are real, but the headline claim depends on a per-example attribution that is not validated against Adam-trained updates. The concern is addressable with a targeted leave-one-out check; if it lands, the empirical contribution would need to be reframed as an attribution convention rather than a measurement of which examples actually drive progress. I therefore keep the conditional verdict rather than escalating to rejection. The endpoint-sensitivity finding is acknowledged in §5.2 and is an inherent property of defining influence with respect to a reference; scale-consistency is appropriately hedged in §5.3 and Appendix B.3. The SGD/Adam assumption is the load-bearing issue.","tokens_in":33125,"tokens_out":4232,"duration_ms":44474,"concrete_test":"Retrain a 1k-step interval of Pythia-70M-Deduped (e.g., 10k→11k) from the public checkpoint with the original Adam hyperparameters, saving optimizer states. For 200 sampled training examples, rerun the mini-batch with that example removed while holding the prior optimizer state fixed, and measure the change in squared-distance reduction to the final checkpoint. Rank examples by this leave-one-out contribution and compare the domain composition of the top/bottom 5% with the Eq. (3) rankings. If the two rankings correlate poorly, or if the literature/STEM crossover reverses under the leave-one-out measure, the SGD decomposition is driving the headline finding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central domain-level claim (literature-aligned early, STEM-aligned late) is computed from example-level contributions defined in Eq. (3), which assumes Δ_{t,k} = −η_t ∇ℓ(x;θ_t). This decomposition sums to the true mini-batch update Δ_t only under plain SGD. Pythia and PolyPythia are trained with Adam (footnote 1; §A Limitations), where per-example gradients are normalized by accumulated second moments and combined with momentum, so an example's causal contribution to Δ_t depends on optimizer state history and on other examples in the batch. The mini-batch influence Cont(B_t) in Eqs. (1)–(2) is optimizer-agnostic and exact, but it is not what is used to rank examples. The validation in §5.1 compares the checkpoint approximation Eq. (4) only to the exact SGD-defined quantity Eq. (3); both share the same SGD assumption, so the validation does not test whether the decomposition captures which examples actually drive an Adam-trained model's trajectory. The paper explicitly acknowledges the assumption in footnote 1 and §A, but the acknowledgment does not reduce its load-bearing status: if Adam's stateful normalization reorders within-batch contributions, the top/bottom 5% domain compositions in §4.4 and the resulting literature-to-STEM crossover could be artifacts of the decomposition rather than properties of the training data. The moderate early-stage correlation in Table 1 (r≈0.62) further means the checkpoint approximation is weakest exactly in the early regime where the paper claims literature data dominate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a task-agnostic measure of training-data influence for language model pretraining: an example's contribution is the reduction in squared L2 distance to the final parameters caused by its gradient update. The mini-batch version is exact, while the example-level version assumes SGD; both are estimated from public checkpoints using Eq. (4). Applying this to Pythia and PolyPythia runs, the authors report that mean contribution peaks mid-training, opponents (negative-contribution examples) increase late, high-perplexity texts contribute most in the middle stage, and domain composition shifts from literature-related to STEM-related influence across training. They validate the checkpoint approximation on a 70M model, test sensitivity to the reference endpoint, and compare with a task-specific TracIn-style score.","tokens_in":33447,"tokens_out":6438,"duration_ms":63060,"significance":"If the central domain crossover is correct, the paper offers a valuable trajectory-level characterization of pretraining data and a practical demonstration of post-hoc influence analysis without retraining or task labels. The derivation of Cont(B_t) and Cont(x_t^k) is a clean algebraic identity with no fitted parameters, and the use of public checkpoints makes the method easy to apply. The robustness analysis across model scales, initializations, and data orderings is a genuine strength. However, the headline claim currently rests on an unvalidated SGD decomposition for Adam-trained models, and the provided validation is weakest in the early phase where the literature-dominance claim sits. Given the additional reference-endpoint sensitivity, the paper does not yet support its abstract claims at the level claimed.","major_comments":[{"comment":"The example-level decomposition assumes standard SGD with per-example updates Δ_{t,k} = −η_t ∇ℓ(x;θ_t), so the per-example contributions sum to the true mini-batch update only for plain SGD. Pythia and PolyPythia are trained with Adam (footnote 1; §A), where per-example effects on the actual update are mediated by optimizer states and by other examples in the batch. The mini-batch influence Cont(B_t) in Eq. (2) is exact and optimizer-agnostic, but the domain-level rankings in §4.4 are computed from the SGD-based Eq. (3)/(4). The validation in §5.1 compares Eq. (4) only with Eq. (3); both quantities share the same SGD assumption, so the validation does not test whether the decomposition identifies the examples that actually drive an Adam-trained model. Because the abstract's literature-to-STEM crossover is derived from these rankings, this gap is load-bearing. A concrete fix would be to rerun the validation with Adam and compare per-example attribution against an Adam-aware decomposition, or at minimum to show that the domain-level crossover is preserved when Eq. (3) is replaced by an Adam-corrected score.","section":"§3.2, Eq. (3), footnote 1, Appendix A"},{"comment":"The checkpoint approximation has Pearson r=0.622 and Spearman r=0.592 in the early interval (1k→2k), versus 0.808/0.811 mid and 0.950/0.947 late. The paper's claim that literature-related data are most strongly aligned 'early in training' rests precisely on the early interval, where the approximation is weakest. The paper should report, for the early interval, whether the top- and bottom-5% domain compositions from the approximate scores agree with those from the exact scores, or at least report the correlation restricted to the top and bottom ranks. Without this, the early-stage domain claim may be driven by approximation error rather than by true contribution.","section":"§5.1, Table 1"},{"comment":"The contribution rankings are highly dependent on the reference endpoint: using the 70k checkpoint gives Spearman −0.174 with the final-reference scores and top-5% overlap 0.316; using 120k gives 0.566 and 0.680. Since the final checkpoint is itself a choice, the paper needs to show that the domain-level crossover (literature early, STEM late) is not an artifact of that choice. The current analysis reports only aggregate rank correlations and set overlaps; it does not report whether the top/bottom domain composition still exhibits the crossover when alternative near-final endpoints are used. Please provide the domain-composition analogue of Table 3, or explicitly state the range of reference endpoints for which the crossover persists.","section":"§5.2, Table 3"},{"comment":"The evidence for the 'literature-related data are more strongly aligned early' part of the claim is indirect. Figure 4(a) shows that STEM domains are overrepresented among bottom-5% contributors early and that Books & Literature become more common among bottom-5% later; Figure 4(b) shows that the STEM share among top-5% increases late. Neither panel directly shows literature as a top-5% contributor early. The abstract's claim should either be backed by explicit early top-5% domain shares for Books & Literature or softened to 'literature is less likely to be among bottom contributors early.' This matters because the top-5% patterns in Figure 8 are also less uniform across model sizes than the bottom-5% patterns.","section":"§4.4, Figure 4, §B.3"}],"minor_comments":[{"comment":"The x-axis labels in the domain-composition figures appear as encoded strings such as '/aj17' and '/aj19/aj17' rather than training-step labels; this rendering issue must be fixed before publication.","section":"Figures 4, 7–12"},{"comment":"The learning-rate curve is plotted on a secondary axis but the figure has no legend; please specify which curve corresponds to the right-hand axis.","section":"Figure 2(a)"},{"comment":"Min–max normalization within each checkpoint interval removes scale information, so the 'share of normalized contribution' is not directly comparable across intervals; state this explicitly when interpreting the PPL-bin shares.","section":"§4.3"},{"comment":"The abstract says the crossover is 'broadly consistent across model configurations,' but the top-5% cross-scale results in Appendix B.3 show no single directional pattern shared across all six models; consider tightening the wording to reflect the bottom-5% and domain-pair specific evidence.","section":"§5.3"},{"comment":"The TracIn-style comparison reports no confidence intervals or seed variation; the conclusion that this score 'does not consistently recover' the crossover would be strengthened by quantifying variability across the sampled examples and validation subsets.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The measure is genuinely new for task-agnostic attribution: an example's contribution is how much its gradient update shrinks squared distance to the final weights. It is a simple algebraic identity, has no free parameters, and needs no downstream task selection. The empirical payoff is a trajectory-level picture across eighteen Pythia/PolyPythia runs, with a plausible narrative: literature-aligned updates early, STEM-aligned later. I read the paper skeptical and came away thinking the finding is worth taking seriously, but not as established.\n\nWhat it does well: the derivation is transparent, the checkpoint approximation is clearly defined, and the authors actually test reference-endpoint sensitivity instead of burying it. The limitations section is direct about the SGD assumption and the narrow model family. The TracIn comparison in Section 5.4 is a nice sanity check that task-specific attribution goes the other way. People working on data attribution, data curation, or curriculum design will get something useful from the framing.\n\nWhere it is soft. The example-level decomposition in Eq. (3) assumes SGD, so a per-example gradient alone is treated as the entire contribution to the update. Pythia and PolyPythia are trained with Adam. Under Adam, the same example's gradient passes through accumulated second moments and momentum, so the intra-batch ordering of contributions can be very different. The authors flag this in a footnote and in the appendix, but flagging it does not reduce its load-bearing status: all domain rankings are computed with the SGD decomposition. The validation in Section 5.1 checks only the checkpoint approximation against the exact SGD-defined quantity; both share the same assumption, so it cannot tell you whether the rankings reflect what Adam actually did. And the early-interval correlation is the weak spot: r≈0.62 at 1k→2k steps, which is exactly where the paper claims literature dominance.\n\nThe endpoint sensitivity is a second real problem. Table 3 shows that using an earlier reference checkpoint flips the rankings outright (negative correlations, top-5% overlap under 0.15 with a 30k reference). The paper calls this reference-dependent, which is honest, but it means the literature-to-STEM story is tied to choosing the final checkpoint as the reference. The cross-scale claims are also oversold: the top-5% domain patterns are not uniform across sizes, and the abstract's 'broadly consistent' is too strong.\n\nBottom line: this deserves a serious referee, but the central empirical claim needs an explicit test of the SGD decomposition under Adam—for example, a small Adam-trained model where actual per-example contributions are estimated by leave-one-out or a state-aware influence method, not another SGD-defined comparison. If that test lines up, the crossover is a strong result. Right now I would treat it as a hypothesis, not a finding.","headline":"A clean new task-agnostic influence measure, but the literature-to-STEM crossover is only as solid as the unvalidated SGD decomposition applied to Adam-trained models.","tokens_in":33967,"tokens_out":2716,"would_cite":true,"duration_ms":26930,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A checkpoint-based, task-agnostic measure of training data influence reveals that the examples most aligned with a model's final parameters shift from literature early in pretraining to STEM later.","keywords":["training data influence","language model pretraining","task-agnostic attribution","checkpoint-based approximation","Pythia","PolyPythia","domain shift","trajectory matching"],"falsifier":"Retrain or replay a small Pythia run (e.g., 70M) while storing per-step optimizer states, compute an Adam-faithful per-example attribution by simulating the exact update each example induces through the optimizer, and compare the domain-level rankings with the SGD-based contribution score; if the literature-to-STEM crossover disappears or shifts substantially under the true optimizer, the reported result is an artifact of the decomposition. An intervention that swaps the data schedule—placing STEM data early and literature later—should reverse which domains align with the final parameters if the score tracks causal influence.","tokens_in":32951,"feed_emoji":"📈","tokens_out":5119,"duration_ms":46258,"temperature":0.7,"pith_summary":"The paper argues that training data influence across language model pretraining can be measured without choosing a downstream task: an example's influence is the amount by which its gradient update reduces the squared distance between the current parameters and the final pretrained parameters. The authors derive a checkpoint-based approximation that can be computed post hoc from saved checkpoints, and apply it to 18 configurations of the Pythia and PolyPythia suites, each with 154 checkpoints. They report that the most influential examples are not stable over training: literature-related texts dominate early, while STEM-related texts become more influential in later stages, a crossover that is broadly consistent across model sizes, weight initializations, and data orderings. The value of the proposal, if correct, is a task-agnostic trajectory-level view of pretraining data that complements task-specific influence analyses and speaks to stage-aware data curation.","feed_headline":"Influential data shift from literature to STEM during pretraining","feed_subtitle":"A task-agnostic influence measure tracks which examples pull a model toward its final weights across 18 runs.","key_machinery":"The central object is the contribution score $Cont(B_t) = S_t - S_{t+1}$, where $S_t = \\|\\theta^* - \\theta_t\\|_2^2$ is the squared distance from the current parameters to the final parameters $\\theta^*$. Expanding $S_{t+1}$ turns the score into $2\\Delta_t^\\top(\\theta^* - \\theta_t) - \\|\\Delta_t\\|_2^2$, an alignment term between the update and the direction toward the final model minus a squared-norm penalty. Example-level influence is defined by decomposing the update into per-example SGD contributions $\\Delta_{t,k}$, and the checkpoint-based approximation replaces step-level states by adjacent saved checkpoints, treating all examples in an interval as if processed at the earlier checkpoint. This machinery lets the score be estimated post hoc from the released 154-checkpoint trajectories without retraining.","core_discovery":"The central claim is that the set of training examples that most reduce the squared L2 distance to the final model weights undergoes a systematic temporal shift during pretraining: early updates are most aligned with literature-related data, and later updates are most aligned with STEM-related data. The claim is made for the Pythia and PolyPythia model families, and the paper further reports that difficult (high-perplexity) texts contribute disproportionately during the middle stage, that opponent examples—updates that move the model away from the final parameters—become more frequent late in training, and that these dynamics are broadly robust to model scale, seed, weight initialization, and data ordering, with smaller models exhibiting delayed transitions.","pith_inferences":["Editorial inference: if the SGD-based decomposition misattributes credit under Adam, the literature-to-STEM crossover could be a property of the approximation rather than of the actual training dynamics; an optimizer-aware attribution on a small run would settle this.","Editorial inference: the same score could be computed against intermediate reference checkpoints to ask which examples are most aligned with reaching a specific capability stage, not only the final model.","Editorial inference: the measure could transfer to other model families with dense checkpoints and to multimodal pretraining, where domain labels are less clean and task-specific validation sets are even harder to choose.","Editorial inference: because the paper's own comparison shows task-specific TracIn scores do not recover the crossover, the measure offers a distinct signal that may be useful for diagnosing phase transitions, such as complementing loss-spike detection in training runs."],"forward_implications":["If the crossover is real, pretraining curricula that schedule STEM-heavy data later are given trajectory-level support from the model's own parameter-space geometry, not just from downstream benchmark gains.","The contribution score provides a common reference point—the final parameters—so influence rankings can be compared across arbitrary training stages without task labels or validation sets.","The rising opponent share late in training implies that a substantial fraction of late updates push against the eventual final model, suggesting that late-stage data selection could improve alignment with the final parameters.","Higher-PPL examples matter most in the middle stage, so the trajectory does not follow a simple easy-to-hard curriculum; data difficulty and domain interact with training phase."],"supporting_citations":[{"why":"Supplies the Pythia suite: six model sizes, 154 checkpoints per run, and the actual training data stream from which examples are sampled.","marker":"(Biderman et al., 2023)"},{"why":"Supplies the PolyPythia runs used to test robustness to weight initialization and data ordering, and identifies the anomalous 410M seed-3 run.","marker":"(van der Wal et al., 2025)"},{"why":"Provides TracIn, the SGD-based checkpoint tracing method whose decomposition and self-influence term ground the example-level influence definition and the task-specific comparison.","marker":"(Pruthi et al., 2020)"},{"why":"Provides The Pile corpus from which domain-specific held-out validation sets and classifier-labeled domain examples are drawn.","marker":"(Gao et al., 2020)"},{"why":"Inspires the trajectory-matching formulation that measures alignment between updates and progress toward a final parameter state.","marker":"(Cazenavette et al., 2022)"},{"why":"Motivates scalable influence estimation at large model scale, the tractability goal the checkpoint-based approximation pursues.","marker":"(Park et al., 2023)"},{"why":"Demonstrates that task-specific attribution evaluation is sensitive to evaluation design, motivating the task-agnostic perspective.","marker":"(Jiao et al., 2025)"},{"why":"Defines the influence-function approach to training data attribution that the paper positions itself against by replacing task-conditioned targets with progress toward final parameters.","marker":"(Koh & Liang, 2017)"}],"fun_headline_variants":["Literature data steer early pretraining, STEM late","Task-agnostic measure: influential data shift to STEM","Pretraining influence flips from literature to STEM","No task needed: data influence evolves literature to STEM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The example-level decomposition assumes SGD updates, while the analyzed models were trained with Adam, so the per-example attribution—and therefore the domain crossover—may not reflect the updates that actually moved the model.","fun_headline_variants_meta":{"raw":{"variants":["Literature data steer early pretraining, STEM late","Task-agnostic measure: influential data shift to STEM","Pretraining influence flips from literature to STEM","No task needed: data influence evolves literature to STEM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1561,"prompt_tokens":865,"completion_tokens":696,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":634}},"tokens_in":481,"tokens_out":696,"duration_ms":7637,"temperature":1.0,"reasoning_tokens":634,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:16:33.676177+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain or replay a small Pythia run (e.g., 70M) while storing per-step optimizer states, compute an Adam-faithful per-example attribution by simulating the exact update each example induces through the optimizer, and compare the domain-level rankings with the SGD-based contribution score; if the literature-to-STEM crossover disappears or shifts substantially under the true optimizer, the reported result is an artifact of the decomposition. An intervention that swaps the data schedule—placing STEM data early and literature later—should reverse which domains align with the final parameters if the score tracks causal influence.","supporting_citations":[],"review_version":1}