{"id":"15677638-53a5-41cd-b0b2-20fd971f13f5","arxiv_id":"2412.09538","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new influence-estimation method approximates order-aware data influence with a dot product and reveals that early and late training data matter most.","lead":"The paper introduces a way to estimate how much each training example influenced a model's final behavior while accounting for the exact order in which examples were seen, not just the final model. This 'data value embedding' is a compact per-example vector that can be scored against any test input in real time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central temporal-influence claim rests on an unvalidated bridge: an SGD-derived approximation is applied to AdamW-trained LLMs, and the fidelity tests never check DVE against true TSLOO across training time.","rationale":"The reader's weakest assumption identifies the SGD-to-Adam gap, which is indeed a serious concern: all large-scale experiments use AdamW, while the derivation is explicitly SGD-specific, and the Limitations section acknowledges the mismatch. I agree that this is a load-bearing weakness. However, I would sharpen the concern further: even under SGD, the paper does not validate the temporal dimension that is the central contribution. The MNIST fidelity experiments remove a point only from the last epoch or from all epochs; neither setting tests whether DVE correctly orders the influence of points used at early, middle, and late iterations within a trajectory. Thus the three-regime curve and the early/late selection timing insight are supported only by DVE's own output on AdamW-trained models, with no ground-truth check that this temporal signal corresponds to true TSLOO. This makes the central claim conditional on a validation that has not been reported, rather than on a known false assumption. The paper does have independent support: the recursive computation and checkpointing scheme are nontrivial, the small-scale SGD fidelity results are suggestive, and the data-selection comparisons on CIFAR-10 with Adam show competitive performance. But those results do not establish temporal fidelity. A targeted experiment computing exact TSLOO for early/middle/late removal under both SGD and AdamW would settle the concern. Because this is an addressable empirical gap rather than a demonstrated internal inconsistency, the reader's CONDITIONAL verdict remains appropriate; I would not reject the paper or mark it unverdictable based on this concern alone.","tokens_in":34703,"tokens_out":7557,"duration_ms":76136,"concrete_test":"Run a small-scale controlled fidelity experiment on a model where exact trajectory-specific LOO is computable, e.g., a two-layer MLP or small CNN on MNIST. Train with both plain SGD and AdamW, and for each optimizer choose a sample of training points removed at early (e.g., 5%), middle (e.g., 50%), and late (e.g., 95%) training iterations; retrain the model for each removal keeping the batch order fixed, and compute the exact TSLOO. Report Spearman correlation between DVE scores and ground-truth TSLOO separately for each removal time, plus the correlation between the full DVE time curve and the true time curve. If early/middle correlations are materially lower than late correlations, or if AdamW correlations are near zero, then the temporal regimes and the data-selection timing recommendations in Figures 1(a) and 1(b) are not supported by the evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (2) and Theorem 2 are derived from the exact SGD update theta_{t+1} = theta_t - eta_t * sum_{z in B_t} grad l(theta_t, z). AdamW's update includes first- and second-moment normalization, bias correction, and weight decay, so the recursion (I - eta_k H_k) and the GGN-based backward algorithm do not describe the counterfactual trajectory under AdamW. The Limitations section (Section 6) concedes that DVE is not directly extendable to Adam. Yet the paper's headline empirical claims—the three temporal influence regimes in Figure 1(a) and the early+late data-selection timing result in Figure 1(b)—are computed for Pythia-410M and GPT-2 trained with AdamW, as detailed in Appendix E.1. The fidelity experiments that could support the SGD-to-Adam proxy use only SGD-trained MLP/CNN on MNIST, and they do not test the temporal dimension: 'single epoch removal' removes from the last epoch and 'all-epoch removal' removes from every epoch, so the reported Spearman correlations never isolate early, middle, or late removal times. No comparison reports ground-truth TSLOO versus DVE as a function of ts. Consequently, the central claim that DVE captures the temporal dependence of data influence is not directly validated anywhere, and the large-scale temporal findings rely on an unverified bridge from an SGD derivation to AdamW training dynamics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that classical influence functions are inadequate for modern training pipelines because they assume permutation-invariance with respect to training data. It formalizes trajectory-specific leave-one-out (TSLOO) influence, which measures the change in validation loss when a training point is removed from the specific iteration at which it was used, and proposes data value embedding (DVE) as an efficient approximation. The core identity is Eq. (2): DVE for a training point z* at iteration ts is the product of a learning-rate factor, a product of Hessian terms (I - η_k H_k), and the gradient at z*, so that TSLOO can be approximated by a dot product of DVE with the final-model validation gradient. The paper derives a recursive GGN-based computation, introduces influence checkpointing for parallelism, and compares favorably to LoGRA in efficiency. Experiments include MNIST fidelity checks against exact retraining LOO, mislabeled-data detection and data selection benchmarks, and large-scale studies on Pythia-410M and GPT-2 that suggest three temporal influence regimes and support early+late online data selection. The paper closes by acknowledging that DVE is specifically tailored to SGD and not directly extendable to Adam.","tokens_in":34983,"tokens_out":10388,"duration_ms":99360,"significance":"If the core approximation holds, this is a valuable contribution: DVE provides a validation-agnostic, dot-product-based attribution estimator that is sensitive to training order, and the projected-gradient storage scheme is markedly cheaper than the most efficient influence-function implementation on the reported Pythia run. The claimed empirical phenomena—high influence at early and late training stages and the effectiveness of early+late data selection—are concrete, falsifiable, and potentially useful for data curation. The algorithmic design, including per-layer computation and checkpointing, is thoughtful. The main weakness is that the load-bearing temporal and optimizer claims are not directly validated: the fidelity experiments do not isolate removal time, the large-scale experiments use AdamW while the derivation assumes SGD, and the theory bounds parameter error rather than the loss difference that TSLOO defines. The paper would also benefit from code and seed-level details to make the large-scale results reproducible.","major_comments":[{"comment":"The large-scale experiments in Figures 1, 4, and 9 are run with AdamW (Appendix E.1: optimizer AdamW, learning rate 3e-4, weight decay 0.1), while Eq. (1)–(2) and Theorem 2 are derived from the plain SGD update θ_{t+1} = θ_t − η_t Σ_{z∈B_t} ∇ℓ(θ_t,z). Section 6 concedes that DVE is not directly extendable to Adam due to its normalization terms. Thus the three-regime curve in Figure 1(a) and the early+late data-selection result in Figure 1(b) rely on an unvalidated bridge from an SGD derivation to AdamW training dynamics. Please either extend the recursion to AdamW (for example under a diagonal-preconditioner approximation) or validate DVE against ground-truth TSLOO on small AdamW-trained models across training time; at minimum, the large-scale claims should be labeled as relying on an SGD proxy for Adam.","section":"§5.3, Fig. 1; Appendix E.1; Eq. (2)"},{"comment":"The fidelity experiments do not directly validate the temporal dependence that is the paper's central claim. Single-epoch removal removes a point only from the last epoch, and all-epoch removal removes it from every epoch; neither setting reports DVE versus exact TSLOO as a function of ts. I ask for an experiment in which the same training point is removed at several different iterations (e.g., early, middle, and late training) and the Spearman correlation between DVE scores and exact retraining-based TSLOO is reported for each removal time. Without such a test, the statement that DVE 'captures the temporal dependence of data influence' is not directly supported by the fidelity evidence.","section":"§5.1, Figs. 3, 6, 7"},{"comment":"Theorem 3 bounds the parameter-space error ∥(θ_T − θ'_T) − Δθ∥, but TSLOO is defined as a loss difference and Eq. (1) approximates it by a first-order Taylor expansion of the loss. Bounding the parameter error does not by itself bound |ℓ(θ'_T,z_val) − ℓ(θ_T,z_val) − ∇ℓ(θ_T,z_val)ᵀ Δθ|; an additional Taylor remainder bound is needed. The proof also relies on the unverified assumption H_t ⪯ Λ/√t I, which is asserted with anecdotal justification and is not checked in any of the experimental settings. As stated, the theoretical guarantee is one step removed from the quantity the method claims to approximate.","section":"Appendix C.2, Theorem 3, Eq. (11)"},{"comment":"The key practical claim—that early+late selection recovers 96% of the improvement of continuous selection while reducing computational overhead by more than 5×—is reported without error bars, seed counts, or the total number of training iterations in the main text. Since this result drives the proposed data-curation strategy, please report multiple seeds with variance and specify the full training length and the precise computational-overhead accounting.","section":"Fig. 1(b), §5.3"}],"minor_comments":[{"comment":"The sentence 'While using as a proxy for Adam has proved to be effective in our experiment' is missing its subject and should read 'While using SGD as a proxy for Adam ...'; as written, it also overstates the evidence, since no fidelity experiment uses AdamW.","section":"§6, Limitations"},{"comment":"The sentence 'this assumption aligns with empirical findings in deep learning literature ()' contains an empty citation; either provide the reference or remove the claim.","section":"Appendix C.2, Hessian decay justification"},{"comment":"The text says 'In Figure 7(b), we consider a simplified setting...' but the discussion is about the analytical curves shown in Figure 8(b); the figure reference should be corrected.","section":"Appendix E.3.1"},{"comment":"The claim that influence checkpointing 'reduces the overall computational cost by K times' should be stated as reducing wall-clock time when K parallel workers are available; the total number of FLOPs is not reduced by a factor of K.","section":"§4.4, Appendix C.7"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to interest the data-attribution community, and the DVE formulation is a genuine step toward order-aware influence estimation. The main risk is not the core SGD derivation but its unvalidated application to AdamW-trained LLMs, together with the absence of a time-indexed fidelity test. I would recommend requiring the additional experiments described in the major comments before acceptance. I would also encourage the editor to ask for code and configuration details, since the large-scale experiments are expensive and otherwise difficult to reproduce."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth taking seriously: a per-training-point data value embedding that lets you approximate trajectory-specific LOO by a dot product with the test gradient. The recursive backward algorithm, influence checkpointing, and the empirical finding of three temporal influence regimes are the new pieces. Fidelity on MNIST under SGD is genuinely good, with high Spearman correlation to ground-truth LOO and clear improvements over influence functions.\n\nThe soft spots are real but not fatal. The biggest is the optimizer bridge: everything derived in Section 3 is for plain SGD, yet all the large-scale results train with AdamW. The Limitations section admits this but doesn't test it. The fidelity experiments never vary the removal time: single-epoch removal only removes from the last epoch, and all-epoch removal sums over epochs, so the temporal dimension of the central claim is never directly validated against ground truth. That is a gap a referee should push on.\n\nThe theory is also a bit weaker than advertised. Theorem 3 bounds the difference in parameters, not the difference in validation loss that TSLOO defines, and it assumes a learning-rate schedule (eta_max = C/sqrt(T)) that does not match the cosine schedule used in the large-scale runs. Minor: no error bars in Figure 1, and no code released.\n\nOn balance I think this deserves a serious referee. The method is clearly presented, the related work is cited honestly (Hara et al. gets proper credit for the unrolled approximation), and the computational machinery is a real advance. The empirical timing insights are plausible but currently supported only by the AdamW proxy, so the referee should ask for a temporal fidelity check under SGD, and ideally a small-scale check of whether SGD-derived embeddings predict AdamW loss changes at all. None of this is a takedown; it is a paper that would benefit from one more round of controlled experiments.","headline":"A solid, honest contribution to order-aware data attribution with one under-supported bridge: the SGD-derived embedding is used to make claims about AdamW-trained LLMs.","tokens_in":35503,"tokens_out":1987,"would_cite":true,"duration_ms":19659,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training-data influence can be approximated as a dot product between a precomputed data value embedding and the final model gradient.","keywords":["data influence estimation","trajectory-specific leave-one-out","data value embedding","training order dependence","foundation model pretraining","online data selection","influence functions"],"falsifier":"Retrain the small MLP fidelity experiment of Section 5.1 switching the optimizer from SGD to AdamW with the same batch order, and measure the Spearman correlation between ground-truth trajectory-specific LOO and data value embedding scores; a correlation collapse would show the method fails outside SGD. Alternatively, rerun the Pythia-410M pretraining with an actual canonicalized SGD schedule and check whether the three-regime curve matches the AdamW-trained results.","tokens_in":34502,"feed_emoji":"🎯","tokens_out":6233,"duration_ms":57803,"temperature":0.7,"pith_summary":"The paper claims that the influence of a training point on the final model depends on when in training it appeared, and that this temporal dependence can be captured cheaply. It formalizes trajectory-specific leave-one-out influence, which measures the effect of deleting a point from the exact iteration where it was used, and shows that this quantity can be approximated by a dot product between a per-training-point data value embedding and the test point's final gradient. The embedding is computed once, backward through the training trajectory, so attributions for new test points are near-instantaneous. The method also reveals distinct phases of influence in LLM pretraining, which the paper uses to argue that online data selection only needs to run during the early and late stages.","feed_headline":"Data influence becomes a dot product of precomputed embeddings","feed_subtitle":"Precompute one vector per training point; then any test point's attribution is a single GPU dot product.","key_machinery":"The load-bearing object is the data value embedding $\\mathrm{DVEmb}^{(t_s)}(z_*)$, a vector in parameter-gradient space defined at each training iteration as the gradient of the point at that iteration propagated forward through the products $(I-\\eta_k H_k)$ over all subsequent iterations. The central identity is Equation (2) in the paper, which converts any test-time attribution into a dot product with this embedding. To make the embedding computable, the paper relies on the Generalized Gauss-Newton approximation of the Hessians, a recursive backward algorithm (Theorem 2) that expresses each embedding in terms of later embeddings and gradient similarities, and a layer-wise independence assumption plus random projections for storage. Influence checkpointing parallelizes the backward pass by running the recursion from multiple checkpoints, and also enables tracking how a fixed point's influence changes across intermediate checkpoints.","core_discovery":"On its own terms, the paper establishes that the trajectory-specific leave-one-out loss change from removing a training point $z_*$ used at iteration $t_s$ is well approximated by $\\nabla\\ell(\\theta_T,z^{(\\text{val})})^\\top \\mathrm{DVEmb}^{(t_s)}(z_*)$, where $\\mathrm{DVEmb}^{(t_s)}(z_*) = \\eta_{t_s}\\left[\\prod_{k=t_s+1}^{T-1}(I-\\eta_k H_k)\\right]\\nabla\\ell(\\theta_{t_s},z_*)$ accumulates the point's gradient through the remaining training steps. Using the Generalized Gauss-Newton approximation for each Hessian, the embeddings can be computed recursively from the final iteration backward, and gradient decomposition plus random projection keeps storage and compute within reach of foundation-model training. The consequence is that influence estimates are no longer blind to data ordering: identical points at different training times receive different scores, and duplicate points no longer collapse to one attribution. Empirically, the paper reports a consistent three-regime pattern in LLM pretraining and shows that applying online data selection only during the early and late regimes recovers the bulk of the benefit of continuous selection.","pith_inferences":["If the SGD-derived embedding transfers to optimizers like Adam or Lion, the same dot-product machinery would give a practical attribution tool for routine LLM training runs; a direct test is to compare embedding scores against ground-truth LOO on a small AdamW-trained model.","The low-influence basin suggests that gradient similarity to validation data is least informative mid-training; this might mean that curriculum or data-pruning schedules should concentrate utility measurements at the start and near the end, which is testable by ablating selection windows at finer granularity.","The recursive interpretation (an early point's influence is reduced when later similar points appear) implies a notion of influence saturation that could be used to estimate when a corpus is redundant, e.g., by measuring how much embedding norms decay after repeated similar examples.","The checkpoint reconstruction from saved checkpoints alone (Appendix C.10) suggests that the method can be retrofitted to existing training logs that did not store per-step gradients, assuming one update per checkpoint."],"forward_implications":["If the approximation holds, influence attribution for any test query becomes a single dot product per training point, so no retraining and no advance knowledge of validation data are needed.","Data selection during only the high-influence early phase (under 4% of training) yields roughly half of the gain of continuous selection, and early-plus-late selection yields most of it at over 5x less compute.","The three-regime influence pattern gives a principled way to schedule online data curation and checkpoint-based attribution in LLM pretraining.","Influence functions, which assign identical scores to identical points regardless of position, are shown to be a limiting case of the trajectory-based view under strong stationarity assumptions, so their known instability is explained rather than incidental."],"supporting_citations":[{"why":"Defines trajectory-specific LOO as SGD-influence and supplies the unrolled differentiation approximation in Equation (1) that the data value embedding is built on.","marker":"Hara et al. (2019)"},{"why":"Introduces the influence function baseline and the self-influence concept used for qualitative evaluation and comparison.","marker":"Koh & Liang (2017)"},{"why":"Justifies the Generalized Gauss-Newton approximation of the Hessian that makes Theorem 2's recursive computation possible.","marker":"Martens (2020)"},{"why":"Provides the layer-wise independence assumption and the influence-function scaling approach that the paper adapts for tractable embedding computation.","marker":"Grosse et al. (2023)"},{"why":"Gives the LoGRA algorithm and the gradient-decomposition/random-projection implementation that the paper compares against in efficiency and extends for data value embedding.","marker":"Choe et al. (2024)"},{"why":"Supplies the gradient decomposition technique used to store per-sample gradients without extra backpropagation passes during training.","marker":"Wang et al. (2024c)"},{"why":"The online data-selection strategy based on gradient cosine similarity to a validation batch that the paper uses to test timing-dependent selection.","marker":"Fan et al. (2024)"},{"why":"The decaying learning-rate schedule that anchors the theoretical error bound for the unrolled differentiation estimator.","marker":"Vaswani (2017)"}],"fun_headline_variants":["Training data influence now tracks the optimization path","Influence functions go temporal: one dot product per test point","Data value embeddings make influence a dot product","Early and late training data matter more for models","Trajectory-aware LOO influence via data value embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation assumes plain SGD, yet the large-scale experiments are run with AdamW; if the SGD-based embedding does not faithfully represent influence under Adam-style updates, the reported temporal regimes and the data-selection timing conclusions are not supported.","fun_headline_variants_meta":{"raw":{"variants":["Training data influence now tracks the optimization path","Influence functions go temporal: one dot product per test point","Data value embeddings make influence a dot product","Early and late training data matter more for models","Trajectory-aware LOO influence via data value embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1796,"prompt_tokens":1062,"completion_tokens":734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":670}},"tokens_in":678,"tokens_out":734,"duration_ms":6783,"temperature":1.0,"reasoning_tokens":670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:56:53.669073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the small MLP fidelity experiment of Section 5.1 switching the optimizer from SGD to AdamW with the same batch order, and measure the Spearman correlation between ground-truth trajectory-specific LOO and data value embedding scores; a correlation collapse would show the method fails outside SGD. Alternatively, rerun the Pythia-410M pretraining with an actual canonicalized SGD schedule and check whether the three-regime curve matches the AdamW-trained results.","supporting_citations":[{"cited_title":"Understanding black-box predictions via influence functions","cited_arxiv_id":null,"evidence_quote":"Introduces the influence function baseline and the self-influence concept used for qualitative evaluation and comparison."}],"review_version":1}