Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Capturing the Temporal Dependence of Training Data Influence

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training-data influence can be approximated as a dot product between a precomputed data value embedding and the final model gradient.

desk verdict A solid, honest contribution to order-aware data attribution with one under-supported bridge: the SGD-derived embedding is used to make claims about AdamW-trained LLMs. read the letter →

arxiv 2412.09538 v1 pith:VV5TQXKP submitted 2024-12-12 cs.LG stat.ML

Jiachen T. Wang , Dawn Song , James Zou , Prateek Mittal , Ruoxi Jia This is my paper · ORCID
classification cs.LGstat.ML
keywords datainfluenceestimationtrajectory-specificleave-one-outvalueembeddingtrainingorderdependencefoundationmodelpretrainingonlineselectionfunctions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the influence of a training point on the final model depends on when in training it appeared, and that this temporal dependence can be captured cheaply. It formalizes trajectory-specific leave-one-out influence, which measures the effect of deleting a point from the exact iteration where it was used, and shows that this quantity can be approximated by a dot product between a per-training-point data value embedding and the test point's final gradient. The embedding is computed once, backward through the training trajectory, so attributions for new test points are near-instantaneous. The method also reveals distinct phases of influence in LLM pretraining, which the paper uses to argue that online data selection only needs to run during the early and late stages.

What carries the argument

The load-bearing object is the data value embedding $\mathrm{DVEmb}^{(t_s)}(z_*)$, a vector in parameter-gradient space defined at each training iteration as the gradient of the point at that iteration propagated forward through the products $(I-\eta_k H_k)$ over all subsequent iterations. The central identity is Equation (2) in the paper, which converts any test-time attribution into a dot product with this embedding. To make the embedding computable, the paper relies on the Generalized Gauss-Newton approximation of the Hessians, a recursive backward algorithm (Theorem 2) that expresses each embedding in terms of later embeddings and gradient similarities, and a layer-wise independence assumption plus random projections for storage. Influence checkpointing parallelizes the backward pass by running the recursion from multiple checkpoints, and also enables tracking how a fixed point's influence changes across intermediate checkpoints.

What would settle it

Retrain the small MLP fidelity experiment of Section 5.1 switching the optimizer from SGD to AdamW with the same batch order, and measure the Spearman correlation between ground-truth trajectory-specific LOO and data value embedding scores; a correlation collapse would show the method fails outside SGD. Alternatively, rerun the Pythia-410M pretraining with an actual canonicalized SGD schedule and check whether the three-regime curve matches the AdamW-trained results.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the trajectory-specific leave-one-out loss change from removing a training point $z_*$ used at iteration $t_s$ is well approximated by $\nabla\ell(\theta_T,z^{(\text{val})})^\top \mathrm{DVEmb}^{(t_s)}(z_*)$, where $\mathrm{DVEmb}^{(t_s)}(z_*) = \eta_{t_s}\left[\prod_{k=t_s+1}^{T-1}(I-\eta_k H_k)\right]\nabla\ell(\theta_{t_s},z_*)$ accumulates the point's gradient through the remaining training steps. Using the Generalized Gauss-Newton approximation for each Hessian, the embeddings can be computed recursively from the final iteration backward, and gradient decomposition plus random projection keeps storage and compute within reach of foundation-model training. The consequence is that influence estimates are no longer blind to data ordering: identical points at different training times receive different scores, and duplicate points no longer collapse to one attribution. Empirically, the paper reports a consistent three-regime pattern in LLM pretraining and shows that applying online data selection only during the early and late regimes recovers the bulk of the benefit of continuous selection.

Load-bearing premise

The derivation assumes plain SGD, yet the large-scale experiments are run with AdamW; if the SGD-based embedding does not faithfully represent influence under Adam-style updates, the reported temporal regimes and the data-selection timing conclusions are not supported.

Editorial extensions

If this is right

  • If the approximation holds, influence attribution for any test query becomes a single dot product per training point, so no retraining and no advance knowledge of validation data are needed.
  • Data selection during only the high-influence early phase (under 4% of training) yields roughly half of the gain of continuous selection, and early-plus-late selection yields most of it at over 5x less compute.
  • The three-regime influence pattern gives a principled way to schedule online data curation and checkpoint-based attribution in LLM pretraining.
  • Influence functions, which assign identical scores to identical points regardless of position, are shown to be a limiting case of the trajectory-based view under strong stationarity assumptions, so their known instability is explained rather than incidental.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the SGD-derived embedding transfers to optimizers like Adam or Lion, the same dot-product machinery would give a practical attribution tool for routine LLM training runs; a direct test is to compare embedding scores against ground-truth LOO on a small AdamW-trained model.
  • The low-influence basin suggests that gradient similarity to validation data is least informative mid-training; this might mean that curriculum or data-pruning schedules should concentrate utility measurements at the start and near the end, which is testable by ablating selection windows at finer granularity.
  • The recursive interpretation (an early point's influence is reduced when later similar points appear) implies a notion of influence saturation that could be used to estimate when a corpus is redundant, e.g., by measuring how much embedding norms decay after repeated similar examples.
  • The checkpoint reconstruction from saved checkpoints alone (Appendix C.10) suggests that the method can be retrofitted to existing training logs that did not store per-step gradients, assuming one update per checkpoint.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper argues that classical influence functions are inadequate for modern training pipelines because they assume permutation-invariance with respect to training data. It formalizes trajectory-specific leave-one-out (TSLOO) influence, which measures the change in validation loss when a training point is removed from the specific iteration at which it was used, and proposes data value embedding (DVE) as an efficient approximation. The core identity is Eq. (2): DVE for a training point z* at iteration ts is the product of a learning-rate factor, a product of Hessian terms (I - η_k H_k), and the gradient at z*, so that TSLOO can be approximated by a dot product of DVE with the final-model validation gradient. The paper derives a recursive GGN-based computation, introduces influence checkpointing for parallelism, and compares favorably to LoGRA in efficiency. Experiments include MNIST fidelity checks against exact retraining LOO, mislabeled-data detection and data selection benchmarks, and large-scale studies on Pythia-410M and GPT-2 that suggest three temporal influence regimes and support early+late online data selection. The paper closes by acknowledging that DVE is specifically tailored to SGD and not directly extendable to Adam.

Significance. If the core approximation holds, this is a valuable contribution: DVE provides a validation-agnostic, dot-product-based attribution estimator that is sensitive to training order, and the projected-gradient storage scheme is markedly cheaper than the most efficient influence-function implementation on the reported Pythia run. The claimed empirical phenomena—high influence at early and late training stages and the effectiveness of early+late data selection—are concrete, falsifiable, and potentially useful for data curation. The algorithmic design, including per-layer computation and checkpointing, is thoughtful. The main weakness is that the load-bearing temporal and optimizer claims are not directly validated: the fidelity experiments do not isolate removal time, the large-scale experiments use AdamW while the derivation assumes SGD, and the theory bounds parameter error rather than the loss difference that TSLOO defines. The paper would also benefit from code and seed-level details to make the large-scale results reproducible.

major comments (4)
  1. [§5.3, Fig. 1; Appendix E.1; Eq. (2)] The large-scale experiments in Figures 1, 4, and 9 are run with AdamW (Appendix E.1: optimizer AdamW, learning rate 3e-4, weight decay 0.1), while Eq. (1)–(2) and Theorem 2 are derived from the plain SGD update θ_{t+1} = θ_t − η_t Σ_{z∈B_t} ∇ℓ(θ_t,z). Section 6 concedes that DVE is not directly extendable to Adam due to its normalization terms. Thus the three-regime curve in Figure 1(a) and the early+late data-selection result in Figure 1(b) rely on an unvalidated bridge from an SGD derivation to AdamW training dynamics. Please either extend the recursion to AdamW (for example under a diagonal-preconditioner approximation) or validate DVE against ground-truth TSLOO on small AdamW-trained models across training time; at minimum, the large-scale claims should be labeled as relying on an SGD proxy for Adam.
  2. [§5.1, Figs. 3, 6, 7] The fidelity experiments do not directly validate the temporal dependence that is the paper's central claim. Single-epoch removal removes a point only from the last epoch, and all-epoch removal removes it from every epoch; neither setting reports DVE versus exact TSLOO as a function of ts. I ask for an experiment in which the same training point is removed at several different iterations (e.g., early, middle, and late training) and the Spearman correlation between DVE scores and exact retraining-based TSLOO is reported for each removal time. Without such a test, the statement that DVE 'captures the temporal dependence of data influence' is not directly supported by the fidelity evidence.
  3. [Appendix C.2, Theorem 3, Eq. (11)] Theorem 3 bounds the parameter-space error ∥(θ_T − θ'_T) − Δθ∥, but TSLOO is defined as a loss difference and Eq. (1) approximates it by a first-order Taylor expansion of the loss. Bounding the parameter error does not by itself bound |ℓ(θ'_T,z_val) − ℓ(θ_T,z_val) − ∇ℓ(θ_T,z_val)ᵀ Δθ|; an additional Taylor remainder bound is needed. The proof also relies on the unverified assumption H_t ⪯ Λ/√t I, which is asserted with anecdotal justification and is not checked in any of the experimental settings. As stated, the theoretical guarantee is one step removed from the quantity the method claims to approximate.
  4. [Fig. 1(b), §5.3] The key practical claim—that early+late selection recovers 96% of the improvement of continuous selection while reducing computational overhead by more than 5×—is reported without error bars, seed counts, or the total number of training iterations in the main text. Since this result drives the proposed data-curation strategy, please report multiple seeds with variance and specify the full training length and the precise computational-overhead accounting.
minor comments (4)
  1. [§6, Limitations] The sentence 'While using as a proxy for Adam has proved to be effective in our experiment' is missing its subject and should read 'While using SGD as a proxy for Adam ...'; as written, it also overstates the evidence, since no fidelity experiment uses AdamW.
  2. [Appendix C.2, Hessian decay justification] The sentence 'this assumption aligns with empirical findings in deep learning literature ()' contains an empty citation; either provide the reference or remove the claim.
  3. [Appendix E.3.1] The text says 'In Figure 7(b), we consider a simplified setting...' but the discussion is about the analytical curves shown in Figure 8(b); the figure reference should be corrected.
  4. [§4.4, Appendix C.7] The claim that influence checkpointing 'reduces the overall computational cost by K times' should be stated as reducing wall-clock time when K parallel workers are available; the total number of FLOPs is not reduced by a factor of K.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the data value embedding is defined as the test-independent factor of a previously established unrolled-differentiation approximation, and the method is validated against externally computed trajectory-specific LOO.

full rationale

The paper's derivation chain is self-contained and not circular. Trajectory-specific LOO (Definition 1) is defined by counterfactual retraining with a data point removed at a specific iteration. Equation (1) is the standard first-order unrolled-differentiation approximation, credited to Hara et al. (2019), and Equation (2) defines DVEmb as exactly the factor multiplying the test gradient in that approximation. Thus the central dot-product formula is true by construction, but this is a definition, not a validation: the paper's fidelity experiments (Section 5.1 and Appendix E.2) compare DVEmb-based scores against ground-truth TSLOO values obtained by actual retraining, on MNIST MLPs and CNNs, with no fitted constants and no prediction of a quantity used to fit the method. Theorem 2 is an algebraic recursive rearrangement of the same definition under the generalized Gauss-Newton approximation, so it introduces no new empirical circularity. The main weakness flagged in the Limitations section is that DVEmb is derived for SGD while the large-scale experiments use AdamW; this is an external validity gap, not a circular step. Self-citations, such as Wang et al. (2024c) for the gradient decomposition trick, are implementation details and are not load-bearing for the core approximation. No uniqueness theorem or ansatz from the authors' prior work is invoked to force the choice of data value embedding. Therefore the central claim does not reduce to its own inputs.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

The central derivation rests on several approximations and assumptions: the GGN replacement of the Hessian, layer-wise independence, the Hessian spectral decay rate used in the error bound, a learning rate schedule that does not match the experiments, bounded gradients and Lipschitz Hessians, the SGD-to-Adam proxy, and the unproven preservation of inner products under random projections. None of these are fitted to data, but their validity in the large-scale setting is not established.

free parameters (2)
  • Projection dimension per layer = 1024
    All large-scale experiments set the random projection dimension p̃ to 1024 per layer, a user-chosen hyperparameter that trades storage against approximation fidelity. The paper provides no theoretical guarantee for this choice and only shows robustness against larger projections.
  • Checkpoint count K = not reported for main experiments
    The influence checkpointing algorithm uses K evenly spaced checkpoints, a user-chosen parallelization parameter. Its value is not reported for the main experiments, so the reported speedups and checkpoint-based dynamics analysis depend on an undisclosed choice.
assumptions (7)
  • domain assumption Generalized Gauss-Newton (GGN) approximation H_t ≈ Σ_{z∈B_t} ∇ℓ(θ_t,z) ∇ℓ(θ_t,z)^T for the Hessian
    Theorem 2 and Algorithm 1 replace every Hessian in the unrolled product with a batch gradient outer-product. The approximation is standard for cross-entropy loss, but its accuracy at every step of a deep-network trajectory is not validated independently.
  • domain assumption Layer-wise independence of gradients
    Section 4.3 computes data value embeddings per layer, borrowing the EK-FAC assumption that gradients across layers are independent, so cross-layer Hessian blocks are ignored. This reduces cost but is an approximation.
  • ad hoc to paper Hessian spectral norm decay H_t ⪯ Λ/√t I
    Appendix C.2 assumes the Hessian norm decreases as 1/√t to obtain the uniform error bound. This behavior is not guaranteed for the warmup and cosine-decay schedules used in experiments.
  • domain assumption Learning rate schedule η_t = η_max/√t with η_max = O(1/√T)
    The error-bound theorem uses this schedule, but the actual large-scale runs use AdamW with 2000-step warmup and cosine decay, so the theorem does not cover the experiments.
  • standard math Bounded gradient norm ∥∇ℓ∥ ≤ G and L-Lipschitz Hessian
    Standard regularity assumptions in Appendix C.2 used to bound the unrolling error; they are plausible but not verified for the trained networks.
  • domain assumption SGD can serve as a proxy for AdamW influence
    The paper states this in Limitations. All Pythia and GPT-2 experiments use AdamW, while the DVEmb formula is derived for SGD.
  • domain assumption Random projections preserve the gradient inner products needed for recursion
    Section 4.2 and Appendix C.6 project activations and output derivatives; the recursion and final dot product use projected vectors, and the paper concedes there is no multiplicative guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Capturing the Temporal Dependence of Training Data Influence." pith.science (2026). https://pith.science/paper/VV5TQXKP

@misc{pith2026241209538,
  author       = {Pith},
  title        = {Pith review of: Capturing the Temporal Dependence of Training Data Influence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VV5TQXKP}},
  note         = {Machine review of arXiv:2412.09538}
}
read the original abstract

Traditional data influence estimation methods, like influence function, assume that learning algorithms are permutation-invariant with respect to training data. However, modern training paradigms, especially for foundation models using stochastic algorithms and multi-stage curricula, are sensitive to data ordering, thus violating this assumption. This mismatch renders influence functions inadequate for answering a critical question in machine learning: How can we capture the dependence of data influence on the optimization trajectory during training? To address this gap, we formalize the concept of trajectory-specific leave-one-out (LOO) influence, which quantifies the impact of removing a data point from a specific iteration during training, accounting for the exact sequence of data encountered and the model's optimization trajectory. However, exactly evaluating the trajectory-specific LOO presents a significant computational challenge. To address this, we propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO. Specifically, we compute a training data embedding that encapsulates the cumulative interactions between data and the evolving model parameters. The LOO can then be efficiently approximated through a simple dot-product between the data value embedding and the gradient of the given test data. As data value embedding captures training data ordering, it offers valuable insights into model training dynamics. In particular, we uncover distinct phases of data influence, revealing that data points in the early and late stages of training exert a greater impact on the final model. These insights translate into actionable strategies for managing the computational overhead of data selection by strategically timing the selection process, potentially opening new avenues in data curation research.

Figures

Figures reproduced from arXiv: 2412.09538 by the authors.

Figure 1
Figure 1. (a) Average data influence scores computed from data value embedding per training batch, measured against the final model’s loss on Pile’s validation set. Setting: Pythia-410M trained on 1% of Pile. (b) Comparison of online data selection strategies for training Pythia-410M on Pile. All strategies use gradient cosine similarity to Pile’s validation set to select high-quality training batches (Fan et al., 2024), and … view at source ↗
Figure 2
Figure 2. An illustrative example of data value embedding for a 2-step training. The influence of [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The correlation between ground-truth LOO when the MLP is trained for 3 epochs and the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Evolution of influence scores across training checkpoints. The x-axis shows train￾ing iterations, and the y-axis shows the average influence of training examples on each check￾point. Examples are grouped according to the iterations they are being trained on [PITH_FULL…
Figure 5
Figure 5. Figure 5: Visualization of (left) the evolution of the top-3 most valuable training data points identified [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: The correlation between ground-truth LOO when the MLP is trained for [PITH_FULL_IMAGE:figures/full_fig_p032_6.png]
Figure 7
Figure 7. Figure 7: Scatter plot showing the correlation between ground-truth LOO and data value embedding [PITH_FULL_IMAGE:figures/full_fig_p033_7.png]
Figure 8
Figure 8. Figure 8: (a) (same as [PITH_FULL_IMAGE:figures/full_fig_p035_8.png]
Figure 9
Figure 9. Figure 9: Average data influence scores per training batch, measured against the final model’s loss [PITH_FULL_IMAGE:figures/full_fig_p036_9.png]
Figure 10
Figure 10. Figure 10: Average data influence scores per training batch, measured against the final model’s loss [PITH_FULL_IMAGE:figures/full_fig_p037_10.png]
Figure 11
Figure 11. Figure 11: Loss curve for the training. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_11.png]
Figure 12
Figure 12. Figure 12: (a) Comparison of Spearman correlation between data influence scores as a function of [PITH_FULL_IMAGE:figures/full_fig_p038_12.png]
Figure 13
Figure 13. Figure 13: Examples of training data from the Pile dataset identified as potentially problematic by [PITH_FULL_IMAGE:figures/full_fig_p039_13.png]
Figure 14
Figure 14. Figure 14: Additional examples of training data identified as potentially problematic, showing text [PITH_FULL_IMAGE:figures/full_fig_p040_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Better Training Data Attribution via Better Inverse Hessian-Vector Products

    cs.LG 2025-07 conditional novelty 6.0 of 10

    ASTRA, an EKFAC-preconditioned Neumann series iteration, computes more accurate inverse Hessian-vector products and improves training data attribution scores over EKFAC baselines.

  2. Newfluence: Boosting Model interpretability and Understanding in High Dimensions

    stat.ML 2025-07 conditional novelty 6.0 of 10

    In high-dimensional regression, classical influence functions underestimate true leave-one-out influence by a per-point factor, and the proposed Newfluence estimator corrects this bias.

Reference graph

Works this paper leans on

80 extracted references · 51 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  5. [5]

    Fundamentals of Task-Agnostic Data Valuation

    Mohammad Mohammadi Amiri, Frederic Berdoz, and Ramesh Raskar. Fundamentals of task-agnostic data valuation. arXiv preprint arXiv:2208.12354, 2022

    work page Pith review arXiv 2022

  6. [6]

    If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35: 0 17953--17967, 2022

    Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35: 0 17953--17967, 2022

    2022

  7. [7]

    Training data attribution via approximate unrolled differentation

    Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate unrolled differentation. arXiv preprint arXiv:2405.12186, 2024

    arXiv 2024

  8. [8]

    Relatif: Identifying explanatory training samples via relative influence

    Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference on Artificial Intelligence and Statistics, pp.\ 1899--1909. PMLR, 2020

    work page 1909

Show all 80 references
  1. [9]

    Approximate confidence intervals

    MS Bartlett. Approximate confidence intervals. Biometrika, 40 0 (1/2): 0 12--19, 1953

    1953

  2. [10]

    Influence functions in deep learning are fragile

    Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint arXiv:2006.14651, 2020

    2006 arXiv

  3. [11]

    Stability and generalization

    Olivier Bousquet and Andr \'e Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2: 0 499--526, 2002

    2002

  4. [12]

    Approximating the shapley value using stratified empirical bernstein sampling

    Mark Alexander Burgess and Archie C Chapman. Approximating the shapley value using stratified empirical bernstein sampling. In IJCAI, pp.\ 73--81, 2021

    2021

  5. [13]

    Multi-stage influence function

    Hongge Chen, Si Si, Yang Li, Ciprian Chelba, Sanjiv Kumar, Duane Boning, and Cho-Jui Hsieh. Multi-stage influence function. Advances in Neural Information Processing Systems, 33: 0 12732--12742, 2020

    2020

  6. [14]

    Hydra: Hypergradient data relevance analysis for interpreting deep neural networks

    Yuanyuan Chen, Boyang Li, Han Yu, Pengcheng Wu, and Chunyan Miao. Hydra: Hypergradient data relevance analysis for interpreting deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 7081--7089, 2021

    2021

  7. [15]

    What is your data worth to gpt? llm-scale data valuation with influence functions

    Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, et al. What is your data worth to gpt? llm-scale data valuation with influence functions. arXiv preprint arXiv:2405.13954, 2024

    2024 arXiv

  8. [16]

    Characterizations of an empirical influence function for detecting influential cases in regression

    R Dennis Cook and Sanford Weisberg. Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22 0 (4): 0 495--508, 1980

    1980

  9. [17]

    Stochastic amortization: A unified approach to accelerate feature and data attribution

    Ian Covert, Chanwoo Kim, Su-In Lee, James Zou, and Tatsunori Hashimoto. Stochastic amortization: A unified approach to accelerate feature and data attribution. arXiv preprint arXiv:2401.15866, 2024

    2024 arXiv

  10. [18]

    Computational copyright: Towards a royalty model for ai music generation platforms

    Junwei Deng and Jiaqi Ma. Computational copyright: Towards a royalty model for ai music generation platforms. arXiv preprint arXiv:2312.06646, 2023

    2023

  11. [19]

    Efficient ensembles improve training data attribution

    Junwei Deng, Ting-Wei Li, Shichang Zhang, and Jiaqi Ma. Efficient ensembles improve training data attribution. arXiv preprint arXiv:2405.17293, 2024

    2024 arXiv

  12. [20]

    Understanding forgetting in continual learning with linear regression

    Meng Ding, Kaiyi Ji, Di Wang, and Jinhui Xu. Understanding forgetting in continual learning with linear regression. In Forty-first International Conference on Machine Learning, 2024

    2024

  13. [21]

    Calibrating noise to sensitivity in private data analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp.\ 265--284. Springer, 2006

    2006

  14. [22]

    Revisiting the fragility of influence functions

    Jacob R Epifano, Ravi P Ramachandran, Aaron J Masino, and Ghulam Rasool. Revisiting the fragility of influence functions. Neural Networks, 162: 0 581--588, 2023

    2023

  15. [23]

    How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp.\ 4028--4079

    Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp.\ 4028--4079. PMLR, 2022

    2022

  16. [24]

    Doge: Domain reweighting with generalization estimation

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. In Forty-first International Conference on Machine Learning, 2024

    2024

  17. [25]

    What neural networks memorize and why: Discovering the long tail via influence estimation

    Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 0 2881--2891, 2020

    2020

  18. [26]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  19. [27]

    Data shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pp.\ 2242--2251. PMLR, 2019

    2019

  20. [28]

    A distributional framework for data valuation

    Amirata Ghorbani, Michael Kim, and James Zou. A distributional framework for data valuation. In International Conference on Machine Learning, pp.\ 3535--3544. PMLR, 2020

    2020

  21. [29]

    Studying large language model generalization with influence functions

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023

    2023 arXiv

  22. [30]

    Fastif: Scalable influence functions for efficient model interpretation and debugging

    Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.\ 10333--10350, 2021

    2021

  23. [31]

    How to start training: The effect of initialization and architecture

    Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. Advances in neural information processing systems, 31, 2018

    2018

  24. [32]

    Data cleansing for models trained with sgd

    Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data cleansing for models trained with sgd. Advances in Neural Information Processing Systems, 32, 2019

    2019

  25. [33]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.\ 1026--1034, 2015

    2015

  26. [34]

    Estimation of the shapley value by ergodic sampling

    Ferenc Ill \'e s and P \'e ter Ker \'e nyi. Estimation of the shapley value by ergodic sampling. arXiv preprint arXiv:1906.05224, 2019

    1906

  27. [35]

    Datamodels: Predicting predictions from training data

    Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint arXiv:2202.00622, 2022

    2022 arXiv

  28. [36]

    Efficient task-specific data valuation for nearest neighbor algorithms

    Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. Proceedings of the VLDB Endowment, 2019 a

    2019

  29. [37]

    Towards efficient data valuation based on the shapley value

    Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve G \"u rel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics...

    2019

  30. [38]

    Opendataval: a unified benchmark for data valuation

    Kevin Jiang, Weixin Liang, James Y Zou, and Yongchan Kwon. Opendataval: a unified benchmark for data valuation. Advances in Neural Information Processing Systems, 36, 2023

    2023

  31. [39]

    Lava: Data valuation without pre-specified learning algorithms

    Hoang Anh Just, Feiyang Kang, Tianhao Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. In The Eleventh International Conference on Learning Representations, 2022

    2022

  32. [40]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pp.\ 1885--1894. PMLR, 2017

    2017

  33. [41]

    Beta shapley: a unified and noise-reduced data valuation framework for machine learning

    Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pp.\ 8780--8802. PMLR, 2022

    2022

  34. [42]

    Data-oob: Out-of-bag estimate as a simple and efficient data value

    Yongchan Kwon and James Zou. Data-oob: Out-of-bag estimate as a simple and efficient data value. ICML, 2023

    2023

  35. [43]

    Efficient computation and analysis of distributional shapley values

    Yongchan Kwon, Manuel A Rivas, and James Zou. Efficient computation and analysis of distributional shapley values. In International Conference on Artificial Intelligence and Statistics, pp.\ 793--801. PMLR, 2021

    2021

  36. [44]

    Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models

    Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  37. [45]

    Handwritten digit recognition with a back-propagation network

    Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems, 2, 1989

    1989

  38. [46]

    Faster approximation of probabilistic and distributional values via least squares

    Weida Li and Yaoliang Yu. Faster approximation of probabilistic and distributional values via least squares. In The Twelfth International Conference on Learning Representations, 2023

    2023

  39. [47]

    Robust data valuation with weighted banzhaf values

    Weida Li and Yaoliang Yu. Robust data valuation with weighted banzhaf values. Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [48]

    Measuring the effect of training data on deep learning predictions via randomized experiments

    Jinkun Lin, Anqi Zhang, Mathias L \'e cuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. Measuring the effect of training data on deep learning predictions via randomized experiments. In International Conference on Machine Learning, pp.\ 13468--13504. PMLR, 2022

    2022

  41. [49]

    New insights and perspectives on the natural gradient method

    James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020

    2020

  42. [50]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  43. [51]

    Sampling permutations for shapley value estimation

    Rory Mitchell, Joshua Cooper, Eibe Frank, and Geoffrey Holmes. Sampling permutations for shapley value estimation. 2022

    2022

  44. [52]

    A bayesian approach to analysing training data attribution in deep learning

    Elisa Nguyen, Minjoon Seo, and Seong Joon Oh. A bayesian approach to analysing training data attribution in deep learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  45. [53]

    Data valuation without training of a model

    Ki Nohyun, Hoyong Choi, and Hye Won Chung. Data valuation without training of a model. In The Eleventh International Conference on Learning Representations, 2022

    2022

  46. [54]

    A multilinear sampling algorithm to estimate shapley values

    Ramin Okhrati and Aldo Lipani. A multilinear sampling algorithm to estimate shapley values. In 2020 25th International Conference on Pattern Recognition (ICPR), pp.\ 7992--7999. IEEE, 2021

    2020

  47. [55]

    Trak: attributing model behavior at scale

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: attributing model behavior at scale. In Proceedings of the 40th International Conference on Machine Learning, pp.\ 27074--27113, 2023

    2023

  48. [56]

    Estimating training data influence by tracing gradient descent

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 0 19920--19930, 2020

    2020

  49. [57]

    Scaling up influence functions

    Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 8179--8186, 2022

    2022

  50. [58]

    Fast curvature matrix-vector products for second-order gradient descent

    Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14 0 (7): 0 1723--1738, 2002

    2002

  51. [59]

    A value for n-person games

    Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2 0 (28): 0 307--317, 1953

    1953

  52. [60]

    ingredients

    Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data valuation in machine learning:“ingredients”, strategies, and open challenges. In Proc. IJCAI, 2022

    2022

  53. [61]

    Revisiting methods for finding influential examples

    Anders S gaard et al. Revisiting methods for finding influential examples. arXiv preprint arXiv:2111.04683, 2021

    2021 arXiv

  54. [62]

    Incentivizing collaboration in machine learning via synthetic data rewards

    Sebastian Shenghong Tay, Xinyi Xu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Incentivizing collaboration in machine learning via synthetic data rewards. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 9448--9456, 2022

    2022

  55. [63]

    Private data valuation and fair payment in data marketplaces

    Zhihua Tian, Jian Liu, Jingyu Li, Xinle Cao, Ruoxi Jia, and Kui Ren. Private data valuation and fair payment in data marketplaces. arXiv preprint arXiv:2210.08723, 2022

    2022 arXiv

  56. [64]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  57. [65]

    Data banzhaf: A robust data valuation framework for machine learning

    Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pp.\ 6388--6421. PMLR, 2023 a

    2023

  58. [66]

    A note on" towards efficient data valuation based on the shapley value''

    Jiachen T Wang and Ruoxi Jia. A note on" towards efficient data valuation based on the shapley value''. arXiv preprint arXiv:2302.11431, 2023 b

    2023 arXiv

  59. [67]

    efficient task-specific data valuation for nearest neighbor algorithms

    Jiachen T Wang and Ruoxi Jia. A note on" efficient task-specific data valuation for nearest neighbor algorithms". arXiv preprint arXiv:2304.04258, 2023 c

    2023 arXiv

  60. [68]

    Threshold knn-shapley: A linear-time and privacy-friendly approach to data valuation

    Jiachen T Wang, Yuqing Zhu, Yu-Xiang Wang, Ruoxi Jia, and Prateek Mittal. Threshold knn-shapley: A linear-time and privacy-friendly approach to data valuation. arXiv preprint arXiv:2308.15709, 2023

    2023 arXiv

  61. [69]

    An economic solution to copyright challenges of generative ai

    Jiachen T Wang, Zhun Deng, Hiroaki Chiba-Okabe, Boaz Barak, and Weijie J Su. An economic solution to copyright challenges of generative ai. Technical report, 2024 a

    2024

  62. [70]

    Efficient data shapley for weighted nearest neighbor algorithms

    Jiachen T Wang, Prateek Mittal, and Ruoxi Jia. Efficient data shapley for weighted nearest neighbor algorithms. arXiv preprint arXiv:2401.11103, 2024 b

    2024 arXiv

  63. [71]

    Data shapley in one training run

    Jiachen T Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. Data shapley in one training run. arXiv preprint arXiv:2406.11011, 2024 c

    2024 arXiv

  64. [72]

    The power and limitation of pretraining-finetuning for linear regression under covariate shift

    Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham Kakade. The power and limitation of pretraining-finetuning for linear regression under covariate shift. Advances in Neural Information Processing Systems, 35: 0 33041--33053, 2022 a

    2022

  65. [73]

    How many pretraining tasks are needed for in-context learning of linear regression? In The Twelfth International Conference on Learning Representations, 2024

    Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In The Twelfth International Conference on Learning Representations, 2024

    2024

  66. [74]

    Davinz: Data valuation using deep neural networks at initialization

    Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. Davinz: Data valuation using deep neural networks at initialization. In International Conference on Machine Learning, pp.\ 24150--24176. PMLR, 2022 b

    2022

  67. [75]

    Validation free and replication robust volume-based data valuation

    Xinyi Xu, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Validation free and replication robust volume-based data valuation. Advances in Neural Information Processing Systems, 34: 0 10837--10848, 2021

    2021

  68. [76]

    Gmvaluator: Similarity-based data valuation for generative models

    Jiaxi Yang, Wenglong Deng, Benlin Liu, Yangsibo Huang, James Zou, and Xiaoxiao Li. Gmvaluator: Similarity-based data valuation for generative models. arXiv preprint arXiv:2304.10701, 2023

    2023 arXiv

  69. [77]

    On the inflation of knn-shapley value

    Ziao Yang, Han Yue, Jian Chen, and Hongfu Liu. On the inflation of knn-shapley value. arXiv preprint arXiv:2405.17489, 2024

    2024 arXiv

  70. [78]

    Timeinf: Time series data contribution via influence functions

    Yizi Zhang, Jingyan Shen, Xiaoxue Xiong, and Yongchan Kwon. Timeinf: Time series data contribution via influence functions. arXiv preprint arXiv:2407.15247, 2024

    2024 arXiv

  71. [79]

    The elements of statistical learning, 2003

    Eric R Ziegel. The elements of statistical learning, 2003

    2003

  72. [80]

    Benign overfitting of constant-stepsize sgd for linear regression

    Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Benign overfitting of constant-stepsize sgd for linear regression. In Conference on Learning Theory, pp.\ 4633--4635. PMLR, 2021

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.