REVIEW 5 cited by
Training Data Attribution via Approximate Unrolled Differentiation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Many training data attribution (TDA) methods aim to estimate how a model's behavior would change if one or more data points were removed from the training set. Methods based on implicit differentiation, such as influence functions, can be made computationally efficient, but fail to account for underspecification, the implicit bias of the optimization algorithm, or multi-stage training pipelines. By contrast, methods based on unrolling address these issues but face scalability challenges. In this work, we connect the implicit-differentiation-based and unrolling-based approaches and combine their benefits by introducing Source, an approximate unrolling-based TDA method that is computed using an influence-function-like formula. While being computationally efficient compared to unrolling-based approaches, Source is suitable in cases where implicit-differentiation-based approaches struggle, such as in non-converged models and multi-stage training pipelines. Empirically, Source outperforms existing TDA techniques in counterfactual prediction, especially in settings where implicit-differentiation-based approaches fall short.
Forward citations
Cited by 5 Pith papers
-
Better Training Data Attribution via Better Inverse Hessian-Vector Products
ASTRA, an EKFAC-preconditioned Neumann series iteration, computes more accurate inverse Hessian-vector products and improves training data attribution scores over EKFAC baselines.
-
Daunce: Data Attribution through Uncertainty Estimation
DAUNCE computes training-data attribution as the covariance of per-example losses across an ensemble of perturbed fine-tuned models, reporting state-of-the-art LDS scores and the first attribution runs on proprietary LLMs.
-
MAGIC: Near-Optimal Data Attribution for Deep Learning
MAGIC computes the exact influence function for smooth, deterministic deep learning training runs and achieves near-perfect linear datamodeling scores on CIFAR-10, GPT-2, and Gemma-2B, far outperforming TRAK and EK-FAC.
-
Capturing the Temporal Dependence of Training Data Influence
A new influence-estimation method approximates order-aware data influence with a dot product and reveals that early and late training data matter most.
-
What Is The Performance Ceiling of My Classifier? Utilizing Category-Wise Influence Functions for Pareto Frontier Analysis
Category-wise influence vectors plus linear programming and a genetic algorithm reweight training data to improve all classes at once, with an unproven criterion for when a classifier has reached its Pareto ceiling.
Discussion (0). Continue with ORCID to comment.