REVIEW 4 major objections 5 minor 2 cited by
Training Data Attribution (TDA): Examining Its Adoption & Use Cases
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This report argues that training data attribution can become practical for frontier LLMs through compressed-gradient influence functions, with major benefits for AI research and safety.
desk verdict A useful, honestly hedged scoping report on TDA whose central feasibility claim is conditional on untested accuracy at scale and is internally tensioned against its own statement that influence functions are inapplicable to foundation models. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the compressed-gradient influence function pipeline. Influence functions estimate the effect of removing a training example $z_i$ on a query's loss through the product $g_{\text{query}}^T H^{-1} g_i$; because the Hessian $H$ is too large to form explicitly and computing $g_i$ for every training example costs as much as a pre-training pass, the report relies on three accelerations: approximate inverse-Hessian-vector products, gradient compression by random projection or low-rank structure, and caching compressed gradients for all training sequences on disk with approximate nearest-neighbor search to answer queries in $\mathcal{O}(\log N)$ time. The compressed gradients are the object that converts an $O(NP)$ per-query cost into a one-time pre-training-scale investment plus a fast search.
What would settle it
Run a compressed-gradient influence-function pipeline (say, 16,000-dimensional gradients with approximate nearest-neighbor search) on a frontier-scale model, and compare its top-attributed training examples for a fixed set of queries against ground truth from retraining-based methods on a representative subset, or against the actual change in query loss when the attributed examples are removed; if the agreement is close to chance, or exact nearest-neighbor search on the same compressed gradients finds materially different neighbors, the feasibility claim fails.
Extended reading notes
Core claim
The discovery the report argues for is that the main computational obstacle to TDA at frontier scale is not the Hessian but the per-example gradient computation, and that this obstacle can be pushed to a one-time cost: approximate the inverse-Hessian-vector product once per query, compress each training gradient to a low-dimensional vector (for example, roughly 16,000 dimensions), cache those vectors for all on the order of $10^{11}$ training sequences, and then answer a query with approximate nearest-neighbor search in $\mathcal{O}(\log N)$ time. The report estimates training-cost parity with a single epoch of pretraining, storage of roughly a petabyte, and per-query inference that is slower than standard generation but feasible for leading labs. It is explicit that these optimized techniques have not been run on frontier models and that their accuracy at that scale is unknown; early small-scale results are promising but scalable evaluation methods do not yet exist. The paper's own framing is that TDA is no longer infeasible, not that it is already accurate.
Load-bearing premise
The whole estimate rests on the hope that compressing the training data's fingerprints and searching quickly for similar fingerprints still finds the truly influential examples when applied to a model with roughly 100 billion parameters trained on roughly 100 billion sequences; the report explicitly says this has not been tested.
Editorial extensions
If this is right
- If the feasibility estimate holds, AI labs can run TDA as an internal research tool on frontier LLMs within roughly 2–5 years, with fine-tuned models becoming addressable much sooner.
- Researchers could trace hallucinations, bias, and toxic outputs to specific training examples, improving data filtering and selection and allowing smaller, more efficient models.
- Public access to TDA inference would enable copyright detection, fact-checking, external audits, and accountability, but is blocked by labs' unwillingness to disclose training data, and governments are unlikely to mandate it.
- TDA's clearest contribution to reducing large-scale AI risk would be as a complement to mechanistic interpretability for alignment, while its capability improvements may increase short-to-medium-term risk.
Reading between the lines
- An implication the report leaves implicit is that the cost model assumes a frozen, single-stage training run; under continual learning or multi-stage data mixing the cached compressed gradients would need to be refreshed, otherwise attribution queries describe an outdated model.
- A testable extension would benchmark compressed-gradient influence functions against retraining-based ground truth at the 1–10 billion parameter scale and plot accuracy against compression dimension, giving an empirical curve for whether 16,000 dimensions can plausibly hold at roughly 100 billion parameters.
- If public TDA access arrives, the paper's own incentive analysis suggests it will come as selective, source-metadata-only access from a transparency-oriented lab, so the first public deployments may be more like citation tools than full audit access.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This report analyzes Training Data Attribution (TDA) for frontier large language models from a policy and research perspective. It reviews three TDA families (retraining-based, representation-based, gradient-based), argues that compressed-gradient influence functions combined with approximate nearest-neighbor search are the most plausible route to tractable TDA, and provides back-of-the-envelope estimates suggesting that petabyte-scale storage and sublinear query search are within reach of large AI labs. The report then discusses research benefits, the training-data-disclosure bottleneck, societal benefits of public TDA access, incentives of labs and governments, and an assessment of TDA's overall effect on large-scale AI risk. Its main conclusion is that TDA may become an internal lab tool within 2-5 years, with unclear but potentially significant safety benefits and some capability-acceleration risks.
Significance. If the feasibility estimate held, TDA would constitute a practical research and safety tool for frontier LLMs, enabling data debugging, improved interpretability, and new accountability mechanisms. The report is a useful synthesis for a policy audience: it cleanly separates method families, flags the main unknowns, and gives a balanced account of societal risks and benefits, including the possible net-negative short-term effect via capability acceleration. Its strengths include explicit caveats about the lack of large-scale validation and a sober evaluation of incentives for AI labs. However, the central technical claim rests on the untested assumption that compressed gradients retain attribution accuracy at N≈1e11 and P≈1e11; the report itself states that no such research has been conducted. In addition, an internal contradiction about the applicability of influence functions to foundation models is not resolved. The paper is therefore a valuable but not yet fully supported analysis, and it requires substantive revision before the feasibility claim can be accepted.
major comments (4)
- [Executive Summary and §3 ('How plausible is implementing efficient, accurate TDA on frontier LLMs?')] The central claim that TDA is 'no longer infeasible to run on frontier LLMs' is not fully supported, because 'efficient, accurate TDA' requires the compressed-gradient influence-function approach to retain enough attribution signal at scale. The report admits that 'the accuracy of these optimized TDA techniques on large models is unclear' and that 'no such research has been conducted,' and it notes that optimization 'results in necessary decreases in accuracy.' The arithmetic in §3 establishes only that storing compressed gradients and performing approximate nearest-neighbor search is plausible; it does not establish that the returned training examples are actually the ones that influenced the output. The authors should either reframe the conclusion as a conditional feasibility hypothesis, or provide small- to medium-scale evidence (or a rigorous argument) that top-k ranking quality survives 16k-dimensional compression and N≈1e11 scale.
- [§2, page 8] The report states that implicit-differentiation-based influence functions are 'fundamentally inapplicable to analyzing multi-stage procedures such as continual learning or foundation models (Guu et al., 2023),' yet the rest of the paper, including the central feasibility analysis in §3, proposes influence functions as the path to TDA on frontier pre-trained LLMs. This is a direct internal tension. The authors should explain why pre-training (or a single fine-tuning stage) escapes the stated limitation, or alternatively restrict the feasibility claim to settings where the method is applicable and acknowledge the limitation for multi-stage training.
- [§3] The statement that 'the inference time complexity for calculating TDA on a single query output may be around O(log(N))' omits the per-query cost of computing the query's compressed gradient and the inverse-Hessian-vector product (iHVP). These computations scale at least linearly with the model size P (and with the number of iterations in the iHVP approximation), which for P≈1e11 far exceeds the O(log N) search cost. The claimed complexity therefore understates the actual per-query work and does not provide a meaningful comparison with standard LLM inference. Please present a full query-time decomposition (query gradient + iHVP + ANN search) with explicit scaling in both N and P.
- [§3] The petabyte storage estimate is arithmetically inconsistent. For N=60 billion training sequences and d=16,384 compressed-gradient dimensions, storing one 4-byte float per dimension gives 60e9 × 16,384 × 4 bytes ≈ 3.9 PB, not 'approximately 1 PB.' This factor-of-four discrepancy matters for a report whose central argument is feasibility of storage. Please correct the calculation and state the assumed numeric format (e.g., float32 vs. float16) and any quantization.
minor comments (5)
- [Throughout] The title and several running headers misspell 'ATTRIBUTION' as 'AT TRIBUTION' (e.g., pages 3, 4, 7, 8); this should be corrected in final copy.
- [Reference list] The paper has no bibliography or reference list despite many inline citations. A complete citation list is needed for the report to be usable as a citable document.
- [§4 (Timing)] The estimate of 2-5 years to achieve efficient, accurate TDA is presented without a methodology or supporting evidence; it should be explicitly labeled as the authors' informal judgment, and the sensitivity of the timeline to intermediate research milestones should be acknowledged.
- [§3] The compressed-gradient dimension '~16k' is introduced as 'an example number,' but the feasibility conclusion depends on this value. The paper should at least discuss how the conclusions change for alternative dimensions (e.g., 4k or 64k) or explicitly note that this is a free parameter whose appropriate value is unknown.
- [§5 and §7] The report cites 'discussions with TDA researchers' as evidence that fine-tuned TDA applications are plausible today, but it does not provide names, affiliations, or dates. This non-verifiable evidence claim should either be documented more fully or removed.
Circularity Check
No circularity: the feasibility estimate rests on independent complexity arithmetic, and the report's own accuracy caveat makes the central claim explicitly conditional rather than definitionally forced.
full rationale
The report's central feasibility estimate is not circular. It derives the cost of compressed-gradient TDA from stated quantities: N training sequences, P parameters, a ~16k compressed gradient dimension, petabyte-scale storage for 16k * N vectors, and O(log N) approximate nearest-neighbor search. These are independent complexity facts about the proposed methods, not fitted parameters relabeled as predictions. The 16k dimension is introduced as an illustrative example, and the O(log N) query bound is inherited from ANN search rather than from the conclusion being proved. The paper explicitly states that "the accuracy of these optimized TDA techniques on large models is unclear" and that "no such research has been conducted," so the practical claim is honestly conditional on unvalidated accuracy retention. The self-citations, including Bae et al. 2022 and Bae 2024, appear only as background support for how influence functions approximate counterfactuals and for possible evaluation techniques; they are not load-bearing inputs that force the feasibility conclusion. The internal tension that implicit-differentiation influence functions are "fundamentally inapplicable to analyzing multi-stage procedures such as continual learning or foundation models" is a substantive scaling and correctness concern, not a circularity. No step in the paper reduces by construction to its own inputs, so no significant circularity is present.
Assumptions & free parameters
free parameters (4)
- Compressed gradient dimension (example) =
16,384
- Number of pre-training sequences =
60-100 billion
- Model parameter count =
~100 billion
- Timeline estimate =
2-5 years
assumptions (4)
- domain assumption Influence functions (based on the Implicit Function Theorem) provide a valid approximation of the counterfactual effect of removing training data in non-convex LLM training.
- domain assumption Compressed gradients (random projection or low-rank) and approximate nearest neighbor search preserve sufficient attribution accuracy at frontier scale.
- domain assumption AI labs' incentives and government behavior are as the authors describe.
- domain assumption The cost of computing per-sample gradients is roughly equivalent to pre-training cost.
Cite this review
Pith. "Pith review of Training Data Attribution (TDA): Examining Its Adoption & Use Cases." pith.science (2026). https://pith.science/paper/NQ6P6DAI
@misc{pith2026250112642,
author = {Pith},
title = {Pith review of: Training Data Attribution (TDA): Examining Its Adoption & Use Cases},
year = {2026},
howpublished = {\url{https://pith.science/paper/NQ6P6DAI}},
note = {Machine review of arXiv:2501.12642}
}
read the original abstract
This report investigates Training Data Attribution (TDA) and its potential importance to and tractability for reducing extreme risks from AI. First, we discuss the plausibility and amount of effort it would take to bring existing TDA research efforts from their current state, to an efficient and accurate tool for TDA inference that can be run on frontier-scale LLMs. Next, we discuss the numerous research benefits AI labs will expect to see from using such TDA tooling. Then, we discuss a key outstanding bottleneck that would limit such TDA tooling from being accessible publicly: AI labs' willingness to disclose their training data. We suggest ways AI labs may work around these limitations, and discuss the willingness of governments to mandate such access. Assuming that AI labs willingly provide access to TDA inference, we then discuss what high-level societal benefits you might see. We list and discuss a series of policies and systems that may be enabled by TDA. Finally, we present an evaluation of TDA's potential impact on mitigating large-scale risks from AI systems.
Forward citations
Cited by 2 Pith papers
-
Influence Dynamics and Stagewise Data Attribution
Using Bayesian influence functions and singular learning theory, the authors show that a sample's influence on a model varies non-monotonically over training, peaking and flipping sign at phase transitions.
-
Low-Perplexity LLM-Generated Sequences and Where To Find Them
Only about 40% of low-perplexity 6-token spans generated by Pythia-6.9B can be exactly matched to The Pile, and the authors categorize matched and unmatched spans into four classes.
Reference graph
Works this paper leans on
-
[1]
Competitive advantages: The training datasets used to generate frontier LLMs is a key competitive advantage, as it constitutes part of their “secret sauce”. It is well established that good training data curation substantially improves model quality (Zhou et al, 2023)
work page 2023
-
[2]
Proprietary data: In many cases, the training data includes proprietary or licensed information that the labs do not have the rights to distribute publicly. This could include data from books, websites, or other sources that are subject to copyright or terms of service restrictions
-
[3]
Such data cannot be shared publicly for legal and ethical reasons
Privacy: Training data may include sensitive personal data that could be used to identify individuals, such as logs from ChatGPT users. Such data cannot be shared publicly for legal and ethical reasons. TRAINING DATA AT TRIBUTION (TDA): EXAMINING ITS ADOPTION & USE CASES 16 ASSUME THAT TDA CAN BE CONDUCTED EFFICIENTLY ON LARGE MODELS WITH REASONABLY HIGH ...
-
[4]
Legal liabilities: Releasing training datasets increases exposure to legal liabilities from data collection and usage. AI labs are actively defending their right to use publicly available copyrighted works in their training data. Public access to data sources would exponentially increase the number of opt-outs and legal challenges. Without the incentives ...
work page 2022
-
[6]
Potentially improve the quality of the LLM response by surfacing and integrating additional information missed by the first inference step
-
[7]
Identify if the LLM response is obviously wrong or misaligned with factual data. Or, it could identify if the primary source of the response is unreliable (e.g. Reddit) and the results should be discounted. ▪ The results from this TDA analysis would be combined with the original LLM response to produce a potentially better LLM response, and served to the ...
-
[8]
Accuracy & Precision: TDA measures the influence of training examples on the model's outputs, but this influence can be complex and indirect. A model's output may be influenced by multiple training examples in combination, rather than a single direct source. a. As a result, it may not consistently return relevant sources, or may distribute influence acros...
-
[9]
Speed: TDA would likely be slower than RAG, as performing nearest- neighbor search would take longer than searching an optimized vector database
Show all 11 references
-
[10]
If the relevant information is just a small part of a larger training example, TDA may not provide the necessary granularity to identify the specific source
Granularity: TDA typically attributes influence to entire training examples or passages, rather than pinpointing the exact sentence or phrase that contains the desired information. If the relevant information is just a small part of a larger training example, TDA may not provi...
-
[11]
deceptiveness
Maintainability: RAG is easier to keep up to date, as it is much easier to update a vector database rather than continually retraining a new model. Most importantly, TDA cannot be directly compared to RAG as they have fundamentally different methods of retrieving data. The res...
-
[2017]
why the LLM model does what it does
with these compressed gradients. Park et al. 2023 use random projection to compress the gradient, while Kwon et al. 2023 impose a low-rank structure to the gradient. As the gradients are now represented in a lower dimensional space, these approaches can also reduce the cost of...
2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.