{"id":"270aee21-740d-4357-9135-ed1c03d7b2d6","arxiv_id":"2501.12642","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training data attribution is plausibly scalable to frontier LLMs using compressed-gradient influence functions, but public access will likely remain limited because AI labs have strong incentives to keep training data private.","lead":"This report assesses whether training data attribution (TDA) can be made practical for frontier AI models and what benefits and risks would follow. It argues that TDA is plausibly feasible with enterprise-level compute and storage, but that public access is blocked by AI labs' secrecy about their training data.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frontier-scale TDA feasibility hinges on untested accuracy retention under 16k-dim gradient compression; the report's own caveat makes the central claim conditional.","rationale":"The reader's weakest assumption is exactly the accuracy of compressed-gradient influence functions and ANN search at frontier scale, and the report's own text confirms that no such validation exists. I agree this is the most load-bearing concern: all of the report's downstream benefits—research tooling, audits, royalties, risk mitigation—presuppose that the attribution output is useful, not merely that the computation is affordable. A petabyte-scale pipeline that returns near-random training examples would not make TDA 'feasible' in any meaningful sense. The report deserves credit for flagging this limitation prominently and for hedging the main claim with 'appears' and 'unclear.' However, the hedge is exactly why the central claim cannot be accepted as established; it is a conditional feasibility hypothesis. The concern is not an ad hominem or a disagreement with consensus; it is an internal scaling gap. I considered whether the more pointed issue is the statement that influence functions are 'fundamentally inapplicable' to foundation models. That is a serious internal tension, but the report also cites empirical work connecting influence functions to counterfactuals on neural networks, so the better framing is that the theoretical basis is shaky and the empirical scaling evidence is absent. The random-projection variance argument makes the accuracy concern concrete rather than a general plea for more testing. The proposed test—measuring rank correlation across compression width and database size at an intermediate scale—directly settles whether the concern lands, because it gives a quantitative extrapolation target for the frontier regime. Since the reader already issued a conditional verdict and the concern is the same one, the correct recommendation is to leave the verdict unchanged.","tokens_in":14127,"tokens_out":8195,"duration_ms":89067,"concrete_test":"Pick an intermediate-scale model (≈1B parameters, ≈1B training sequences) with a reference attribution computed by retraining-based methods on a downsampled subset or by un-compressed influence functions. Compute rank correlation and top-100 recall for compressed-gradient influence functions with ANN search, sweeping compression dimensions (64k, 16k, 8k, 4k) and database size from 1e8 to 1e9 sequences. Extrapolate whether top-k recall remains above a usable threshold at N=1e11; if recall decays monotonically, the frontier-scale feasibility claim collapses. If feasible, repeat at 10B parameters to separate parameter-scale from data-scale effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility estimate ('How plausible is implementing efficient, accurate TDA...') assumes that compressing each training gradient to ~16k dimensions and using approximate nearest-neighbor search preserves enough attribution signal at N≈1e11 sequences and P≈1e11 parameters. This is the load-bearing link: without it, the O(log N) query time and petabyte storage are irrelevant because the returned examples are not the ones that actually influenced the output. The report itself states that 'the accuracy of these optimized TDA techniques on large models is unclear' and that 'no such research has been conducted.' There is also a mechanistic reason for concern: random-projection influence scores estimate g_query^T H^{-1} g_i from 16k random coordinates. If true influence scores are sparse and typical cosine similarities between the iHVP vector and training gradients are small, the variance of the projected estimate can swamp the signal, so the top-k ranking may be near-random at frontier scale. Early small-model results (Park et al. 2023; Keun Choe et al. 2024) do not establish the scaling behavior. Additionally, the text states that implicit-differentiation influence functions are 'fundamentally inapplicable to analyzing multi-stage procedures such as continual learning or foundation models' (page 8), which creates internal tension with proposing the same method for pretraining-scale LLMs. Thus the practical claim—TDA as an efficient, accurate tool—is not supported until scaling of attribution accuracy is demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This report analyzes Training Data Attribution (TDA) for frontier large language models from a policy and research perspective. It reviews three TDA families (retraining-based, representation-based, gradient-based), argues that compressed-gradient influence functions combined with approximate nearest-neighbor search are the most plausible route to tractable TDA, and provides back-of-the-envelope estimates suggesting that petabyte-scale storage and sublinear query search are within reach of large AI labs. The report then discusses research benefits, the training-data-disclosure bottleneck, societal benefits of public TDA access, incentives of labs and governments, and an assessment of TDA's overall effect on large-scale AI risk. Its main conclusion is that TDA may become an internal lab tool within 2-5 years, with unclear but potentially significant safety benefits and some capability-acceleration risks.","tokens_in":14396,"tokens_out":5769,"duration_ms":58101,"significance":"If the feasibility estimate held, TDA would constitute a practical research and safety tool for frontier LLMs, enabling data debugging, improved interpretability, and new accountability mechanisms. The report is a useful synthesis for a policy audience: it cleanly separates method families, flags the main unknowns, and gives a balanced account of societal risks and benefits, including the possible net-negative short-term effect via capability acceleration. Its strengths include explicit caveats about the lack of large-scale validation and a sober evaluation of incentives for AI labs. However, the central technical claim rests on the untested assumption that compressed gradients retain attribution accuracy at N≈1e11 and P≈1e11; the report itself states that no such research has been conducted. In addition, an internal contradiction about the applicability of influence functions to foundation models is not resolved. The paper is therefore a valuable but not yet fully supported analysis, and it requires substantive revision before the feasibility claim can be accepted.","major_comments":[{"comment":"The central claim that TDA is 'no longer infeasible to run on frontier LLMs' is not fully supported, because 'efficient, accurate TDA' requires the compressed-gradient influence-function approach to retain enough attribution signal at scale. The report admits that 'the accuracy of these optimized TDA techniques on large models is unclear' and that 'no such research has been conducted,' and it notes that optimization 'results in necessary decreases in accuracy.' The arithmetic in §3 establishes only that storing compressed gradients and performing approximate nearest-neighbor search is plausible; it does not establish that the returned training examples are actually the ones that influenced the output. The authors should either reframe the conclusion as a conditional feasibility hypothesis, or provide small- to medium-scale evidence (or a rigorous argument) that top-k ranking quality survives 16k-dimensional compression and N≈1e11 scale.","section":"Executive Summary and §3 ('How plausible is implementing efficient, accurate TDA on frontier LLMs?')"},{"comment":"The report states that implicit-differentiation-based influence functions are 'fundamentally inapplicable to analyzing multi-stage procedures such as continual learning or foundation models (Guu et al., 2023),' yet the rest of the paper, including the central feasibility analysis in §3, proposes influence functions as the path to TDA on frontier pre-trained LLMs. This is a direct internal tension. The authors should explain why pre-training (or a single fine-tuning stage) escapes the stated limitation, or alternatively restrict the feasibility claim to settings where the method is applicable and acknowledge the limitation for multi-stage training.","section":"§2, page 8"},{"comment":"The statement that 'the inference time complexity for calculating TDA on a single query output may be around O(log(N))' omits the per-query cost of computing the query's compressed gradient and the inverse-Hessian-vector product (iHVP). These computations scale at least linearly with the model size P (and with the number of iterations in the iHVP approximation), which for P≈1e11 far exceeds the O(log N) search cost. The claimed complexity therefore understates the actual per-query work and does not provide a meaningful comparison with standard LLM inference. Please present a full query-time decomposition (query gradient + iHVP + ANN search) with explicit scaling in both N and P.","section":"§3"},{"comment":"The petabyte storage estimate is arithmetically inconsistent. For N=60 billion training sequences and d=16,384 compressed-gradient dimensions, storing one 4-byte float per dimension gives 60e9 × 16,384 × 4 bytes ≈ 3.9 PB, not 'approximately 1 PB.' This factor-of-four discrepancy matters for a report whose central argument is feasibility of storage. Please correct the calculation and state the assumed numeric format (e.g., float32 vs. float16) and any quantization.","section":"§3"}],"minor_comments":[{"comment":"The title and several running headers misspell 'ATTRIBUTION' as 'AT TRIBUTION' (e.g., pages 3, 4, 7, 8); this should be corrected in final copy.","section":"Throughout"},{"comment":"The paper has no bibliography or reference list despite many inline citations. A complete citation list is needed for the report to be usable as a citable document.","section":"Reference list"},{"comment":"The estimate of 2-5 years to achieve efficient, accurate TDA is presented without a methodology or supporting evidence; it should be explicitly labeled as the authors' informal judgment, and the sensitivity of the timeline to intermediate research milestones should be acknowledged.","section":"§4 (Timing)"},{"comment":"The compressed-gradient dimension '~16k' is introduced as 'an example number,' but the feasibility conclusion depends on this value. The paper should at least discuss how the conclusions change for alternative dimensions (e.g., 4k or 64k) or explicitly note that this is a free parameter whose appropriate value is unknown.","section":"§3"},{"comment":"The report cites 'discussions with TDA researchers' as evidence that fine-tuned TDA applications are plausible today, but it does not provide names, affiliations, or dates. This non-verifiable evidence claim should either be documented more fully or removed.","section":"§5 and §7"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is a policy-style analysis rather than a primary technical contribution, and the editor should weigh whether the journal's scope accommodates speculative feasibility reports. The main technical gap is the untested accuracy retention of compressed influence functions at frontier scale; this is explicitly admitted by the authors, which is commendable, but it still leaves the central claim conditional. The internal contradiction regarding influence functions' applicability to foundation models and the arithmetic error in the storage estimate are fixable. With a clearer framing of what is established versus conjectured, and with the internal tension resolved, the paper could be a useful and honest assessment. I would not recommend rejection on this basis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a competent, clearly written scoping report on training data attribution (TDA), not a new technical result. It does a genuinely good job of categorizing current methods (retraining, representation, gradient-based) and explaining why compressed-gradient influence functions look like the most plausible path to practical TDA. The authors are also refreshingly honest: they repeatedly flag that optimized TDA has not been tested on frontier-scale LLMs, that accuracy under compression is unclear, and that their timeline estimates are fuzzy. That transparency is the report's main strength.\n\nThe real soft spot, as you might expect, is the load-bearing feasibility claim. The arithmetic showing O(log N) query time and single-digit-petabyte storage is simple and explicit, but it only matters if the 16k-dimensional compressed gradients retain enough attribution signal at N≈1e11 sequences. The report admits no such research exists, so the claim that TDA is \"no longer infeasible\" is really a hypothesis, not a demonstrated result. The stress-test note is right to flag this. What I find more troubling, though, is an internal tension the report never resolves: on page 8 it says implicit-differentiation influence functions are \"fundamentally inapplicable to analyzing multi-stage procedures such as continual learning or foundation models,\" yet the whole frontier-scale feasibility section proposes using exactly those methods for pretraining data. Maybe the authors mean only that influence functions can't attribute to early versus late training stages, but as written it undercuts their central premise. That needs a footnote, if not a rewrite.\n\nThe rest of the report—use cases, incentives, policy implications—is speculative but appropriately labeled. The discussion of fine-tuning as a near-term target for TDA is sensible, and the comparison with RAG for content attribution is fair. The citation pattern looks fine; citing Bae et al. for influence-function theory is legitimate background, not circularity.\n\nFor a serious referee, I'd send it out. It's a useful synthesis that could guide research investment, and the limitations are mostly acknowledged. But I'd ask the authors to fix the foundation-model contradiction and to explicitly frame the frontier-scale feasibility as a conditional hypothesis pending accuracy scaling experiments. That would turn a decent scoping report into a genuinely useful one.","headline":"A useful, honestly hedged scoping report on TDA whose central feasibility claim is conditional on untested accuracy at scale and is internally tensioned against its own statement that influence functions are inapplicable to foundation models.","tokens_in":14906,"tokens_out":2668,"would_cite":false,"duration_ms":27803,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This report argues that training data attribution can become practical for frontier LLMs through compressed-gradient influence functions, with major benefits for AI research and safety.","keywords":["training data attribution","influence functions","large language models","AI interpretability","AI safety","gradient compression","approximate nearest neighbor","data provenance"],"falsifier":"Run a compressed-gradient influence-function pipeline (say, 16,000-dimensional gradients with approximate nearest-neighbor search) on a frontier-scale model, and compare its top-attributed training examples for a fixed set of queries against ground truth from retraining-based methods on a representative subset, or against the actual change in query loss when the attributed examples are removed; if the agreement is close to chance, or exact nearest-neighbor search on the same compressed gradients finds materially different neighbors, the feasibility claim fails.","tokens_in":13931,"feed_emoji":"🔍","tokens_out":9650,"duration_ms":86779,"temperature":0.7,"pith_summary":"This report asks whether training data attribution (TDA)—identifying which training examples most influenced a given model output—can be made efficient and accurate enough to run on frontier LLMs, and whether doing so would reduce extreme AI risks. Its central claim is that with compressed-gradient influence functions and approximate nearest-neighbor search, TDA is no longer infeasible at enterprise compute and storage levels: roughly one pre-training epoch's compute to build the gradient index, single-digit petabytes of storage, and $\\mathcal{O}(\\log N)$ query time. From there the report catalogs internal research benefits, the training-data-disclosure bottleneck that blocks public access, policies TDA could enable, and an assessment that TDA's largest risk-reduction payoff would come through technical AI safety and interpretability rather than societal or governance effects. A sympathetic reader should care because the report connects a concrete scaling estimate to a concrete timeline (2–5 years) and to the choice facing AI labs between private use and public TDA access.","feed_headline":"Compressed gradients put training-data attribution within reach","feed_subtitle":"With ~1 PB of storage and O(log N) search, AI labs could trace LLM outputs to training data.","key_machinery":"The central mechanism is the compressed-gradient influence function pipeline. Influence functions estimate the effect of removing a training example $z_i$ on a query's loss through the product $g_{\\text{query}}^T H^{-1} g_i$; because the Hessian $H$ is too large to form explicitly and computing $g_i$ for every training example costs as much as a pre-training pass, the report relies on three accelerations: approximate inverse-Hessian-vector products, gradient compression by random projection or low-rank structure, and caching compressed gradients for all training sequences on disk with approximate nearest-neighbor search to answer queries in $\\mathcal{O}(\\log N)$ time. The compressed gradients are the object that converts an $O(NP)$ per-query cost into a one-time pre-training-scale investment plus a fast search.","core_discovery":"The discovery the report argues for is that the main computational obstacle to TDA at frontier scale is not the Hessian but the per-example gradient computation, and that this obstacle can be pushed to a one-time cost: approximate the inverse-Hessian-vector product once per query, compress each training gradient to a low-dimensional vector (for example, roughly 16,000 dimensions), cache those vectors for all on the order of $10^{11}$ training sequences, and then answer a query with approximate nearest-neighbor search in $\\mathcal{O}(\\log N)$ time. The report estimates training-cost parity with a single epoch of pretraining, storage of roughly a petabyte, and per-query inference that is slower than standard generation but feasible for leading labs. It is explicit that these optimized techniques have not been run on frontier models and that their accuracy at that scale is unknown; early small-scale results are promising but scalable evaluation methods do not yet exist. The paper's own framing is that TDA is no longer infeasible, not that it is already accurate.","pith_inferences":["An implication the report leaves implicit is that the cost model assumes a frozen, single-stage training run; under continual learning or multi-stage data mixing the cached compressed gradients would need to be refreshed, otherwise attribution queries describe an outdated model.","A testable extension would benchmark compressed-gradient influence functions against retraining-based ground truth at the 1–10 billion parameter scale and plot accuracy against compression dimension, giving an empirical curve for whether 16,000 dimensions can plausibly hold at roughly 100 billion parameters.","If public TDA access arrives, the paper's own incentive analysis suggests it will come as selective, source-metadata-only access from a transparency-oriented lab, so the first public deployments may be more like citation tools than full audit access."],"forward_implications":["If the feasibility estimate holds, AI labs can run TDA as an internal research tool on frontier LLMs within roughly 2–5 years, with fine-tuned models becoming addressable much sooner.","Researchers could trace hallucinations, bias, and toxic outputs to specific training examples, improving data filtering and selection and allowing smaller, more efficient models.","Public access to TDA inference would enable copyright detection, fact-checking, external audits, and accountability, but is blocked by labs' unwillingness to disclose training data, and governments are unlikely to mandate it.","TDA's clearest contribution to reducing large-scale AI risk would be as a complement to mechanistic interpretability for alignment, while its capability improvements may increase short-to-medium-term risk."],"supporting_citations":[{"why":"Supplies the efficient Hessian-approximation method that makes inverse-Hessian-vector products feasible for LLMs and anchors the report's scaling estimate.","marker":"Grosse et al. (2023)"},{"why":"Shows random-projection compressed gradients can approximate influence functions with counterfactual evaluations, and is cited for both the compression approach and a proxy evaluation technique.","marker":"Park et al. 2023"},{"why":"Provides the low-rank gradient structure variant of compressed-gradient TDA used to reduce storage and inverse-Hessian-vector-product cost.","marker":"Kwon et al. 2023"},{"why":"The approximate nearest-neighbor search approach underlying the O(log N) query-time estimate for matching a query gradient against cached training gradients.","marker":"Johnson et al., 2017"},{"why":"Early small-scale evidence that optimized TDA maintains competitive accuracy, which the report uses to justify the feasibility claim.","marker":"Keun Choe et. al, 2024"},{"why":"Datamodels formulate the retraining-based approach and supply the linear data-modeling score used as an evaluation technique for TDA accuracy.","marker":"Ilyas et. al, 2022"},{"why":"Theoretically characterizes what implicit-differentiation influence functions approximate in neural networks, grounding the method's connection to the counterfactual.","marker":"Bae et al., 2022"},{"why":"Introduces influence functions for training-data attribution, the foundational method whose Hessian-inverse computation the report analyzes.","marker":"Koh & Liang, 2017"}],"fun_headline_variants":["Compressed gradients bring training-data attribution to frontier scale","One-time gradient compression unlocks TDA for AI labs","Petabyte gradient cache makes TDA inference feasible","TDA becomes feasible with compressed gradients and fast search","From infeasible to feasible: compressed gradients enable TDA at scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole estimate rests on the hope that compressing the training data's fingerprints and searching quickly for similar fingerprints still finds the truly influential examples when applied to a model with roughly 100 billion parameters trained on roughly 100 billion sequences; the report explicitly says this has not been tested.","fun_headline_variants_meta":{"raw":{"variants":["Compressed gradients bring training-data attribution to frontier scale","One-time gradient compression unlocks TDA for AI labs","Petabyte gradient cache makes TDA inference feasible","TDA becomes feasible with compressed gradients and fast search","From infeasible to feasible: compressed gradients enable TDA at scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001152,"raw_usage":{"total_tokens":4769,"prompt_tokens":934,"completion_tokens":3835,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":3756}},"tokens_in":550,"tokens_out":3835,"duration_ms":28231,"temperature":1.0,"reasoning_tokens":3756,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:57:14.528921+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a compressed-gradient influence-function pipeline (say, 16,000-dimensional gradients with approximate nearest-neighbor search) on a frontier-scale model, and compare its top-attributed training examples for a fixed set of queries against ground truth from retraining-based methods on a representative subset, or against the actual change in query loss when the attributed examples are removed; if the agreement is close to chance, or exact nearest-neighbor search on the same compressed gradients finds materially different neighbors, the feasibility claim fails.","supporting_citations":[],"review_version":1}