Pith. sign in

REVIEW 2 major objections 3 minor 21 references

A Versatile Influence Function for Data Attribution with Non-Decomposable Loss

T0 review · 2 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that the classic influence function can be extended to any loss that depends jointly on multiple training points, by replacing the per-point weight perturbation with a finite difference between the full dataset and the…

desk verdict A clever and useful method, but the paper's central claim that VIF approximates the general influence function is not actually established by the derivation. read the letter →

arxiv 2412.01335 v1 pith:22ZFAPGX submitted 2024-12-02 cs.LG stat.ML

Junwei Deng , Weijing Tang , Jiaqi W. Ma This is my paper · ORCID
classification cs.LGstat.ML
keywords influencefunctiondataattributionnon-decomposablelossleave-one-outretrainingCoxregressioncontrastivelearning-to-rankauto-differentiation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that data attribution by influence functions—quantifying how removing one training object changes a model—can be extended from the usual per-point loss sums (M-estimators) to any loss that depends jointly on many training objects. The proposed Versatile Influence Function (VIF) approximates the leave-one-out parameter change using only quantities at the original optimum: the inverse Hessian of the total loss and the gradient of the difference between the full loss and the loss with object $i$ removed. For decomposable losses VIF reduces exactly to the classical influence function, and for Cox regression its deviation from the exact influence function is $O_p(1/n)$. If the paper is right, influence-based attribution becomes available for contrastive embeddings, ranking losses, and survival models without case-by-case derivations, computed purely through auto-differentiation.

What carries the argument

The central object is the finite-difference influence function $\widehat{\mathrm{IF}}_\varepsilon(\tilde{\theta}(P);Q)$, which replaces the limit in the Gateaux derivative of the minimum of the loss with a finite difference at $\varepsilon$; choosing $P$ as the empirical distribution on the full data and $Q$ as the empirical distribution with object $i$ removed converts that derivative into a comparison of $L(\theta,1)$ and $L(\theta,1_{-i})$. The identity that carries the argument is the linearized optimality condition $$\nabla_\$\theta$ \tilde{L}(\tilde{\$\theta$}(P),P)=0$$ together with the finite-difference form, which yields the inverse-Hessian-times-gradient formula. Because the formula only requires gradients and Hessians of the total loss at the original optimum, it can be implemented with auto-differentiation instead of per-loss analytic derivations.

What would settle it

Compute VIF and the exact influence function on a Cox regression at several finite sample sizes $n$ and check that the difference actually scales as $1/n$; if the difference does not shrink (or fails to match brute-force leave-one-out in absolute value, not just rank correlation), the central approximation claim is falsified. The same test can be run on any convex non-decomposable loss whose exact influence function is known.

Watch

Extended reading notes

Core claim

The paper's central claim is that the leave-one-out effect $\hat{\theta}(1_{-i})-\hat{\theta}(1)$ for any non-decomposable loss can be approximated by $$\mathrm{VIF}(\hat{\$\theta$}(1);i) := -\left[\frac{1}{n}\$nabla^{2}$_\$\theta$ L(\hat{\$\theta$}(1),1)\right]^{-1}\nabla_\$\theta$\left[L(\hat{\$\theta$}(1),1)-L(\hat{\$\theta$}(1),1_{-i})\right],$$ where $L(\theta,b)$ is the loss as a function of the presence vector $b$. The argument starts from the robust-statistics influence function as a functional derivative, replaces the limit by a finite difference between the full empirical distribution and the distribution with object $i$ removed, and then writes that difference directly on the non-decomposable loss's presence vector. The paper shows that under the M-estimator specification the finite-difference influence function coincides exactly with the classical influence function (up to a constant), and that under Cox regression the VIF differs from the exact influence function of Reid and Crepeau by $O_p(1/n)$. Empirically, VIF matches brute-force leave-one-out retraining correlations on survival analysis, node embedding, and listwise learning-to-rank while running orders of magnitude faster.

Load-bearing premise

The load-bearing premise is that the loss has a unique minimizer with an invertible Hessian for every subset of training objects, so the leave-one-out parameter $\hat{\theta}(1_{-i})$ is well defined and the first-order linearization in the finite-difference step is valid; the paper states this explicitly and falls back on heuristics for non-convex neural networks.

Editorial extensions

If this is right

  • Influence-function attribution becomes applicable to contrastive losses, ranking losses, and survival partial likelihoods without new derivations.
  • For decomposable losses, VIF coincides with the classical influence function, so existing methods are recovered as a special case.
  • VIF computation costs no model retraining; experiments report speed-ups of up to about $10^3\times$ over brute-force leave-one-out retraining.
  • The same inverse-Hessian acceleration tricks (conjugate gradient, LiSSA) carry over, letting VIF scale to larger models.
  • Cox regression's approximation error of $O_p(1/n)$ suggests accuracy improves as the training set grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • VIF's finite-difference view suggests a natural test: for any convex non-decomposable loss where leave-one-out can be computed exactly on small problems, the absolute (not just rank) error of VIF should shrink with $n$; verifying this on new losses would show the Cox guarantee is not special.
  • The same construction could apply to discrete or combinatorial objects beyond data points, such as removing labels, features, or edges in a graph, as long as the loss can be evaluated with the object's presence toggled.
  • For non-convex models the paper uses heuristics with no guarantee; a plausible extension is to check VIF against trajectory-based attribution on small neural networks and see where the convex approximation breaks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper proposes the Versatile Influence Function (VIF), defined in Eq. (10) as a Hessian-inverse times the parameter gradient of the difference between the full loss and the leave-one-out loss evaluated at the trained parameters. The intended contribution is to extend influence-function-based data attribution from decomposable M-estimator losses to general non-decomposable losses such as Cox partial likelihood, contrastive losses, and listwise ranking losses. The paper proves that VIF reduces exactly to the classical influence function for M-estimators, proves an O_p(1/n) approximation error relative to the Reid--Crepeau influence function for Cox regression (Theorem 3.2 / Theorem A.1), and reports experiments on survival analysis, node embedding, and learning-to-rank showing high correlations with brute-force leave-one-out retraining and large speedups.

Significance. If the claimed generality held, VIF would be a valuable practical tool: it would let practitioners compute influence-style attributions for losses that are not sums of per-point terms, using only automatic differentiation. The M-estimator exactness result and the Cox regression rate are real mathematical contributions, and the empirical protocol is strengthened by reporting a brute-force retraining noise ceiling rather than only comparing against one retraining run. The Cox proof in Appendix A.5 is nontrivial and appears carefully executed. However, the central derivation from the robust-statistics influence function to VIF has a domain gap for generic non-decomposable losses, so the broader ``arbitrary non-decomposable loss'' claim is currently supported only heuristically and by case-specific experiments.

major comments (2)
  1. [§3.3--3.4, Definition 3.3, Eq. (8), Proposition 3.1] The derivation of VIF from the influence-function formulation in Eq. (7) is not valid for generic non-decomposable losses. Eq. (7) is a Gateaux derivative that requires evaluating L~(θ, (1−ε)P_n + εQ) for ε in a neighborhood of 0. However, Definition 3.3 defines L~(θ, ·) only on P(n), the set of uniform distributions supported on subsets of the n training points. For any ε ∈ (0,1), the mixture (1−ε)P_n + εδ_z_i is not uniform on a subset, so L~ is undefined and the limit in Eq. (7) does not exist. The paper instead uses the endpoint ε = 1, where the perturbed measure Q(−i) does lie in P(n); but a one-point secant between P_n and Q(−i) is a finite-difference approximation of a derivative only if the relevant function is defined and sufficiently regular on the whole interval. For a general non-decomposable loss, L(θ,b) is not linear in b, so no such regularity is available. Proposition 3.1 establishes an algebraic identity for the endpoint, not a connection to Eq. (7). This gap is independent of convexity: even for strictly convex non-decomposable losses, the link from Eq. (7) to Eq. (10) is missing. The paper should either define a continuous extension of L(θ,b) to mixtures and prove an approximation bound, or explicitly reframe VIF as a direct leave-one-out approximation heuristic and revise the claims that it ``serves as an approximation of the general formulation of IF''.
  2. [§3.4--3.5, Theorem 3.2, Table 1] The theoretical support for the general non-decomposable claim is limited to two cases: exact recovery for M-estimators and an O_p(1/n) bound for Cox regression. Theorem 3.2 is specifically about Cox partial likelihood and does not extend to the other two showcased settings, contrastive node embedding and listwise ranking. For those settings, the paper provides empirical correlations against brute-force leave-one-out retraining (Table 1), but leave-one-out retraining is not the influence function of Eq. (7), so these experiments do not fill the derivation gap. The abstract and introduction state that VIF ``can be straightforwardly applied to machine learning models trained with any non-decomposable loss''; as written, that generality is not established. At minimum, the paper should clearly state which losses are covered by formal guarantees and which are supported only by empirical evidence.
minor comments (3)
  1. [Throughout] There are several typos and grammatical slips, including ``mathetical'' (Section 1), ``Tabel 2'' (Section 4.3), ``Senario'' (Table 2), ``Actural'' (Figure 1), and ``close-form'' (Section 3.5). These should be corrected in a revision.
  2. [Appendix D, Figure 2] The heatmap appendix already notes a color-scale misalignment between VIF and LOO values. It would be helpful to state explicitly in the main text that VIF is designed for ranking/correlation agreement rather than for exact absolute-value agreement, since the Pearson correlations in Table 1 measure the former.
  3. [Appendix C, Table 5] For the neural-network Cox model, VIF correlations are around 0.2 for all methods, including the brute-force baseline. The paper should clarify whether this is a noise ceiling of the retraining procedure and, if so, report the correlation between two brute-force runs for the neural network model as was done in Table 1; otherwise readers may infer that the method degrades sharply on non-convex models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VIF is an explicit first-order finite-difference approximation of the classical influence function, benchmarked against independent LOO retraining and the Reid–Crepeau analytical Cox IF.

full rationale

The paper's derivation chain is not circular. VIF is defined in Eq. (10) as -n times a finite-difference gradient expression, and the target leave-one-out parameter change θ̂(1−i) never appears in the definition; only the gradient of L(θ,1) − L(θ,1−i) at the full-data optimum is used. For M-estimators, Theorem 3.1 and Lemma 3.1 provide an algebraic proof that this expression coincides with the classical influence function, not an assumption of that equality. For Cox regression, Theorem 3.2 bounds the difference between VIF and the analytical influence function of Reid & Crepeau (1985), an external, independently derived formula. The empirical evaluation compares VIF to brute-force leave-one-out retraining, which is an external ground truth not used to fit any constant in VIF. The paper also explicitly acknowledges the convexity assumption and the lack of a universal approximation guarantee for non-convex losses, which is an honest limitation rather than a hidden assumption of the target result. The self-citations to Koh & Liang (2017) and Grosse et al. (2023) are used for heuristic acceleration tricks and for framing, not as load-bearing evidence for the main derivation. The skeptical concern that Definition 3.3's L̃ is only defined on uniform subset distributions, making the ε→0 limit in Eq. (7) undefined for non-decomposable losses, is a mathematical approximation gap rather than a circular reduction: the paper explicitly replaces the limit with a finite difference at ε=1 and provides no claim that the limit exists in general. Missing rigor in a derivation step is not circularity, because no input is being relabeled as a prediction and no conclusion is assumed by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

VIF is a new computational approximation of a known statistical object, so it introduces no new physical or mathematical entities. The method has no fitted free parameters; the only constants are the data size n and the fixed finite-difference step ε=1. The load-bearing assumptions are convexity/differentiability of the loss (needed for the Hessian and implicit function) and the regularity conditions for the Cox asymptotic bound. These are standard for influence function theory but are assumptions the reader must accept.

assumptions (3)
  • domain assumption The loss L(θ,b) is strictly convex in θ and twice differentiable, ensuring a unique minimizer and invertible Hessian
    Invoked in Section 3.4 (Definition 3.3 and Proposition 3.1) and explicitly acknowledged in the Limitation paragraph. Without this, the implicit function theorem and the Hessian inverse in Eq. (10) are not fully justified.
  • domain assumption For the Cox regression error bound, standard regularity conditions hold: compact parameter space, bounded support of X, no tied events, non-informative censoring, non-singular information matrix
    Stated in Appendix A.5 as Theorem A.1 assumptions (1)-(4). These are common but nontrivial conditions for asymptotic analysis of the Cox model.
  • standard math Learned parameters θ̂(1) are an unconstrained local/global minimizer, so the gradient ∇_θ L(θ̂(1),1) = 0
    Used throughout Section 3.1 and in the VIF derivation. Standard for influence function analysis, but in practice the optimizer may not exactly reach a stationary point.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Versatile Influence Function for Data Attribution with Non-Decomposable Loss." pith.science (2026). https://pith.science/paper/22ZFAPGX

@misc{pith2026241201335,
  author       = {Pith},
  title        = {Pith review of: A Versatile Influence Function for Data Attribution with Non-Decomposable Loss},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/22ZFAPGX}},
  note         = {Machine review of arXiv:2412.01335}
}
abstract

Influence function, a technique rooted in robust statistics, has been adapted in modern machine learning for a novel application: data attribution -- quantifying how individual training data points affect a model's predictions. However, the common derivation of influence functions in the data attribution literature is limited to loss functions that can be decomposed into a sum of individual data point losses, with the most prominent examples known as M-estimators. This restricts the application of influence functions to more complex learning objectives, which we refer to as non-decomposable losses, such as contrastive or ranking losses, where a unit loss term depends on multiple data points and cannot be decomposed further. In this work, we bridge this gap by revisiting the general formulation of influence function from robust statistics, which extends beyond M-estimators. Based on this formulation, we propose a novel method, the Versatile Influence Function (VIF), that can be straightforwardly applied to machine learning models trained with any non-decomposable loss. In comparison to the classical approach in statistics, the proposed VIF is designed to fully leverage the power of auto-differentiation, hereby eliminating the need for case-specific derivations of each loss function. We demonstrate the effectiveness of VIF across three examples: Cox regression for survival analysis, node embedding for network analysis, and listwise learning-to-rank for information retrieval. In all cases, the influence estimated by VIF closely resembles the results obtained by brute-force leave-one-out retraining, while being up to $10^3$ times faster to compute. We believe VIF represents a significant advancement in data attribution, enabling efficient influence-function-based attribution across a wide range of machine learning paradigms, with broad potential for practical use cases.

Figures

Figures reproduced from arXiv: 2412.01335 by the authors.

Figure 1
Figure 1. VIF is applied to Zachary’s Karate network to estimate the influence of each node on the contrastive loss of a [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. VIF is applied to Zachary’s Karate network to estimate the influence of each node on the contrastive loss of a [PITH_FULL_IMAGE:figures/full_fig_p021_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages

  1. [8]

    Beta shapley: a unified and noise-reduced data valuation framework for machine learning

    Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint arXiv:2110.14049,

    arXiv

  2. [10]

    (a) VIF on (12,10) 01 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1920 21 22 23 24 2526 27 28 29 30 31 32 33 Groundtruth Heatmap; Node: (12,

    work page 1920

  3. [13]

    Cees GM Snoek, Marcel Worring, Jan C Van Gemert, Jan-Mark Geusebroek, and Arnold WM Smeulders

    doi: 10.1609/aaai.v36i8.20791. Cees GM Snoek, Marcel Worring, Jan C Van Gemert, Jan-Mark Geusebroek, and Arnold WM Smeulders. The challenge problem for automated detection of 101 semantic concepts in multimedia. In Proceedings of the 14th ACM international conference on Multimedia, pp. 421–430,

    doi:10.1609/aaai.v36i8.20791

  4. [18]

    Specifically, we consider two methods used by Koh & Liang [2017], Conjugate Gradient (CG) and LiSSA [Agarwal et al., 2017]

    C Efficient Inverse Hessian Approximation Existing methods for efficient inverse Hessian approximation used by the conventional IF for decomposable losses can be adapted to accelerate VIF. Specifically, we consider two methods used by Koh & Liang [2017], Conjugate Gradient (CG) and LiSSA [Agarwal et al., 2017]. The application of CG to VIF is straightforw...

    work page 2017

  5. [20]

    01 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1920 21 22 23 24 2526 27 28 29 30 31 32 33 VIF Heatmap; Node (15,

    work page 1920

  6. [21]

    (c) VIF on (15,13) 01 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1920 21 22 23 24 2526 27 28 29 30 31 32 33 Groundtruth Heatmap; Node: (15,

    work page 1920

  7. [1972]

    doi: 10.1111/j.2517-6161.1972.tb00899.x

    ISSN 1369-7412,1467-9868. doi: 10.1111/j.2517-6161.1972.tb00899.x. Junwei Deng and Jiaqi Ma. Computational Copyright: Towards A Royalty Model for Music Generative AI. arXiv [cs.AI], December

    arXiv 1972

  8. [1974]

    doi: 10.1080/01621459.1974.10482962

    ISSN 0162-1459,1537-274X. doi: 10.1080/01621459.1974.10482962. Peter J Huber and Elvezio M Ronchetti. Robust Statistics. Wiley Series in Probability and Statistics. Wiley-Blackwell, Hoboken, NJ, 2 edition, January

    arXiv 1974

Show all 21 references
  1. [1977]

    Improving recommendation accuracy using networks of substitutable and complementary products

    11 A Versatile Influence Function for Data Attribution with Non-Decomposable Loss A PREPRINT Tong Zhao, Julian McAuley, Mengya Li, and Irwin King. Improving recommendation accuracy using networks of substitutable and complementary products. In 2017 International Joint Conferen...

    2017

  2. [1985]

    doi: 10.2307/2336329

    ISSN 0006-3444,1464-3510. doi: 10.2307/2336329. Xin Rong. word2vec Parameter Learning Explained. arXiv [cs.CL], November

    doi

  3. [2006]

    Effective and efficient multilabel classification in domains with large number of labels

    Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. Effective and efficient multilabel classification in domains with large number of labels. In Proc. ECML/PKDD 2008 Workshop on Mining Multidimensional Data (MMD’08), volume 21, pp. 53–59,

    2008

  4. [2008]

    ISBN 9781605582054

    ACM Press. ISBN 9781605582054. doi: 10.1145/1390156.1390306. Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31,

  5. [2009]

    doi: 10.1002/9780470434697

    ISBN 9780470129906,9780470434697. doi: 10.1002/9780470434697. 10 A Versatile Influence Function for Data Attribution with Non-Decomposable Loss A PREPRINT Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions f...

    arXiv

  6. [2012]

    doi: 10.1017/cbo9780511802256

    ISBN 9780511802256,9780521496032. doi: 10.1017/cbo9780511802256. Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 6388–6421. PMLR,

    doi

  7. [2013]

    Trak: Attributing model behavior at scale

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186,

    arXiv

  8. [2014]

    ISBN 9781450329569

    ACM. ISBN 9781450329569. doi: 10.1145/2623330.2623732. Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920–19930,

  9. [2017]

    URL http://jmlr.org/papers/v18/16-491. html. Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference on Artificial Intelligence and Statistics, pp. 1899–1909. PMLR,

    1909

  10. [2020]

    What is your data worth to gpt? llm-scale data valuation with influence functions

    Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, et al. What is your data worth to gpt? llm-scale data valuation with influence functions. arXiv preprint arXiv:2405.13954,

    arXiv

  11. [2021]

    Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models

    Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. arXiv preprint arXiv:2310.00902,

    arXiv

  12. [2022]

    Evilseed: A guided approach to finding malicious web pages

    Luca Invernizzi, Paolo Milani Comparetti, Stefano Benvenuti, Christopher Kruegel, Marco Cova, and Giovanni Vigna. Evilseed: A guided approach to finding malicious web pages. In 2012 IEEE symposium on Security and Privacy, pp. 428–442. IEEE,

    2012

  13. [2023]

    Fastif: Scalable influence functions for efficient model interpretation and debugging

    Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. arXiv preprint arXiv:2012.15781,

    2012 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.