Pith. sign in

REVIEW 3 major objections 7 minor 79 references

Laplace Sample Information: Data Informativeness Through a Bayesian Lens

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One KL divergence scores every sample's unique information

desk verdict Solid, publishable proposal for a Laplace-based per-sample informativeness measure; the probe-transfer claim is overstated but fixable, and the paper deserves serious peer review. read the letter →

arxiv 2505.15303 v1 pith:ZYHDTOIP submitted 2025-05-21 cs.LG cs.AIcs.ITmath.IT

Johannes Kaiser , Kristian Schwethelm , Daniel Rueckert , Georgios Kaissis This is my paper
classification cs.LGcs.AIcs.ITmath.IT
keywords sampleinformativenessdatavaluationLaplaceapproximationKLdivergenceleave-one-outestimationmislabeldetectiondatasetdifficultyBayesiandeeplearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Laplace Sample Information (LSI), a per-sample measure of how much unique information a training example contributes to a neural network's parameters. LSI is defined as the $\mathrm{KL}$ divergence between two Gaussian approximations of the weight posterior, one obtained from a model trained on the full dataset and one from a model trained with that sample removed, and it inherits an information-theoretic justification as an upper bound on the pointwise mutual information between the sample and the parameters. The authors show experimentally that LSI ranks typical and atypical samples, detects mislabeled examples, orders classes and whole datasets by difficulty, and that the ranking computed on a small probe over frozen features transfers to larger models trained from scratch. If this is right, LSI gives a scalable, architecture-agnostic way to value data, prune redundant samples, clean mislabels, and audit how much influence any single example has on a trained model.

What carries the argument

The load-bearing object is the Laplace approximation to the weight posterior: once a network is trained, its posterior is approximated as $\mathcal{N}(\hat{\theta}, (\nabla_\theta^2 L(D,\hat{\theta}))^{-1})$, a Gaussian centered at the trained weights with covariance given by the inverse loss Hessian. LSI is then the closed-form $\mathrm{KL}$ divergence between this Gaussian for the full dataset and the corresponding Gaussian for the leave-one-out dataset. Because $\mathrm{KL}$ between Gaussians has an explicit expression, only the Hessian, or a diagonal or K-FAC approximation to it, is needed, and the paper uses a frozen pretrained feature extractor plus a one-hidden-layer probe to make leave-one-out retraining cheap while preserving the sample ordering.

What would settle it

Compute true leave-one-out LSI on a random sample of a thousand images from a large dataset using the actual large model, and compare its Spearman rank correlation with the probe-based LSI; if the correlation falls well below the reported roughly 0.9 at this scale, the efficient probe version of the method no longer measures what the full method measures.

Watch

Extended reading notes

Core claim

The paper's central claim is that the leave-one-out change in a Laplace-approximated Bayesian posterior, $\mathrm{LSI}(z_i,A,D_{-i}) = \mathrm{KL}(\mathcal{N}(\hat{\theta},\Sigma)\,\|\,\mathcal{N}(\hat{\theta}_{-i},\Sigma_{-i}))$, is a faithful measure of the unique information that sample $z_i$ contributes to a neural network's parameters. Here $\hat{\theta}$ and $\hat{\theta}_{-i}$ are the parameters trained with and without the sample, and $\Sigma$ and $\Sigma_{-i}$ are the inverse Hessians of the loss at those parameters, so the quantity is computable after training by automatic differentiation. The authors report that LSI follows a long-tailed distribution across samples, that high-LSI samples are disproportionately mislabeled, cropped, or atypical, that intermediate-LSI samples support generalization best while low-LSI samples are easily fit but add little, and that the ordering obtained from a small probe transfers to larger architectures trained from scratch.

Load-bearing premise

The ranking produced by LSI on a small probe trained over frozen pretrained features is assumed to match the ranking a large model trained from scratch on the raw data would produce.

Editorial extensions

If this is right

  • Samples with high LSI can be flagged automatically as mislabeled, out-of-distribution, or otherwise atypical, which supports automated dataset cleaning before training.
  • Training on only low-LSI samples gives high training accuracy but degraded test accuracy, whereas intermediate-LSI samples generalize best, so LSI-based pruning should remove the lowest-information samples rather than the hardest ones.
  • Because LSI rises as datasets shrink and is higher for harder datasets, it provides a quantitative, data-driven measure of dataset difficulty that aligns with human judgment on the pneumonia classes.
  • Gradient clipping in DP-SGD lowers both the magnitude and the spread of LSI, confirming that clipping bounds per-sample information flow, while additive noise leaves the LSI distribution essentially unchanged.
  • The probe-computed ordering transfers to larger architectures, so LSI can be applied at a fraction of the cost of full leave-one-out retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the frozen feature extractor is what supplies the typicality structure, LSI measured on a probe may conflate dataset informativeness with the inductive bias of the pretrained extractor; recomputing probe LSI with several different extractors and checking whether the sample rankings agree would test this directly.
  • Beyond the paper: the observed link to per-instance Rényi differential privacy suggests LSI could serve as an empirical audit of how much information individual samples actually leak under DP-SGD, complementing worst-case theoretical privacy bounds with per-sample measurements.
  • Beyond the paper: the generalization results imply a data-selection rule that removes the lowest-LSI samples while keeping the middle and high range, but the paper itself only compares thirds and does not optimize a selection threshold; that threshold remains a testable design choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Laplace Sample Information (LSI), a per-sample informativeness measure defined as the KL divergence between Laplace-approximated Gaussian posteriors over neural-network parameters, comparing a model trained on a full dataset with a model trained after removing one sample. The paper claims that LSI is information-theoretically grounded as a bound on pointwise conditional mutual information, that it orders samples by typicality, detects mislabeled samples, measures class-wise and dataset-level difficulty, and that it can be computed efficiently via a probe on frozen pretrained features while transferring to larger models. Experiments are reported on CIFAR-10/100, ImageNet subsets, a pediatric pneumonia dataset, IMDb text classification, and COCO contrastive learning.

Significance. If the claims are borne out, LSI would be a valuable, architecture-agnostic tool for data valuation, sample selection, and dataset curation. The paper has notable strengths: it reports comparisons with SUI, PSMI, and TRAK; it includes robustness checks for Hessian approximations and early-training behavior; and it provides code and precomputed LSI values. The core definition is simple and computationally feasible. However, the information-theoretic justification is asserted rather than proved, and the probe-transfer evidence, which underpins the scalability claim, is thinner than the abstract and Section 4.4 suggest. These are load-bearing issues that require additional support.

major comments (3)
  1. [Section 3, Definition 1] The claim that SI is an upper bound on the pointwise conditional mutual information between the model parameters and the datapoint is asserted without proof. Since this is the stated information-theoretic grounding for LSI, the paper should either provide a derivation or explicitly cite the exact result in Harutyunyan et al. (2021) that establishes this bound. As written, the central theoretical justification is not self-contained.
  2. [Section 4.4 and Appendix D, Table 1] The claim that probe-computed LSI transfers to larger models trained from scratch rests on a Spearman correlation of 0.93 on only 400 CIFAR-10 samples, together with qualitative subset train/test curves in Appendix J. These curves are consistent with the probe ordering but are never compared against subsets selected by an LSI computed on raw-input models. Consequently, the experiments cannot rule out that the ranking largely reflects the typicality structure of the frozen ImageNet-pretrained feature extractor rather than information relevant to a from-scratch model. Please provide either a larger-scale direct correlation between probe LSI and full-model LSI, with confidence intervals, or a raw-input LSI baseline on the target architectures to substantiate the transfer claim.
  3. [Section 4.2 and Appendix E] Mislabel detection is demonstrated only through overlapping density plots (Figure 5 and Figure 14). No AUROC, average precision, precision/recall, or threshold-based classification metric is reported, even though detecting mislabeled samples is listed as a contribution. Because the distributions shown are overlapping, a quantitative evaluation is necessary to support the claim that LSI can effectively distinguish mislabeled from correctly labeled samples.
minor comments (7)
  1. [Definition 2, Eq. (4)] The logarithm term in the KL formula is written as ln(det(Σ_-i)/Σ); the denominator should be det(Σ), otherwise the expression is dimensionally inconsistent. Please correct the typo.
  2. [Definition 2] The notation θ_MLE conflicts with Eq. (1), which includes a regularizer r(θ) acting as a log-prior; the resulting estimate is a MAP estimate, not an MLE. Please clarify the terminology or define the unregularized MLE.
  3. [Section 4.1] The claim of a speedup of "at least three orders of magnitude" is not accompanied by a timing comparison. Please add a measurement or soften the claim to reflect what is shown.
  4. [Appendix D, Table 1] The table caption does not state that the correlations are computed on only 400 CIFAR-10 samples, and no confidence intervals are reported. Please include the sample size and interval estimates.
  5. [Section 4.2, Figure 5] The density plot comparison would be more interpretable if the caption stated whether the densities are normalized per group; as shown, the visual comparison of overlap is ambiguous.
  6. [Section 4.3] The subset experiments compare against a dummy baseline but not against a random-subset baseline. Adding a random-selection curve would strengthen the practical claim that LSI-based ordering is useful for sample selection.
  7. [Section 4.1] All experiments use full-batch gradient descent. Given the paper's claim that LSI is agnostic to the training process, a brief discussion of how stochastic optimization might affect the Laplace approximation and the LSI values would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LSI is defined directly from parameter distributions, and all validations test the measure externally rather than fitting it.

full rationale

The paper's load-bearing definition, Eq. (4), computes LSI as KL(N(θ̂,Σ) ∥ N(θ̂₋ᵢ,Σ₋ᵢ)), i.e. it directly measures the change in a Laplace-approximated parameter posterior when sample i is removed. No evaluation target (mislabel indicator, learnability, subset accuracy) enters the definition, and no parameter of the LSI computation is fitted to those targets. The typicality/mislabel results are distributional comparisons after the fact; the learnability and transfer results train fresh models on LSI-selected subsets and observe the resulting curves, which is an external consistency check rather than a construction. The probe approximation is validated by Spearman correlation against full-CNN LSI on 400 CIFAR-10 samples (Appendix D, Table 1), which is empirical evidence, albeit limited, not a definitional equivalence. The Laplace approximation is imported from external literature (Daxberger et al., 2021) with stated assumptions; this is standard independent support, not a self-citation chain. The paper's acknowledged relation to average leave-one-out KL stability (Section 2) is an honest positioning, not a renaming that hides a fitted input. Overall, the central derivation is self-contained and the downstream experiments test the measure rather than fitting it.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to the evaluation benchmarks. The weight decay, learning rate, diagonal Hessian approximation, and probe architecture are standard design choices, not numbers tuned to make the reported accuracies come out.

assumptions (4)
  • domain assumption The KL divergence between parameter posteriors with and without a sample upper-bounds the pointwise conditional mutual information between the parameters and that sample.
    Stated after Definition 1 in Section 3 without proof; the paper cites Harutyunyan et al. (2021) as the source. If the bound is false, the information-theoretic interpretation of LSI is weakened, though the empirical ordering could still hold.
  • domain assumption The loss landscape near the converged solution is quadratic, so the trained parameters can be treated as a Gaussian posterior via the Laplace approximation.
    Invoked in Section 3 (Equation 3) to construct the Gaussian posterior; the paper acknowledges it as a common assumption and cites Daxberger et al. (2021) and MacKay (1992).
  • domain assumption A small probe head trained on features from a frozen ImageNet-pretrained ResNet-18 preserves the informativeness ordering obtained by training larger models on raw data.
    Used in Sections 4.1 and 4.4; validated on 400 CIFAR-10 samples (Spearman R>0.9) and via subset train/test curves, but not across the full range of scales or domains the paper claims transfer for.
  • domain assumption The empirical loss can be read as an i.i.d. log-likelihood and the regularizer as a log-prior, so the regularized MLE is a MAP estimate.
    This identification in Section 3 is what makes the Laplace posterior meaningful; it follows standard Bayesian practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Laplace Sample Information: Data Informativeness Through a Bayesian Lens." pith.science (2026). https://pith.science/paper/ZYHDTOIP

@misc{pith2026250515303,
  author       = {Pith},
  title        = {Pith review of: Laplace Sample Information: Data Informativeness Through a Bayesian Lens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZYHDTOIP}},
  note         = {Machine review of arXiv:2505.15303}
}
read the original abstract

Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. We propose Laplace Sample Information (LSI) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings. LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset. We experimentally show that LSI is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty. We demonstrate these capabilities of LSI on image and text data in supervised and unsupervised settings. Moreover, we show that LSI can be computed efficiently through probes and transfers well to the training of large models.

Figures

Figures reproduced from arXiv: 2505.15303 by the authors.

Figure 1
Figure 1. LSI distribution across individual samples of the investigated datasets. Investigating samples with low and high LSI, we find that samples that are strongly representative of their class provide little unique information to the model parameters. Consequently, representative samples have low LSI, while samples with high LSI often are cropped, mislabeled, or show the object from an uncommon perspective. This indicates… view at source ↗
Figure 2
Figure 2. Selected images with low/high LSI in ImageNet and the medical dataset. Samples with low LSI are representative of their underlying class, whereas high LSI samples are often mislabeled/ out-of-distribution (red dots) or atypical with respect to exposure, viewing angle, etc. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. LSI in supervised text classification on the IMDb dataset using BERT Even in multi-modal tasks, we find that typical samples with a prominent portrayal of class represen￾tatives (e.g. bunches of bananas) have low LSI, as they carry little unique information. Images in which the subject is not prominently shown, not corresponding to the banana label, or do not contain any bananas (e.g. box of cucumbers) have high LSI… view at source ↗
Figures from the paper (38 more)
Figure 4
Figure 4. Figure 4: Selected images with low/high LSI of contrastive learning on COCO Focusing on Mislabeled Samples To further show the effect of mislabeled data on sample in￾formativeness and demonstrate the capabilities of LSI in detecting mislabeled samples, we apply deliberate label …
Figure 5
Figure 5. Figure 5: LSI distribution on data with corrupted labels (mislabeled) vs. uncorrupted labels LSI can furthermore effectively distinguish label corruption (mislabeling, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: LSI in supervised text classification on the IMDb dataset using BERT Establishing a Class-wise Informativeness Ordering using LSI We next investigate the class-wise distribution of LSI, which allows for reasoning about which class imparts the most information to the mo…
Figure 7
Figure 7. Figure 7: LSI distributions across three classes of the pneumonia dataset. By increasing the additive Gaussian noise standard deviation (scaled to the initial standard deviation of the embedding) on the normal class, their dissimilarity increases, requiring the model to extract …
Figure 8
Figure 8. Figure 8: Training accuracy (left) and test accuracy (right) of models trained on subsets of CIFAR-10 [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Training accuracy (left) and test accuracy (right) of a ResNet-9 trained (from scratch) on [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Correlation plots between LSI, Smooth Unique Information (SUI) and Point-wise Sliced [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Correlation with confidence interval of 0.95 of [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Correlation between LSI on the full Hessian, on the diagonal approximation and the KFAC [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: (left) shows the distribution of LSI across correctly labeled samples with human labeling error. As with label flipping ( [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Correlation of the LSI sample ordering between the last epoch (1000) and during training [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: The left plot shows the distribution of the [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Correlation between LSI computed on the kernel density estimate drawn from Riemann Laplace approximation, the kernel density estimate drawn from Laplace approximation and LSI computed on the gaussian distributions estimated by the Laplace approximation without kernel …
Figure 19
Figure 19. Figure 19: Correlation between LSI computed on the kernel density estimate drawn from Riemann Laplace approximation, the kernel density estimate drawn from Laplace approximation and LSI computed on the gaussian distributions estimated by the Laplace approximation without kernel …
Figure 20
Figure 20. Figure 20: Accuracy of LSI based subsets of CIFAR10 across an MLP (flattened input), CNN, ResNet-9, ResNet-18 26 [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]
Figure 21
Figure 21. Figure 21: Accuracy of LSI based subsets of CIFAR100 across an MLP (flattened input), CNN, ResNet-9, ResNet-18 27 [PITH_FULL_IMAGE:figures/full_fig_p027_21.png]
Figure 22
Figure 22. Figure 22: Accuracy of LSI based subsets of Imagenette across ResNet-18 0 100 200 300 400 500 Epochs 0.0 0.2 0.4 0.6 0.8 1.0 Train Accuracy Imagewoof - Proxy High LSI Mid LSI Low LSI Dummy Baseline 0 100 200 300 400 500 Epochs 0.0 0.2 0.4 0.6 0.8 1.0 Test Accuracy [PITH_FULL_IM…
Figure 23
Figure 23. Figure 23: Accuracy of LSI based subsets of Imagewoof across ResNet-18 28 [PITH_FULL_IMAGE:figures/full_fig_p028_23.png]
Figure 24
Figure 24. Figure 24: Accuracy of LSI based subsets of the Pneumonia dataset across ResNet-18 K EXEMPLARY IMAGES OF IMAGENET WITH LOW AND HIGH LSI We show the applicability of LSI on ImageNet (Deng et al., 2009) using two popular subsets, Ima￾genette and Imagewoof (Howard, 2019). While Ima…
Figure 25
Figure 25. Figure 25: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p029_25.png]
Figure 26
Figure 26. Figure 26: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p030_26.png]
Figure 27
Figure 27. Figure 27: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p030_27.png]
Figure 28
Figure 28. Figure 28: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p031_28.png]
Figure 29
Figure 29. Figure 29: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p031_29.png]
Figure 30
Figure 30. Figure 30: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p032_30.png]
Figure 31
Figure 31. Figure 31: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p032_31.png]
Figure 32
Figure 32. Figure 32: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p032_32.png]
Figure 33
Figure 33. Figure 33: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p033_33.png]
Figure 34
Figure 34. Figure 34: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p033_34.png]
Figure 35
Figure 35. Figure 35: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p033_35.png]
Figure 36
Figure 36. Figure 36: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p034_36.png]
Figure 37
Figure 37. Figure 37: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p034_37.png]
Figure 38
Figure 38. Figure 38: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p035_38.png]
Figure 39
Figure 39. Figure 39: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p035_39.png]
Figure 40
Figure 40. Figure 40: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p036_40.png]
Figure 41
Figure 41. Figure 41: Exemplary images of the highest and lowest [PITH_FULL_IMAGE:figures/full_fig_p036_41.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 42 canonical work pages

  1. [1]

    Deep learning with differential privacy

    Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.\ 308--318, 2016

    2016

  2. [2]

    Estimating example difficulty using variance of gradients

    Chirag Agarwal, Daniel D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10368--10378, 2022

    work page 2022

  3. [3]

    The importance of better models in stochastic optimization

    Hilal Asi and John C Duchi. The importance of better models in stochastic optimization. Proceedings of the National Academy of Sciences, 116 0 (46): 0 22924--22930, 2019

    work page 2019

  4. [4]

    Eigenvalue corrected noisy natural gradient

    Juhan Bae, Guodong Zhang, and Roger Grosse. Eigenvalue corrected noisy natural gradient. arXiv preprint arXiv:1811.12565, 2018

    arXiv 2018

  5. [5]

    If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35: 0 17953--17967, 2022

    Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35: 0 17953--17967, 2022

    work page 2022

  6. [6]

    Measures of information reflect memorization patterns

    Rachit Bansal, Danish Pruthi, and Yonatan Belinkov. Measures of information reflect memorization patterns. Advances in Neural Information Processing Systems, 35: 0 18472--18486, 2022

    work page 2022

  7. [7]

    Algorithmic stability for adaptive data analysis

    Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pp.\ 1046--1059, 2016

    work page 2016

  8. [8]

    Influence functions in deep learning are fragile

    Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint arXiv:2006.14651, 2020

    arXiv 2006

Show all 79 references
  1. [9]

    Riemannian laplace approximations for bayesian neural networks

    Federico Bergamin, Pablo Moreno-Mu \ n oz, S ren Hauberg, and Georgios Arvanitidis. Riemannian laplace approximations for bayesian neural networks. Advances in Neural Information Processing Systems, 36, 2024

    2024

  2. [10]

    The promises and pitfalls of stochastic gradient langevin dynamics

    Nicolas Brosse, Alain Durmus, and Eric Moulines. The promises and pitfalls of stochastic gradient langevin dynamics. Advances in Neural Information Processing Systems, 31, 2018

    2018

  3. [11]

    On the convergence and calibration of deep learning with differential privacy

    Zhiqi Bu, Hua Wang, and Qi Long. On the convergence and calibration of deep learning with differential privacy. arXiv preprint arXiv:2106.07830, 2021

    2021 arXiv

  4. [12]

    Coherent gradients: An approach to understanding generalization in gradient descent-based optimization

    Satrajit Chatterjee. Coherent gradients: An approach to understanding generalization in gradient descent-based optimization. arXiv preprint arXiv:2002.10657, 2020

    2002 arXiv

  5. [13]

    Private set generation with discriminative information

    Dingfan Chen, Raouf Kerkouche, and Mario Fritz. Private set generation with discriminative information. Advances in Neural Information Processing Systems, 35: 0 14678--14690, 2022

    2022

  6. [14]

    On lazy training in differentiable programming

    Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019

    2019

  7. [15]

    Accuracy for sale: Aggregating data with a variance constraint

    Rachel Cummings, Katrina Ligett, Aaron Roth, Zhiwei Steven Wu, and Juba Ziani. Accuracy for sale: Aggregating data with a variance constraint. In Proceedings of the 2015 conference on innovations in theoretical computer science, pp.\ 317--324, 2015

    2015

  8. [16]

    Laplace redux-effortless bayesian deep learning

    Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34: 0 20089--20103, 2021

    2021

  9. [17]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  10. [18]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805

    2019 arXiv

  11. [19]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2010 arXiv

  12. [20]

    Generalization in adaptive data analysis and holdout reuse

    Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toni Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. Advances in Neural Information Processing Systems, 28, 2015

    2015

  13. [21]

    Understanding dataset difficulty with V -usable information

    Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V -usable information. In International Conference on Machine Learning, pp.\ 5988--6008. PMLR, 2022

    2022

  14. [22]

    Liberty or depth: Deep bayesian neural nets do not need complex weight posterior approximations

    Sebastian Farquhar, Lewis Smith, and Yarin Gal. Liberty or depth: Deep bayesian neural nets do not need complex weight posterior approximations. Advances in Neural Information Processing Systems, 33: 0 4346--4357, 2020

    2020

  15. [23]

    Does learning require memorization? a short tale about a long tail

    Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp.\ 954--959, 2020

    2020

  16. [24]

    Calibrating noise to variance in adaptive data analysis

    Vitaly Feldman and Thomas Steinke. Calibrating noise to variance in adaptive data analysis. In Conference On Learning Theory, pp.\ 535--544. PMLR, 2018

    2018

  17. [25]

    The butterfly effect in artificial intelligence systems: Implications for ai bias and fairness

    Emilio Ferrara. The butterfly effect in artificial intelligence systems: Implications for ai bias and fairness. Machine Learning with Applications, 15: 0 100525, 2024

    2024

  18. [26]

    how biased are your features?

    Bishwamittra Ghosh, Debabrota Basu, and Kuldeep S Meel. “how biased are your features?”: Computing fairness influence functions with global sensitivity analysis. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 138--148, 2023

    2023

  19. [27]

    Sliced mutual information: A scalable measure of statistical dependence

    Ziv Goldfeld and Kristjan Greenewald. Sliced mutual information: A scalable measure of statistical dependence. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 17567--17578....

    2021

  20. [28]

    A kronecker-factored approximate fisher matrix for convolution layers

    Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning, pp.\ 573--582. PMLR, 2016

    2016

  21. [29]

    Studying large language model generalization with influence functions

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023

    2023 arXiv

  22. [30]

    Estimating informativeness of samples with smooth unique information

    Hrayr Harutyunyan, Alessandro Achille, Giovanni Paolini, Orchid Majumder, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Estimating informativeness of samples with smooth unique information. arXiv preprint arXiv:2101.06640, 2021

    2021 arXiv

  23. [31]

    Control batch size and learning rate to generalize well: Theoretical and empirical evidence

    Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. Advances in neural information processing systems, 32, 2019

    2019

  24. [32]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  25. [33]

    What do compressed deep neural networks forget? arxiv

    S Hooker, A Courville, G Clark, Y Dauphin, and A Frome. What do compressed deep neural networks forget? arxiv. Learning, 2020 a

    2020

  26. [34]

    Characterising bias in compressed models

    Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characterising bias in compressed models. arXiv preprint arXiv:2010.03058, 2020 b

    2010 arXiv

  27. [35]

    Imagenette, 2019

    Jeremy Howard. Imagenette, 2019. URL https://github.com/fastai/imagenette/. https://github.com/fastai/imagenette

    2019

  28. [36]

    Characterizing structural regularities of labeled data in overparameterized models

    Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206, 2020

    2002 arXiv

  29. [37]

    Not all samples are created equal: Deep learning with importance sampling

    Angelos Katharopoulos and Fran c ois Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp.\ 2525--2534. PMLR, 2018

    2018

  30. [38]

    Kermany, Michael Goldbaum, Wenjia Cai, Carolina C.S

    Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C.S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Magdalene Y.L. Ting, Jie Zhu, Christina Li, Sierra Hewett, Jason Dong, Ian ...

    2018

  31. [39]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...

    2017

  32. [40]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp.\ 1885--1894. PMLR, 2017

    2017

  33. [41]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL https://www.cs.toronto.edu/ kriz/learning-features-2009-TR.pdf

    2009

  34. [42]

    Stochastic modified equations and adaptive stochastic gradient algorithms

    Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp.\ 2101--2110. PMLR, 2017

    2017

  35. [43]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. URL https://arxiv.org/abs/1405.0312

    2015 arXiv

  36. [44]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp.\ 142--150,...

    2011

  37. [45]

    David J. C. MacKay. Bayesian Interpolation . Neural Computation, 4 0 (3): 0 415--447, 05 1992. ISSN 0899-7667. doi:10.1162/neco.1992.4.3.415. URL https://doi.org/10.1162/neco.1992.4.3.415

    1992 doi

  38. [46]

    o ren Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H \

    S \"o ren Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H \"o ltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Internat...

    2022

  39. [47]

    R \'e nyi differential privacy

    Ilya Mironov. R \'e nyi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pp.\ 263--275. IEEE, 2017

    2017

  40. [48]

    Information-theoretic generalization bounds for sgld via data-dependent estimates

    Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information-theoretic generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems, 32, 2019

    2019

  41. [49]

    On the generalization properties of differential privacy

    Kobbi Nissim and Uri Stemmer. On the generalization properties of differential privacy. CoRR, abs/1504.05800, 2015

    2015 arXiv

  42. [50]

    What can linearized neural networks actually say about generalization? In M

    Guillermo Ortiz-Jimenez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neural networks actually say about generalization? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing S...

  43. [51]

    Trak: Attributing model behavior at scale

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023

    2023 arXiv

  44. [52]

    Estimating training data influence by tracing gradient descent

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 0 19920--19930, 2020

    2020

  45. [53]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://ar...

    2021 arXiv

  46. [54]

    Information-theoretic analysis of stability and bias of learning algorithms

    Maxim Raginsky, Alexander Rakhlin, Matthew Tsao, Yihong Wu, and Aolin Xu. Information-theoretic analysis of stability and bias of learning algorithms. In 2016 IEEE Information Theory Workshop (ITW), pp.\ 26--30. IEEE, 2016

    2016

  47. [55]

    On leave-one-out conditional mutual information for generalization

    Mohamad Rida Rammal, Alessandro Achille, Aditya Golatkar, Suhas Diggavi, and Stefano Soatto. On leave-one-out conditional mutual information for generalization. Advances in Neural Information Processing Systems, 35: 0 10179--10190, 2022

    2022

  48. [56]

    Max-information, differential privacy, and post-selection hypothesis testing

    Ryan Rogers, Aaron Roth, Adam Smith, and Om Thakkar. Max-information, differential privacy, and post-selection hypothesis testing. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp.\ 487--494. IEEE, 2016

    2016

  49. [57]

    Scaling up influence functions

    Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 8179--8186, 2022

    2022

  50. [58]

    Theoretical and practical perspectives on what influence functions do

    Andrea Schioppa, Katja Filippova, Ivan Titov, and Polina Zablotskaia. Theoretical and practical perspectives on what influence functions do. Advances in Neural Information Processing Systems, 36, 2024

    2024

  51. [59]

    Reasoning about generalization via conditional mutual information

    Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Conference on Learning Theory, pp.\ 3437--3452. PMLR, 2020

    2020

  52. [60]

    Energy and policy considerations for deep learning in nlp

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019

    1906 arXiv

  53. [61]

    Data pruning via moving-one-sample-out

    Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang, Fan Wang, and Xiaojuan Qi. Data pruning via moving-one-sample-out. arXiv preprint arXiv:2310.14664, 2023

    2023 arXiv

  54. [62]

    Role of training data variability on classifier performance and generalizability

    Ryan Therrien and Scott Doyle. Role of training data variability on classifier performance and generalizability. In Medical Imaging 2018: Digital Pathology, volume 10581, pp.\ 58--70. SPIE, 2018

    2018

  55. [63]

    U nsal and Melek \

    Ay s e \" U nsal and Melek \" O nen. Information-theoretic approaches to differential privacy. ACM Comput. Surv., 56 0 (3), oct 2023. ISSN 0360-0300. doi:10.1145/3604904. URL https://doi.org/10.1145/3604904

    2023 doi

  56. [64]

    Per-instance differential privacy

    Yu-Xiang Wang. Per-instance differential privacy. arXiv preprint arXiv:1707.07708, 2017

    2017 arXiv

  57. [65]

    Learning with noisy labels revisited: A study using real-world human annotations

    Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:2110.12088, 2021

    2021 arXiv

  58. [66]

    An empirical study of stochastic gradient descent with structured covariance noise

    Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. An empirical study of stochastic gradient descent with structured covariance noise. In International Conference on Artificial Intelligence and Statistics, pp.\ 3621--3631. PMLR, 2020

    2020

  59. [67]

    Shelvia Wongso, Rohan Ghosh, and M. Motani. Pointwise sliced mutual information for neural network explainability. 2023 IEEE International Symposium on Information Theory (ISIT), pp.\ 1776--1781, 2023. URL https://api.semanticscholar.org/CorpusID:261081514

    2023

  60. [68]

    Deltagrad: Rapid retraining of machine learning models

    Yinjun Wu, Edgar Dobriban, and Susan Davidson. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning, pp.\ 10355--10366. PMLR, 2020

    2020

  61. [69]

    Machine unlearning: A survey

    Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S Yu. Machine unlearning: A survey. ACM Computing Surveys, 56 0 (1): 0 1--36, 2023

    2023

  62. [70]

    A theory of usable information under computational constraints

    Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689, 2020

    2002 arXiv

  63. [71]

    Leave-one-out distinguishability in machine learning

    Jiayuan Ye, Anastasia Borovykh, Soufiane Hayou, and Reza Shokri. Leave-one-out distinguishability in machine learning. arXiv preprint arXiv:2309.17310, 2023

    2023 arXiv

  64. [72]

    Individual privacy accounting for differentially private stochastic gradient descent

    Da Yu, Gautam Kamath, Janardhan Kulkarni, Tie-Yan Liu, Jian Yin, and Huishuai Zhang. Individual privacy accounting for differentially private stochastic gradient descent. arXiv preprint arXiv:2206.02617, 2022

    2022 arXiv

  65. [73]

    Data-centric artificial intelligence: A survey

    Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158, 2023

    2023 arXiv

  66. [74]

    The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects

    Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. arXiv preprint arXiv:1803.00195, 2018

    2018 arXiv

  67. [75]

    Weak and strong gradient directions: Explaining memorization, generalization, and hardness of examples at scale

    Piotr Zielinski, Shankar Krishnan, and Satrajit Chatterjee. Weak and strong gradient directions: Explaining memorization, generalization, and hardness of examples at scale. arXiv preprint arXiv:2003.07422, 2020

    2003 arXiv

  68. [76]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  69. [77]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  70. [78]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  71. [79]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.