Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Attributing Data for Sharpness-Aware Minimization

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper derives two influence-function estimators for Sharpness-Aware Minimization, SAM-HIF and SAM-GIF, that approximate the leave-one-out retrained model and support mislabeled-data detection and model editing without retraining.

desk verdict SAM-GIF is a sensible reuse of TracIn for SAM and the experiments support it as a heuristic, but the paper's central SAM-HIF derivation relies on a false stationarity condition and needs major rework. read the letter →

arxiv 2507.04059 v1 pith:4OZ2YSA5 submitted 2025-07-05 cs.LG cs.AIcs.CVstat.ML

Chenyang Ren , Yifan Jia , Huanyi Xie , Zhaobin Xu , Tianxing Wei , Liangyu Wang , Lijie Hu , Di Wang This is my paper
classification cs.LGcs.AIcs.CVstat.ML
keywords dataattributioninfluencefunctionssharpness-awareminimizationleave-one-outapproximationgradienttrajectorymislabeleddetectionmodeleditingNeumannseriesiHVP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to make data attribution work for models trained with Sharpness-Aware Minimization, where removing one training point changes both the final weights and the loss-maximizing perturbation computed in the inner loop. It derives two estimators of what the model would become if a point were removed. The Hessian-based estimator uses only the trained weights and the perturbation at the optimum; the gradient-trajectory estimator sums the stored per-step gradients. The paper reports that both match retraining closely, identify mislabeled data with high precision, and update models in seconds instead of hours.

What carries the argument

The machinery is the leave-one-out (LOO) approximation built on Taylor expansion of the SAM optimality condition. Lemma 4.2 reduces the SAM stationary condition to $\nabla L_S(\omega^*+\hat{\epsilon}(\omega^*))=0$; Theorem 4.5 then solves the expanded equation to obtain $\mathrm{SAM\text{-}HIF}$, and Theorem 4.6 sums the checkpoint gradients to obtain $\mathrm{SAM\text{-}GIF}_{\mathrm{SGD}}$. The two formulas are the central objects: they convert the question 'what would retraining without this point produce?' into linear-algebra operations on quantities already available from training.

What would settle it

Compute the gradient norm $\|\nabla L_S(\omega^*+\hat{\epsilon}(\omega^*))\|$ at the end of a standard SAM run; if it is far from zero, the stationary-point premise fails. In parallel, retrain several SAM models with a single point deleted and correlate the measured $\omega_k-\omega^*$ with the SAM-HIF and SAM-GIF predictions; a correlation near zero would refute the central approximation.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the leave-one-out retrained SAM model $\omega_k$ is accurately approximated by $\omega^*$ minus one of two closed-form influence scores. $\mathrm{SAM\text{-}HIF}$ extends the classical influence function by including the derivative of the inner perturbation in the Hessian, giving $\mathrm{SAM\text{-}HIF}(x_k,y_k) = -(H_\omega + H_\omega\, d\hat{\epsilon}(\omega^*)/d\omega)^{-1} \nabla L_k^S(\omega^*+\hat{\epsilon}(\omega^*))$. $\mathrm{SAM\text{-}GIF}$ instead accumulates learning-rate-weighted gradients at the perturbed points along the stored training trajectory, giving $\mathrm{SAM\text{-}GIF}_{\mathrm{SGD}}(x_k,y_k)=\sum_{t=0}^{T-1}\eta_t B_{k,t}\nabla L_k^S(\omega_t+\hat{\epsilon}(\omega_t))$. These are presented as the first influence-based attribution tools for SAM, with experiments on CIFAR-10/100, MNIST, Mini-ImageNet, and HAM10000 showing that removing points flagged as harmful recovers accuracy from label noise, removing points flagged as valuable degrades accuracy like retraining, and the estimated edited parameters stay close to truly retrained parameters.

Load-bearing premise

The load-bearing premise is Lemma 4.2 in Section 4.1: that a SAM optimum satisfies $\nabla L_S(\omega^*+\hat{\epsilon}(\omega^*))=0$, which lets every later Taylor expansion start from a stationary point; if actual SAM training stops at points where this residual is large, the SAM-HIF and SAM-GIF formulas inherit that error.

Editorial extensions

If this is right

  • Removing the points with the lowest influence scores improves test accuracy under label noise, and the paper reports detecting over 90% of noisy labels once roughly 40% of samples are removed.
  • Model editing becomes feasible without retraining: the edited parameters $\omega^* - \mathrm{IF}(x_k,y_k)$ approximate the retrained model, with accuracy close to a full retrain but runtime reduced from thousands of seconds to a few seconds.
  • SAM-HIF works when no training trajectory is stored, and the Neumann-series approximation of the inverse Hessian-vector product avoids storing the full Hessian matrix.
  • When checkpoints are available, SAM-GIF matches retraining more closely than the Hessian-based variant on CIFAR-10 and runs faster, since it avoids Hessian computation.
  • The same influence scores support interpretability by tracing misclassified test predictions back to the most helpful and most harmful training examples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same leave-one-out argument should extend to SAM variants that compute perturbations differently, because only the Jacobian $d\hat{\epsilon}(\omega^*)/d\omega$ changes in the Hessian-based formula.
  • If the reported accuracy holds at scale, these estimators offer a cheap deletion audit for privacy: a deployed SAM model could be adjusted for a removed user's data by subtraction rather than retraining.
  • The trajectory-based estimator's accuracy depends on how many checkpoints are stored; the paper's ablation suggests performance approaches retraining as checkpoints grow, implying a direct cost-accuracy trade-off in deployment.
  • A natural next test is whether the scores remain calibrated for large deletion fractions or shifted validation distributions, where the local Taylor assumptions are most strained.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper proposes two data-attribution methods for Sharpness-Aware Minimization (SAM): a Hessian-based influence function (SAM-HIF) and a gradient-trajectory influence function (SAM-GIF). The authors derive SAM-HIF as a closed-form approximation of the leave-one-out parameter change for SAM-trained models, and SAM-GIF as a checkpoint-based estimator that avoids Hessian inversion. They validate the methods on CIFAR-10/100, MNIST, MiniImageNet, HAM10000, CUB, and FOOD-101, reporting large runtime reductions relative to retraining and demonstrating applications in mislabeled-data detection, model editing, and interpretability. The central theoretical contribution is the claim that SAM-HIF and SAM-GIF accurately approximate LOO retraining for SAM without retraining.

Significance. If the derivations were sound, the paper would address a real and timely gap: existing influence-function methods do not directly handle SAM's bilevel structure. The proposed SAM-GIF in particular, being Hessian-free and trajectory-based, could be practically useful and scalable, and the experiments show order-of-magnitude speedups (e.g., CIFAR-10 retraining 3516 s vs. SAM-GIF 4.9 s). However, the paper's theoretical foundation has multiple load-bearing errors: a false stationarity equivalence, an incorrect gradient expression, an unquantified dropped term in the SAM-HIF derivation, and a sign inconsistency in SAM-GIF. These issues prevent the main closed-form claims from being accepted as derived from the stated SAM objective.

major comments (5)
  1. [Section 4.1, Lemma 4.2 and Appendix A.1, Eq. (10)] Lemma 4.2 states that the SAM stationarity condition is 'equivalent to' ∇LS(ω*+ε^(ω*))=0 after displaying a condition that contains λω* and a dε^(ω*)/dω term. This equivalence is false for λ>0 unless ω*=0 and the dε/dω term also vanishes. Equation (10) in Appendix A.1 then uses ∇LS(ω*+ε^(ω*))=0 as the base point for every Taylor expansion in Theorems 4.3 and 4.5, so the influence formulas are not actually derived from the optimization problem in Eq. (1). Additionally, Eq. (2) includes dε^(ω)/dω·∇LS(ω+ε^(ω)) in the gradient of the SAM objective, which conflicts with Danskin's theorem for a max over a compact ball; the correct Danskin gradient is ∇LS(ω+ε^(ω)) with no dε/dω term. These are not stylistic issues: the claimed equivalence and the gradient expression are the foundation of the subsequent derivations.
  2. [Appendix A.1, Eq. (11)] The Taylor expansion of the stationarity condition omits the λ(ωδ−ω*) term that must appear when the objective contains (λ/2)||ω||². The text defines Hω = ∇²LS(ω*+ε^(ω*)) + λI, so the λI contribution is introduced in the final formula, but it is not present in the displayed expansion that is supposed to justify the formula. As written, the derivation is internally inconsistent: either the expansion should be performed on the regularized gradient, in which case λ(ωδ−ω*) appears, or the Hω definition should not include λI. This indeterminacy affects the central closed-form claim of Theorem 4.3.
  3. [Appendix A.1, Eqs. (13)–(15), Theorem 4.5] The derivation of SAM-HIF drops the term −(Hω + Hω dε^(ω*)/dω)^{-1} Hω · dεδ(ω*)/dδ|_{δ=0} from dωδ/dδ, with the justification 'to enhance the computation efficiency.' However, this term exactly captures how the worst-case perturbation changes when sample k is up-weighted, which is the perturbation influence that SAM-HIF is claimed to model. No argument is given that this term is negligible, and in general it is of the same order as the retained term δ∇Lk. Consequently, SAM-HIF is not a comprehensive estimator of data influence as advertised; it is a further approximation whose error is unquantified.
  4. [Section 4.2, Theorem 4.6] There is a sign inconsistency in the gradient-trajectory influence function. The derivation gives dω_{T,δ}/dδ|_{δ=0} = −Σ_t η_t ∇Lk(ωt+ε^(ωt)) (up to the discarded Hessian term), which implies that for δ=−1 (removal) one obtains ωk ≈ ω* + Σ_t η_t ∇Lk(ωt+ε^(ωt)). The theorem, however, defines SAM-GIFSGD = Σ_t η_t B_{k,t}∇Lk(ωt+ε^(ωt)) with a positive sign and then states ωk ≈ ω* − SAM-GIFSGD. This is the opposite sign from what the derivation implies. Earlier in Section 4.2 the quantity SAM-GIFGD is defined with a minus sign, so the text is internally inconsistent about the sign convention, and the parameter-update formula in Theorem 4.6 does not follow from the displayed derivation.
  5. [Section 4.3.2, Theorem 4.8] The interpretation of the influence score IS is incorrect. The theorem defines IS ≈ Σ_{(x,y)∈Dval} ℓ(x,y;ω*) − ℓ(x,y;ω*_{−k}); that is, IS > 0 means the loss before removal is higher than the loss after removal, so removal improves performance on the validation set. The sentence 'A positive IS indicates that removing the data point will deteriorate the model's performance' is therefore backwards. This sign error also propagates to the downstream use in Section 5.4, where low IS is used to identify harmful data; under the stated definition, harmful data should correspond to positive IS.
minor comments (5)
  1. [Appendix A.2, Lemma A.2 proof] The intermediate quantity εδ(ω*) is defined as an arg min, but it should be an arg max to match the perturbation objective.
  2. [Section 4.3.1] The text refers to 'Proposition 4.3' when the result is Theorem 4.3; the label is incorrect.
  3. [Section 5.2, Tables 2 and 3] The baseline is called 'TARK' in the text and tables, but reference [46] is the TRAK paper by Park et al.; the name should be corrected consistently.
  4. [Section 5.1 and Table 3] CUB and FOOD-101 are used in Table 3 but are not described in the dataset list in Section 5.1; please add their descriptions and the random-seed or train/test split details.
  5. [Section 5.2, Table 1] The runtime column for retraining on MINI-ImageNet is 682.56 s while the accuracy is 0.6835; it would be helpful to state the number of epochs and architecture for that dataset, as the numbers appear to be on a different scale from CIFAR-10/100.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SAM attribution formulas are derived from standard influence-function and TracIn-style expansions, and the empirical evaluations are validated against actual retraining rather than against the method's own fitted values.

full rationale

The paper's central derivations are not circular. SAM-HIF in Theorem 4.5 is obtained by a Taylor expansion of the SAM optimality condition around the trained weights, yielding a closed-form expression in terms of the Hessian and the per-sample gradient at the perturbed point; no parameter is fitted to the leave-one-out changes it later claims to predict. SAM-GIF in Theorem 4.6 is an explicit accumulation of per-step gradient terms with the batch indicator B_{k,t}, directly extending the TracIn formulation of Pruthi et al. and then compared with real leave-one-out retraining, so its outputs are not forced by construction. The paper's self-citations, such as [27], [28], [49], and [65], are related influence-function studies and are not load-bearing for the derivation of SAM-HIF or SAM-GIF. The serious mathematical objections raised by the reader — specifically the questionable equivalence in Lemma 4.2 between the stationarity condition containing λω* and the reduced condition ∇LS(ω*+ε(ω*))=0, the omission of the λ(ωδ−ω*) term in the Taylor expansion, and the nonstandard chain-rule term in Eq. (2) — are correctness and rigor issues, not circularity: even if Lemma 4.2 is false, the argument does not reduce to assuming the conclusion it claims to derive. Likewise, approximating dε(ω*)/dω with Eq. (6), taken from Foret et al., is a standard SAM approximation and not a self-citation or ansatz-smuggled premise. Because the empirical comparisons use retraining as ground truth and the formulas do not contain fitted parameters renamed as predictions, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The methods inherit standard IF and TracIn machinery. The paper's own additions are the perturbation-aware Hessian correction and the sign-invariance assumption; both are unverified and the stationarity premise used for the Taylor expansion is inconsistent with the stated objective.

free parameters (2)
  • Neumann series truncation order J = not specified
    Used in Section 4.3.1 to approximate the inverse Hessian-vector product; accuracy and runtime depend on J, but no value or convergence criterion is given in the main text (Algorithm 4 stops when ||I_j - I_{j-1}|| < ζ).
  • Number of checkpoints s for SAM-GIF = up to 10 in ablation (Figure 12)
    SAM-GIF sums per-checkpoint gradients; the approximation quality and runtime grow with s, as shown in Figure 12, but the main experiments do not state s.
assumptions (5)
  • standard math Danskin's theorem and the equivalent stationarity condition ∇LS(ω*+ε(ω*)) = 0
    Invoked in Lemma 4.2 and Appendix A.1 Eq (10); the paper's use drops the λω* term and is inconsistent with the stated objective.
  • ad hoc to paper Sign of ∇wLS(ω,0) is unchanged when one training point is removed
    Stated after Eq (6) in Section 4.1 to enable computation of dε/dω; no empirical or theoretical justification is offered.
  • ad hoc to paper The δ-perturbed maximizer difference εδ(ω*) - ε(ω*) is negligible as ε→0
    Lemma 4.4 and the drop of the first term in Appendix Eq (15); used to simplify SAM-HIF despite ρ being finite in practice.
  • domain assumption Influence-function Taylor expansion is valid for the non-convex SAM loss at an approximate optimum
    Standard IF assumption inherited from Koh and Liang; deep networks are not at exact stationary points and Hessians are approximated, a known fragility cited by the paper itself.
  • domain assumption The SAM training algorithm in Algorithm 1 (using approximate ε and dropping the second gradient term) faithfully represents SAM
    Section 4.2 assumes this to derive SAM-GIF; it matches the original SAM paper but means the method targets the algorithm rather than the exact SAM objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attributing Data for Sharpness-Aware Minimization." pith.science (2026). https://pith.science/paper/4OZ2YSA5

@misc{pith2026250704059,
  author       = {Pith},
  title        = {Pith review of: Attributing Data for Sharpness-Aware Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4OZ2YSA5}},
  note         = {Machine review of arXiv:2507.04059}
}
read the original abstract

Sharpness-aware Minimization (SAM) improves generalization in large-scale model training by linking loss landscape geometry to generalization. However, challenges such as mislabeled noisy data and privacy concerns have emerged as significant issues. Data attribution, which identifies the contributions of specific training samples, offers a promising solution. However, directly rendering existing data influence evaluation tools such as influence functions (IF) to SAM will be inapplicable or inaccurate as SAM utilizes an inner loop to find model perturbations that maximize loss, which the outer loop then minimizes, resulting in a doubled computational structure. Additionally, this bilevel structure complicates the modeling of data influence on the parameters. In this paper, based on the IF, we develop two innovative data valuation methods for SAM, each offering unique benefits in different scenarios: the Hessian-based IF and the Gradient Trajectory-based IF. The first one provides a comprehensive estimation of data influence using a closed-form measure that relies only on the trained model weights. In contrast, the other IF for SAM utilizes gradient trajectory information during training for more accurate and efficient data assessment. Extensive experiments demonstrate their effectiveness in data evaluation and parameter tuning, with applications in identifying mislabeled data, model editing, and enhancing interpretability.

Figures

Figures reproduced from arXiv: 2507.04059 by the authors.

Figure 1
Figure 1. Harmful data removal experiment. IS means using the influence score to determine which [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Accuracy performance on four datasets after removing varying proportions of helpful data [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Harmful data removal results compared with baselines. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The most helpful and harmful training data tracked by misclassified data [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Ablation studies on CIFAR-10 and CIFAR-100. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Harmful data removal experiment on CIFAR-100 dataset. IS: using the influence score to [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Harmful data removal experiment on Mini-ImageNet dataset. IS: using the influence score [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Harmful data removal experiment on MNIST dataset. IS: using the influence score to [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: The most helpful and harmful training data tracked by misclassified data on MNIST dataset [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: The most helpful and harmful training data tracked by misclassified data on CIFAR-100 [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: The most helpful and harmful training data tracked by misclassified data on Mini-ImageNet [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Ablation study of SAM-GIF on Cifar10 dataset [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. In-Run Data Shapley for Adam Optimizer

    cs.LG 2026-01 reject novelty 4.0 of 10

    An 'Adam-aware' approximation for In-Run Data Shapley is proposed, but its central theorem is not proven and the fidelity test uses a proxy that is not the true Shapley value.

Reference graph

Works this paper leans on

68 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Learning multiple layers of features from tiny images

    Krizhevsky Alex. Learning multiple layers of features from tiny images. https://www. cs. toronto. edu/kriz/learning-features-2009-TR. pdf, 2009

    work page 2009

  2. [2]

    Sharpness- aware minimization leads to low-rank features

    Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness- aware minimization leads to low-rank features. Advances in Neural Information Processing Systems, 36:47032–47051, 2023

    work page 2023

  3. [3]

    Towards understanding sharpness-aware minimization

    Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In International Conference on Machine Learning, pages 639–668. PMLR, 2022

    work page 2022

  4. [4]

    Sharpness-aware minimization improves language model generalization

    Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-aware minimization improves language model generalization. arXiv preprint arXiv:2110.08529, 2021

    arXiv 2021

  5. [5]

    Influence functions in deep learning are fragile

    S Basu, P Pope, and S Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations (ICLR), 2021

    work page 2021

  6. [6]

    Un- derstanding the origins of bias in word embeddings

    Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. Un- derstanding the origins of bias in word embeddings. In International conference on machine learning, pages 803–811. PMLR, 2019

    work page 2019

  7. [7]

    Entropy-sgd: Biasing gradient descent into wide valleys

    Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124018, 2019

    work page 2019

  8. [8]

    Multi-stage influence function

    Hongge Chen, Si Si, Yang Li, Ciprian Chelba, Sanjiv Kumar, Duane Boning, and Cho-Jui Hsieh. Multi-stage influence function. Advances in Neural Information Processing Systems, 33:12732–12742, 2020

    work page 2020

Show all 68 references
  1. [9]

    When vision transformers outperform resnets without pre-training or strong data augmentations

    Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. arXiv preprint arXiv:2106.01548, 2021

    2021 arXiv

  2. [10]

    Detection of influential observation in linear regression

    R Dennis Cook. Detection of influential observation in linear regression. Technometrics, 42(1):65–68, 2000

    2000

  3. [11]

    Characterizations of an empirical influence function for detecting influential cases in regression

    R Dennis Cook and Sanford Weisberg. Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22(4):495–508, 1980

    1980

  4. [12]

    The theory of max-min and its application to weapons allocation problems, volume 5

    John M Danskin. The theory of max-min and its application to weapons allocation problems, volume 5. Springer Science & Business Media, 2012

    2012

  5. [13]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  6. [14]

    Efficient sharpness-aware minimization for improved training of neural networks

    Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural networks. arXiv preprint arXiv:2110.03141, 2021

    arXiv

  7. [15]

    What neural networks memorize and why: Discovering the long tail via influence estimation

    Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems , 33:2881–2891, 2020. 13

    2020

  8. [16]

    Sharpness-aware mini- mization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020

    2010 arXiv

  9. [17]

    short-length

    Shaopeng Fu, Liang Ding, and Di Wang. " short-length" adversarial training helps llms defend" long-length" jailbreak attacks: Theoretical and empirical evidence. arXiv preprint arXiv:2502.04204, 2025

    2025

  10. [18]

    Theoretical analysis of robust overfitting for wide dnns: An ntk approach

    Shaopeng Fu and Di Wang. Theoretical analysis of robust overfitting for wide dnns: An ntk approach. arXiv preprint arXiv:2310.06112, 2023

    2023 arXiv

  11. [19]

    Data shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pages 2242–2251. PMLR, 2019

    2019

  12. [20]

    Mixed-privacy forgetting in deep networks

    Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 792–801, 2021

    2021

  13. [21]

    Eternal sunshine of the spotless net: Selective forgetting in deep networks

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304–9312, 2020

    2020

  14. [22]

    Explaining black box predictions and unveiling data artifacts through influence functions

    Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint arXiv:2005.06676, 2020

    2005 arXiv

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  16. [24]

    Simplifying neural nets by discovering flat minima

    Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994

    1994

  17. [25]

    Flat minima

    Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1–42, 1997

    1997

  18. [26]

    Differentially private natural language models: Recent advances and future directions

    Lijie Hu, Ivan Habernal, Lei Shen, and Di Wang. Differentially private natural language models: Recent advances and future directions. arXiv preprint arXiv:2301.09112, 2023

    2023 arXiv

  19. [27]

    Editable concept bottleneck models

    Lijie Hu, Chenyang Ren, Zhengyu Hu, Hongbin Lin, Cheng-Long Wang, Hui Xiong, Jingfeng Zhang, and Di Wang. Editable concept bottleneck models. arXiv preprint arXiv:2405.15476, 2024

    2024 arXiv

  20. [28]

    Dissecting representation misalignment in contrastive learning via influence function

    Lijie Hu, Chenyang Ren, Huanyi Xie, Khouloud Saadi, Shu Yang, Zhen Tan, Jingfeng Zhang, and Di Wang. Dissecting representation misalignment in contrastive learning via influence function. arXiv preprint arXiv:2411.11667, 2024

    2024 arXiv

  21. [29]

    Privacy-preserving sparse generalized eigenvalue problem

    Lijie Hu, Zihang Xiang, Jiabin Liu, and Di Wang. Privacy-preserving sparse generalized eigenvalue problem. In International Conference on Artificial Intelligence and Statistics, pages 5052–5062. PMLR, 2023

    2023

  22. [30]

    Robust statistics

    Peter J Huber. Robust statistics. Wiley Series in Probability and Mathematical Statistics, 1981

    1981

  23. [31]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  24. [32]

    Towards efficient data valuation based on the shapley value

    Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pa...

    2019

  25. [33]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016

    2016 arXiv

  26. [34]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017. 14

    2017

  27. [35]

    Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks

    Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905–5914. PMLR, 2021

    2021

  28. [36]

    Beta shapley: a unified and noise-reduced data valuation framework for machine learning

    Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, 2022

    2022

  29. [37]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  30. [38]

    Taad: Time-varying adversarial anomaly detection in dynamic graphs

    Guanghua Liu, Jia Zhang, Peng Lv, Chenlong Wang, Huan Wang, and Di Wang. Taad: Time-varying adversarial anomaly detection in dynamic graphs. Information Processing & Management, 62(1):103912, 2025

    2025

  31. [39]

    Certified minimax unlearning with generalization rates and deletion capacity

    Jiaqi Liu, Jian Lou, Zhan Qin, and Kui Ren. Certified minimax unlearning with generalization rates and deletion capacity. Advances in Neural Information Processing Systems, 36, 2024

    2024

  32. [40]

    Tracing privacy leakage of language models to training data via adjusted influence functions

    Jinxin Liu and Zao Yang. Tracing privacy leakage of language models to training data via adjusted influence functions. arXiv preprint arXiv:2408.10468, 2024

    2024 arXiv

  33. [41]

    Towards efficient and scalable sharpness-aware minimization

    Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12360–12370, 2022

    2022

  34. [42]

    A unified approach to interpreting model predictions

    Scott Lundberg. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874, 2017

    2017 arXiv

  35. [43]

    Privacy-preserving low-rank adaptation against membership inference attacks for latent diffusion models

    Zihao Luo, Xilie Xu, Feng Liu, Yun Sing Koh, Di Wang, and Jingfeng Zhang. Privacy-preserving low-rank adaptation against membership inference attacks for latent diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39(6), pages 5883–5891, 2025

    2025

  36. [44]

    Towards deep learning models resistant to adversarial attacks

    Aleksander M ˛ adry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. stat, 1050(9), 2017

    2017

  37. [45]

    Exploring generalization in deep learning

    Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017

    2017

  38. [46]

    Trak: Attributing model behavior at scale

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023

    2023 arXiv

  39. [47]

    Estimating training data influence by tracing gradient descent

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920–19930, 2020

    2020

  40. [48]

    Generalized federated learning via sharpness aware minimization

    Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International conference on machine learning, pages 18250–18280. PMLR, 2022

    2022

  41. [49]

    Evaluating data influence in meta learning

    Chenyang Ren, Huanyi Xie, Shu Yang, Meng Ding, Lijie Hu, and Di Wang. Evaluating data influence in meta learning. arXiv preprint arXiv:2501.15963, 2025

    2025 arXiv

  42. [50]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016

    2016

  43. [51]

    Theoretical and practical perspectives on what influence functions do

    Andrea Schioppa, Katja Filippova, Ivan Titov, and Polina Zablotskaia. Theoretical and practical perspectives on what influence functions do. Advances in Neural Information Processing Systems, 36, 2024

    2024

  44. [52]

    Adversarial training for free!Advances in neural information processing systems, 32, 2019

    Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free!Advances in neural information processing systems, 32, 2019. 15

    2019

  45. [53]

    The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1–9, 2018

    2018

  46. [54]

    Generalized linear models in non-interactive local differential privacy with public data

    Di Wang, Lijie Hu, Huanyu Zhang, Marco Gaboardi, and Jinhui Xu. Generalized linear models in non-interactive local differential privacy with public data. Journal of Machine Learning Research, 24(132):1–57, 2023

    2023

  47. [55]

    Repairing without retraining: Avoiding disparate impact with counterfactual distributions

    Hao Wang, Berk Ustun, and Flavio Calmon. Repairing without retraining: Avoiding disparate impact with counterfactual distributions. In International Conference on Machine Learning, pages 6618–6627. PMLR, 2019

    2019

  48. [56]

    The shapley value

    Eyal Winter. The shapley value. Handbook of game theory with economic applications , 3:2025–2054, 2002

    2025

  49. [57]

    R-drop: Regularized dropout for neural networks

    Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. R-drop: Regularized dropout for neural networks. Advances in Neural Information Processing Systems, 34:10890–10905, 2021

    2021

  50. [58]

    Less: Selecting influential data for targeted instruction tuning

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024

    2024 arXiv

  51. [59]

    Practical differentially private and byzantine-resilient federated learning

    Zihang Xiang, Tianhao Wang, Wanyu Lin, and Di Wang. Practical differentially private and byzantine-resilient federated learning. Proceedings of the ACM on Management of Data , 1(2):1–26, 2023

    2023

  52. [60]

    Preserving node-level privacy in graph neural networks

    Zihang Xiang, Tianhao Wang, and Di Wang. Preserving node-level privacy in graph neural networks. In 2024 IEEE Symposium on Security and Privacy (SP), pages 4714–4732. IEEE, 2024

    2024

  53. [61]

    A theory to instruct differentially- private learning via clipping bias reduction

    Hanshen Xiao, Zihang Xiang, Di Wang, and Srinivas Devadas. A theory to instruct differentially- private learning via clipping bias reduction. In 2023 IEEE Symposium on Security and Privacy (SP), pages 2170–2189. IEEE, 2023

    2023

  54. [62]

    Data valuation using reinforcement learning

    Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pages 10842–10851. PMLR, 2020

    2020

  55. [63]

    Spectral norm regularization for improving the generaliz- ability of deep learning

    Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generaliz- ability of deep learning. arXiv preprint arXiv:1705.10941, 2017

    2017 arXiv

  56. [64]

    Towards user-level private reinforcement learning with human feedback

    Jiaming Zhang, Mingxi Lei, Meng Ding, Mengdi Li, Zihang Xiang, Difei Xu, Jinhui Xu, and Di Wang. Towards user-level private reinforcement learning with human feedback. arXiv preprint arXiv:2502.17515, 2025

    2025 arXiv

  57. [65]

    Mechanistic unveiling of transformer circuits: Self-influence as a key to model reasoning

    Lin Zhang, Lijie Hu, and Di Wang. Mechanistic unveiling of transformer circuits: Self-influence as a key to model reasoning. arXiv preprint arXiv:2502.09022, 2025

    2025 arXiv

  58. [66]

    Improved rates of differentially private nonconvex-strongly-concave minimax optimization

    Ruijia Zhang, Mingxi Lei, Meng Ding, Zihang Xiang, Jinhui Xu, and Di Wang. Improved rates of differentially private nonconvex-strongly-concave minimax optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39(21), pages 22524–22532, 2025

    2025

  59. [67]

    Penalizing gradient norm for efficiently improving generalization in deep learning

    Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning , pages 26982–26992. PMLR, 2022

    2022

  60. [68]

    Towards assessment of randomized mechanisms for certifying adversarial robustness

    Tianhang Zheng, D Wang, Baochun Li, and Jinhui Xu. Towards assessment of randomized mechanisms for certifying adversarial robustness. arXiv preprint arXiv:2005.07347, 2020. 16 A Omitted Proofs A.1 Evaluating Data Attribution in SAM via Hessian-based IF Theorem A.1. Consider th...

    2005 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.