arxiv: 2604.22076 · v1 · submitted 2026-04-23 · 💻 cs.LG · cs.CL

Recognition: unknown

PrivUn: Unveiling Latent Ripple Effects and Shallow Forgetting in Privacy Unlearning

Xiaoyi Chen , Haoyuan Wang , Siyuan Tang , Sijia Liu , Liya Su , Xiaofeng Wang , Haixu Tang

Authors on Pith no claims yet

Pith reviewed 2026-05-09 22:00 UTC · model grok-4.3

classification 💻 cs.LG cs.CL

keywords privacy unlearninglarge language modelsgradient ripple effectsshallow forgettingmachine unlearningprivacy attacksLLM memorization

0 comments

The pith

Privacy unlearning in LLMs propagates information through hidden gradient links and often stops at shallow layers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents PrivUn as a framework to test whether machine unlearning actually removes private data memorized by large language models. It evaluates robustness using three attack types—direct retrieval, recovery through in-context examples, and restoration via further fine-tuning—alongside metrics that track how much is forgotten, how associations spread, and how deep the forgetting reaches. The central results show that unlearning does not track ordinary semantic connections but instead follows latent gradient pathways that link private data to other content in unexpected ways. The work also finds that existing unlearning approaches typically affect only early layers, leaving private details intact deeper in the model. The authors respond by testing two practical adjustments: selecting unlearning examples based on gradient similarity and adding constraints that reach multiple layers at once.

Core claim

Unlearning exhibits gradient-driven ripple effects: unlike traditional forgetting which follows semantic relations, privacy unlearning propagates across latent gradient-based associations; and most methods suffer from shallow forgetting, failing to remove private information distributed across multiple deep model layers.

What carries the argument

PrivUn evaluation framework that combines three-tier attack scenarios (direct retrieval, in-context learning recovery, fine-tuning restoration) with quantitative measures of forgetting scores, association metrics, and forgetting depth.

If this is right

Association-aware selection of training examples using gradient similarity can reduce unwanted propagation during unlearning.
Adding representational constraints across multiple layers can shift unlearning from shallow to deep removal of private data.
Unlearning algorithms must explicitly track and interrupt gradient-based links rather than relying on semantic groupings.
Models may need to be trained or fine-tuned with explicit gradient-isolation steps to make later unlearning more reliable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Training procedures that minimize broad gradient overlap from the outset could reduce the need for later unlearning interventions.
The gradient-ripple pattern may appear in other data-removal tasks, such as removing biased or copyrighted content.
Monitoring gradient flows during unlearning could become a standard diagnostic step for privacy audits.

Load-bearing premise

The three-tier attacks and chosen metrics together give a complete picture of how well unlearning resists real privacy threats.

What would settle it

A demonstration that an unlearning procedure can eliminate private information from all model layers while showing no measurable gradient associations to other data points would contradict the ripple-effect and shallow-forgetting claims.

Figures

Figures reproduced from arXiv: 2604.22076 by Haixu Tang, Haoyuan Wang, Liya Su, Sijia Liu, Siyuan Tang, Xiaofeng Wang, Xiaoyi Chen.

**Figure 2.** Figure 2: Correlation between association metrics and forgetting scores for NPO (top row) and GA (bottom row). (a) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Figure 3: Cross-model CKA analysis among representative unlearning methods. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

**Figure 4.** Figure 4: Shallow forgetting analysis across unlearning [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

**Figure 5.** Figure 5: Subgraph of the Enron sender-recipient network [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

read the original abstract

Large language models (LLMs) often memorize private information during training, raising serious privacy concerns. While machine unlearning has emerged as a promising solution, its true effectiveness against privacy attacks remains unclear. To address this, we propose PrivUn, a new evaluation framework that systematically assesses unlearning robustness through three-tier attack scenarios: direct retrieval, in-context learning recovery, and fine-tuning restoration; combined with quantitative analysis using forgetting scores, association metrics, and forgetting depth assessment. Our study exposes significant weaknesses in current unlearning methods, revealing two key findings: 1) unlearning exhibits gradient-driven ripple effects: unlike traditional forgetting which follows semantic relations (e.g., knowledge graphs), privacy unlearning propagates across latent gradient-based associations; and 2) most methods suffer from shallow forgetting, failing to remove private information distributed across multiple deep model layers. To validate these insights, we explore two strategies: association-aware core-set selection that leverages gradient similarity, and multi-layer deep intervention through representational constraints. These strategies represent a paradigm shift from shallow forgetting to deep forgetting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

PrivUn gives a three-tier attack setup to test unlearning and flags gradient ripples plus shallow forgetting, but the depth metric looks like it may just track overall attack success rather than true layer retention.

read the letter

PrivUn is a framework that runs three attack scenarios—direct retrieval, in-context recovery, and fine-tuning restoration—on unlearned LLMs and then scores forgetting, associations, and depth. The two headline claims are that unlearning spreads through gradient-based links instead of semantic ones and that most existing methods only erase shallow traces while private data stays deeper in the model. The authors also sketch two fixes: picking a core set by gradient similarity and adding representational constraints across layers to push toward deeper forgetting. Those pieces are the actual new material; the attack tiers and the gradient-versus-semantic contrast extend earlier evaluation work in a straightforward way. The proposed interventions are practical and worth trying. The paper does a reasonable job framing the problem and naming concrete next steps. The soft spot is the forgetting-depth claim. If that metric comes only from end-to-end attack success rates across the three scenarios without layer-wise probes or ablations that separate gradient similarity from leftover semantic overlap, it cannot cleanly show that private information sits in deep layers rather than just that the model remains attackable. The abstract and stress-test note both leave this gap visible, and the quantitative tables would need to demonstrate that the depth score is not simply another aggregate success number. The work is aimed at researchers who build or audit unlearning pipelines for LLMs. A reader who needs fresh attack templates or ideas for multi-layer fixes could pull useful pieces from it, though they would still want to re-run the controls themselves. The ideas are timely and the framing is coherent, so the paper deserves a serious referee who can press on the metric validation and the separation of gradient from semantic effects. I would send it out for review rather than desk-reject it.

Referee Report

2 major / 2 minor

Summary. The paper introduces PrivUn, a framework to evaluate privacy unlearning in LLMs via three-tier attack scenarios (direct retrieval, in-context learning recovery, and fine-tuning restoration) combined with forgetting scores, association metrics, and forgetting depth assessment. It reports two empirical findings: unlearning propagates through gradient-driven ripple effects (contrasted with semantic/knowledge-graph forgetting) and that existing methods exhibit shallow forgetting by failing to erase private information from deep layers. The work also proposes association-aware core-set selection and multi-layer representational constraints as mitigation strategies.

Significance. If the empirical claims hold after verification, the work would be moderately significant for LLM privacy research by providing a structured attack-based evaluation and highlighting that current unlearning may leave residual private encodings in deeper layers. The proposed mitigation strategies could inform more robust unlearning designs, though the novelty of the 'gradient-driven ripple' framing depends on how cleanly it separates from existing gradient-based analyses in the literature.

major comments (2)

[Evaluation Framework (metrics description) and Results] The forgetting depth metric is presented as evidence for shallow forgetting (private information distributed across deep layers), yet it appears derived solely from end-to-end attack success rates across the three scenarios without layer-wise probing, representation ablations, or per-layer retention measurements. This is load-bearing for the second key finding and does not distinguish shallow association weakening from intact deep encodings.
[Results and Analysis (association metrics)] The claim of gradient-driven ripple effects (vs. semantic relations) relies on association metrics, but the manuscript does not report explicit controls or ablations to rule out residual semantic overlap as a confounder. Without such controls, the contrast with knowledge-graph-style forgetting cannot be isolated.

minor comments (2)

[Abstract] The abstract and introduction would benefit from including at least one key quantitative result (e.g., forgetting score deltas or depth values) to ground the two findings.
[§3] Notation for the three attack scenarios and the quantitative metrics should be defined more formally (e.g., with equations) to improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed comments on our manuscript. We address each major comment point by point below, providing clarifications and committing to revisions that strengthen the empirical support for our claims.

read point-by-point responses

Referee: [Evaluation Framework (metrics description) and Results] The forgetting depth metric is presented as evidence for shallow forgetting (private information distributed across deep layers), yet it appears derived solely from end-to-end attack success rates across the three scenarios without layer-wise probing, representation ablations, or per-layer retention measurements. This is load-bearing for the second key finding and does not distinguish shallow association weakening from intact deep encodings.

Authors: We appreciate this observation on the forgetting depth metric. The metric is computed from differential attack success rates across the three tiers, which indirectly reflect layer-wise retention through the progressive difficulty of recovery. However, we agree that direct layer-wise evidence would better isolate shallow weakening from preserved deep encodings. In the revision we will add per-layer probing via activation similarity measurements and targeted representation ablations (e.g., freezing or intervening on specific layers post-unlearning) to provide explicit support for the shallow-forgetting claim. revision: yes
Referee: [Results and Analysis (association metrics)] The claim of gradient-driven ripple effects (vs. semantic relations) relies on association metrics, but the manuscript does not report explicit controls or ablations to rule out residual semantic overlap as a confounder. Without such controls, the contrast with knowledge-graph-style forgetting cannot be isolated.

Authors: We acknowledge the need for stronger isolation of gradient-driven effects. Our association metrics are constructed from gradient similarity rather than semantic embeddings, and we already include baseline comparisons against semantic similarity. To further rule out residual semantic overlap as a confounder, we will add explicit ablations in the revision, including controls with semantically orthogonal instance sets and direct propagation comparisons against knowledge-graph baselines. These additions will make the distinction between gradient-based and semantic forgetting more rigorous. revision: yes

Circularity Check

0 steps flagged

Empirical evaluation framework is self-contained with no circular derivation

full rationale

The paper defines PrivUn as an evaluation framework consisting of three-tier attack scenarios (direct retrieval, in-context recovery, fine-tuning restoration) and associated quantitative metrics (forgetting scores, association metrics, forgetting depth). The central claims—gradient-driven ripple effects distinct from semantic forgetting and the prevalence of shallow forgetting—are presented as empirical observations obtained by applying this framework to existing unlearning methods. No load-bearing step reduces a prediction or uniqueness claim to a fitted parameter, self-citation, or definitional tautology; the metrics are used to measure outcomes rather than presuppose them. The derivation chain therefore remains independent of its inputs and does not exhibit any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

The central claims depend on the assumption that the three-tier attacks and depth metrics reliably expose unlearning weaknesses; no free parameters or new physical entities are introduced.

axioms (1)

domain assumption The three-tier attack scenarios (direct retrieval, in-context learning recovery, fine-tuning restoration) combined with forgetting scores and depth assessment accurately reflect privacy unlearning robustness.
The entire evaluation framework and subsequent findings rest on this premise about attack realism and metric validity.

invented entities (2)

gradient-driven ripple effects no independent evidence
purpose: To explain how unlearning propagates through latent model associations rather than semantic ones.
New descriptive concept introduced to interpret the observed propagation behavior.
shallow forgetting no independent evidence
purpose: To characterize the limitation where private information remains in deeper model layers.
Descriptive term for the failure mode identified in current methods.

pith-pipeline@v0.9.0 · 5505 in / 1352 out tokens · 36608 ms · 2026-05-09T22:00:07.402431+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

48 extracted references · 21 canonical work pages · 3 internal anchors

[1]

2024 , isbn =

Chen, Xiaoyi and Tang, Siyuan and Zhu, Rui and Yan, Shijun and Jin, Lei and Wang, Zihao and Su, Liya and Zhang, Zhikun and Wang, XiaoFeng and Tang, Haixu , title =. 2024 , isbn =. doi:10.1145/3658644.3690325 , booktitle =

work page doi:10.1145/3658644.3690325 2024
[2]

Proceedings of the International Conference on Learning Representations (ICLR) , year=

Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=
[3]

The Twelfth International Conference on Learning Representations , year=

Detecting Pretraining Data from Large Language Models , author=. The Twelfth International Conference on Learning Representations , year=
[4]

Privacy issues in large language models: A survey

Privacy issues in large language models: A survey , author=. arXiv preprint arXiv:2312.06717 , year=

work page arXiv
[5]

2024 , eprint=

MUSE: Machine Unlearning Six-Way Evaluation for Language Models , author=. 2024 , eprint=

2024
[6]

Journal of cybersecurity , volume=

Forgetting personal data and revoking consent under the GDPR: Challenges and proposed solutions , author=. Journal of cybersecurity , volume=. 2018 , publisher=

2018
[7]

Overview of the Act on the Protection of Personal Information , author=. Eur. Data Prot. L. Rev. , volume=. 2019 , publisher=

2019
[8]

Advances in neural information processing systems , volume=

Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=
[9]

30th USENIX Security Symposium (USENIX Security 21) , pages=

Extracting training data from large language models , author=. 30th USENIX Security Symposium (USENIX Security 21) , pages=
[10]

2023 IEEE Symposium on Security and Privacy (SP) , pages=

Analyzing leakage of personally identifiable information in language models , author=. 2023 IEEE Symposium on Security and Privacy (SP) , pages=. 2023 , organization=

2023
[11]

Algorithmic Disgorgement: Destruction of Artificial Intelligence Models as the FTC's Newest Enforcement Tool for Bad Data , author=. Rich. JL & Tech. , volume=. 2022 , publisher=

2022
[12]

Do llms really forget? evaluating unlearning with knowledge correlation and confidence awareness.arXiv preprint arXiv:2506.05735, 2025

Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness , author=. arXiv preprint arXiv:2506.05735 , year=

work page arXiv
[13]

arXiv preprint arXiv:2410.15153 , year=

Evaluating deep unlearning in large language models , author=. arXiv preprint arXiv:2410.15153 , year=

work page arXiv
[14]

Transactions of the Association for Computational Linguistics , volume=

Evaluating the ripple effects of knowledge editing in language models , author=. Transactions of the Association for Computational Linguistics , volume=. 2024 , publisher=

2024
[15]

The Thirteenth International Conference on Learning Representations , year=

A Closer Look at Machine Unlearning for Large Language Models , author=. The Thirteenth International Conference on Learning Representations , year=
[16]

arXiv preprint arXiv:2205.12628 , year=

Are large pre-trained language models leaking your personal information? , author=. arXiv preprint arXiv:2205.12628 , year=

work page arXiv
[17]

Quantifying Memorization Across Neural Language Models

Quantifying memorization across neural language models , author=. arXiv preprint arXiv:2202.07646 , year=

work page internal anchor Pith review arXiv
[18]

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica

Jang, Joel and Yoon, Dongkeun and Yang, Sohee and Cha, Sungmin and Lee, Moontae and Logeswaran, Lajanugen and Seo, Minjoon. Knowledge Unlearning for Mitigating Privacy Risks in Language Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/2023.acl-long.805

work page doi:10.18653/v1/2023.acl-long.805 2023
[19]

arXiv preprint arXiv:2405.16720 , year=

Large scale knowledge washing , author=. arXiv preprint arXiv:2405.16720 , year=

work page arXiv
[20]

arXiv preprint arXiv:2407.01920 , year=

To forget or not? towards practical knowledge unlearning for large language models , author=. arXiv preprint arXiv:2407.01920 , year=

work page arXiv
[21]

arXiv preprint arXiv:2404.05868 , year=

Negative preference optimization: From catastrophic collapse to effective unlearning , author=. arXiv preprint arXiv:2404.05868 , year=

work page arXiv
[22]

ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models , year=

TOFU: A Task of Fictitious Unlearning for LLMs , author=. ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models , year=

2024
[23]

The Eleventh International Conference on Learning Representations , year=

Editing models with task arithmetic , author=. The Eleventh International Conference on Learning Representations , year=
[24]

2004 , isbn =

Klimt, Bryan and Yang, Yiming , title =. 2004 , isbn =. doi:10.1007/978-3-540-30115-8_22 , booktitle =

work page doi:10.1007/978-3-540-30115-8_22 2004
[25]

Chalkidis, Ilias and Androutsopoulos, Ion and Aletras, Nikolaos , journal=
[26]

The Llama 3 Herd of Models

The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

work page internal anchor Pith review Pith/arXiv arXiv
[27]

International conference on machine learning , pages=

Similarity of neural network representations revisited , author=. International conference on machine learning , pages=. 2019 , organization=

2019
[28]

arXiv preprint arXiv:2407.00106 , year=

Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai , author=. arXiv preprint arXiv:2407.00106 , year=

work page arXiv
[29]

interpreting GPT: the logit lens , author =
[30]

Backward Lens: Projecting Language Model Gradients into the Vocabulary Space

Katz, Shahar and Belinkov, Yonatan and Geva, Mor and Wolf, Lior. Backward Lens: Projecting Language Model Gradients into the Vocabulary Space. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.142

work page doi:10.18653/v1/2024.emnlp-main.142 2024
[31]

OpenAI blog , volume=

Language models are unsupervised multitask learners , author=. OpenAI blog , volume=
[32]

arXiv preprint arXiv:2406.13356 , year=

Jogging the Memory of Unlearned Model Through Targeted Relearning Attack , author=. arXiv preprint arXiv:2406.13356 , year=

work page arXiv
[33]

The Privacy Onion Effect: Memorization is Relative , url =

Carlini, Nicholas and Jagielski, Matthew and Zhang, Chiyuan and Papernot, Nicolas and Terzis, Andreas and Tramer, Florian , booktitle =. The Privacy Onion Effect: Memorization is Relative , url =
[34]

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

The pile: An 800gb dataset of diverse text for language modeling , author=. arXiv preprint arXiv:2101.00027 , year=

work page internal anchor Pith review arXiv
[35]

A Closer Look at Machine Unlearning for Large Language Models , author=
[36]

2024 , eprint=

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning , author=. 2024 , eprint=

2024
[37]

Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models

Kassem, Aly and Mahmoud, Omar and Saad, Sherif. Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.265

work page doi:10.18653/v1/2023.emnlp-main.265 2023
[38]

Machine Unlearning of Personally Identifiable Information in Large Language Models

Parii, Dan and van Osch, Thomas and Sun, Chang. Machine Unlearning of Personally Identifiable Information in Large Language Models. Proceedings of the Natural Legal Language Processing Workshop 2025. 2025. doi:10.18653/v1/2025.nllp-1.6

work page doi:10.18653/v1/2025.nllp-1.6 2025
[39]

arXiv preprint arXiv:2601.15595 , year=

Data-Free Privacy-Preserving for LLMs via Model Inversion and Selective Unlearning , author=. arXiv preprint arXiv:2601.15595 , year=

work page arXiv
[40]

arXiv preprint arXiv:2512.18035 , year=

Towards Benchmarking Privacy Vulnerabilities in Selective Forgetting with Large Language Models , author=. arXiv preprint arXiv:2512.18035 , year=

work page arXiv
[41]

2025 , eprint=

Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities , author=. 2025 , eprint=

2025
[42]

2025 , eprint=

An Adversarial Perspective on Machine Unlearning for AI Safety , author=. 2025 , eprint=

2025
[43]

2025 , eprint=

Catastrophic Failure of LLM Unlearning via Quantization , author=. 2025 , eprint=

2025
[44]

2024 , eprint=

Eight Methods to Evaluate Robust Unlearning in LLMs , author=. 2024 , eprint=

2024
[45]

2025 , eprint=

Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond , author=. 2025 , eprint=

2025
[46]

Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning , author=
[47]

Advances in neural information processing systems , volume=

Neural tangent kernel: Convergence and generalization in neural networks , author=. Advances in neural information processing systems , volume=
[48]

arXiv preprint arXiv:2407.15549 , year=

Latent adversarial training improves robustness to persistent harmful behaviors in llms , author=. arXiv preprint arXiv:2407.15549 , year=

work page arXiv