Pith. sign in

REVIEW 18 cited by

Eight Methods to Evaluate Robust Unlearning in LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.16835 v1 pith:TDVPYQ7V submitted 2024-02-26 cs.CL

Aengus Lynch , Phillip Guo , Aidan Ewart , Stephen Casper , Dylan Hadfield-Menell This is my paper
classification cs.CL
keywords unlearningmodelcomprehensiveeldanharryknowledgellmsmethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Machine unlearning can be useful for removing harmful capabilities and memorized text from large language models (LLMs), but there are not yet standardized methods for rigorously evaluating it. In this paper, we first survey techniques and limitations of existing unlearning evaluations. Second, we apply a comprehensive set of tests for the robustness and competitiveness of unlearning in the "Who's Harry Potter" (WHP) model from Eldan and Russinovich (2023). While WHP's unlearning generalizes well when evaluated with the "Familiarity" metric from Eldan and Russinovich, we find i) higher-than-baseline amounts of knowledge can reliably be extracted, ii) WHP performs on par with the original model on Harry Potter Q&A tasks, iii) it represents latent knowledge comparably to the original model, and iv) there is collateral unlearning in related domains. Overall, our results highlight the importance of comprehensive unlearning evaluation that avoids ad-hoc metrics.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Suppression Sticks, Locality Is Fragile: A Closed-Loop Target-and-Control Audit of Task-Vector Negation in VLA Policies

    cs.RO 2026-08 conditional novelty 7.0 of 10

    Subtracting a task vector from a vision-language-action robot policy suppresses the target skill but not its collateral damage: only five of ten LIBERO-Goal skills separate cleanly, and held-out control retention aver...

  2. Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Margin Calibration, a non-saturating margin-anchored LoRA polish, crosses the margin cliff and cuts post-attack relearn recovery on all 97 populated cells in the paper's stress matrix.

  3. Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

    cs.LG 2026-07 accept novelty 7.0 of 10

    SUITE defines the forget-retain boundary at semantic, syntactic and lexical levels; training on it plus JensUn++ yields near-complete forgetting with minimal retain and utility loss.

  4. Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech

    cs.SD 2025-07 reject novelty 7.0 of 10

    Teacher-Guided Unlearning makes a zero-shot text-to-speech model ignore the audio prompt of designated speakers, generating a randomized voice instead, while keeping speech quality for other speakers.

  5. Understanding Machine Unlearning Through the Lens of Mode Connectivity

    cs.LG 2026-07 unverdicted novelty 6.0 of 10

    Unlearned models usually connect to their originals by smooth low-loss paths, and the smoothness of that path can predict how hard the unlearning task was.

  6. RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositories

    cs.AI 2025-12 conditional novelty 6.0 of 10

    RippleBench automatically generates questions at increasing semantic distance from unlearned topics and shows all eight tested unlearning methods degrade accuracy that recovers only slowly with distance.

  7. What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests

    cs.CL 2025-07 conditional novelty 6.0 of 10

    WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.

  8. Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A new method, Partial Model Collapse, iteratively fine-tunes an LLM on its own self-generated responses to conditionally collapse its output distribution on forget queries, removing private answers without the true la...

  9. Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Unlearning in LLMs leaves detectable 'fingerprints' that let a simple classifier distinguish an unlearned model from its original, even on unrelated prompts.

  10. LLM Unlearning Should Be Form-Independent

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Existing LLM unlearning is form-dependent; the new ORT benchmark measures this, and the training-free ROCR edit reduces it by redirecting concept representations.

  11. Unlearning's Blind Spots: Over-Unlearning and Prototypical Relearning Attack

    cs.LG 2025-06 reject novelty 6.0 of 10

    The paper proposes an over-unlearning metric and a prototype-based relearning attack for class-level machine unlearning, together with a defense objective called Spotter.

  12. Existing Large Language Model Unlearning Evaluations Are Inconclusive

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Existing LLM unlearning evaluations are inconclusive: they can inject new information, depend heavily on task format, and rely on spurious correlations.

  13. Module-Aware Parameter-Efficient Machine Unlearning on Transformers

    cs.LG 2025-08 conditional novelty 5.0 of 10

    MAPE-Unlearn uses Fisher-information-based scores and greedy search to select important heads and filters, then applies sparse unlearning updates, claiming improved efficacy-fidelity trade-offs on Transformers.

  14. Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    An LLM unlearning method that projects hidden states so harmful information is irreversibly removed while useful knowledge is preserved.

  15. Prompt Attacks Reveal Superficial Knowledge Removal in Unlearning Methods

    cs.CR 2025-06 conditional novelty 5.0 of 10

    Prepending a Hindi filler paragraph to WMDP-bio questions restores 57.3% accuracy in ELM-unlearned models, showing the unlearning is superficial output suppression rather than true knowledge removal.

  16. SoK: Machine Unlearning for Large Language Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.

  17. BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap

    cs.LG 2025-05 conditional novelty 5.0 of 10

    BLUR adds forget-retain overlapping combined queries and relearning data to four unlearning benchmarks, and shows existing unlearning methods degrade on them.

  18. Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models

    cs.CR 2025-06 reject novelty 4.0 of 10

    Step-by-step reasoning prompts can recover purportedly erased facts from unlearned LLMs, but the paper's quantitative evidence is internally inconsistent.

Pith tools