Pith. sign in

REVIEW 18 cited by

Large Language Model Unlearning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.10683 v2 pith:J654KOPU submitted 2023-10-14 cs.CL cs.AIcs.LG

Yuanshun Yao , Xiaojun Xu , Yang Liu This is my paper
classification cs.CLcs.AIcs.LG
keywords unlearningonlyalignmentexamplesfirstharmfulhumanlanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We study how to perform unlearning, i.e. forgetting undesirable misbehaviors, on large language models (LLMs). We show at least three scenarios of aligning LLMs with human preferences can benefit from unlearning: (1) removing harmful responses, (2) erasing copyright-protected content as requested, and (3) reducing hallucinations. Unlearning, as an alignment technique, has three advantages. (1) It only requires negative (e.g. harmful) examples, which are much easier and cheaper to collect (e.g. via red teaming or user reporting) than positive (e.g. helpful and often human-written) examples required in RLHF (RL from human feedback). (2) It is computationally efficient. (3) It is especially effective when we know which training samples cause the misbehavior. To the best of our knowledge, our work is among the first to explore LLM unlearning. We are also among the first to formulate the settings, goals, and evaluations in LLM unlearning. We show that if practitioners only have limited resources, and therefore the priority is to stop generating undesirable outputs rather than to try to generate desirable outputs, unlearning is particularly appealing. Despite only having negative samples, our ablation study shows that unlearning can still achieve better alignment performance than RLHF with just 2% of its computational time.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Cross-modal unlearning transfer in vision-language models is asymmetric, architecture-dependent, and shallow under typographic attacks; influence-guided block selection reduces the measured gap.

  2. MPSelectTune: Prompt-type Selection for Fine-tuning improves Concept Unlearning in LLMs

    cs.LG 2026-07 conditional novelty 6.5 of 10

    After multi-prompt multi-task fine-tuning, further training on the worst concept-predicting prompt type yields stronger concept unlearning and higher main-task accuracy than uniform multi-prompt or recent baselines.

  3. Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A bilevel meta-training method aligns text-only backbone unlearning with full vision-language model forgetting, improving forget-retain trade-offs across two VLMs and two meme datasets.

  4. Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning

    cs.AI 2025-11 conditional novelty 6.0 of 10

    An MLLM unlearning method and benchmark that aim to erase targeted private facts while preserving image understanding.

  5. Leak@$k$: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding

    cs.LG 2025-11 reject novelty 6.0 of 10

    LLM unlearning methods that pass greedy-decoding benchmarks leak forgotten facts when the model is sampled repeatedly, and the new leak@k metric quantifies this.

  6. Towards Source-Free Machine Unlearning

    cs.LG 2025-08 conditional novelty 6.0 of 10

    A Hessian estimation procedure using only the forget set enables instance-level source-free unlearning with claimed theoretical error bounds.

  7. Invisible Watermarks, Visible Gains: Steering Machine Unlearning with Bi-Level Watermarking Design

    cs.CR 2025-08 conditional novelty 6.0 of 10

    Water4MU tunes an invisible watermark on data so that machine unlearning algorithms can remove requested images more effectively, beating prior methods on 'challenging forgets'.

  8. Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Unlearning in LLMs leaves detectable 'fingerprints' that let a simple classifier distinguish an unlearned model from its original, even on unrelated prompts.

  9. Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Models

    cs.CR 2025-06 conditional novelty 6.0 of 10

    PME detects memorized personal information in LLMs and edits the feed-forward layer weights so the model outputs a dummy value instead, reducing extraction attack success while preserving general model quality.

  10. LLM Unlearning Should Be Form-Independent

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Existing LLM unlearning is form-dependent; the new ORT benchmark measures this, and the training-free ROCR edit reduces it by redirecting concept representations.

  11. Soft Weighted Machine Unlearning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Soft-weighted unlearning replaces binary data removal with per-sample weights from a convex quadratic program, improving fairness and robustness gains while preserving utility.

  12. Module-Aware Parameter-Efficient Machine Unlearning on Transformers

    cs.LG 2025-08 conditional novelty 5.0 of 10

    MAPE-Unlearn uses Fisher-information-based scores and greedy search to select important heads and filters, then applies sparse unlearning updates, claiming improved efficacy-fidelity trade-offs on Transformers.

  13. SoK: Machine Unlearning for Large Language Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.

  14. Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning

    cs.CL 2025-06 reject novelty 5.0 of 10

    SU uses two assistant models trained on different data splits to score tokens, then unlearns only tokens whose scores diverge, claiming better retain-set utility with comparable forget quality.

  15. Kaleidoscope Gallery: Exploring Ethics and Generative AI Through Art

    cs.CY 2025-05 conditional novelty 5.0 of 10

    Ethics experts' definitions of five ethical theories, rendered as DALL-E 3 images and re-evaluated by the same experts, yield eight themes showing how morality, society, and learned associations shape and bias the mod...

  16. Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A position paper urging a shift from data-tracing to knowledge-tracing machine unlearning for foundation models, supported by a CLIP case study that shows current methods struggle to generalize.

  17. UCD: Unlearning in LLMs via Contrastive Decoding

    cs.CL 2025-06 conditional novelty 4.0 of 10

    UCD steers an LLM away from forget-set content at inference time by mixing in the difference between forget-tuned and retain-tuned small models.

  18. SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation

    cs.CR 2025-06 conditional novelty 3.0 of 10

    A systematization-of-knowledge survey that categorizes LLM privacy risks into training data, prompts, outputs, and agents, and reviews limitations of current mitigations.

Pith tools