Pith. sign in

REVIEW 3 major objections 5 minor 123 references

SoK: Machine Unlearning for Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gradient-ascent unlearning suppresses knowledge rather than erases it, this SoK argues.

desk verdict A useful intention-based taxonomy and a careful evaluation critique, but the headline claim that GA methods suppress rather than erase rests on recovery probes the paper itself admits can be unreliable. read the letter →

arxiv 2506.09227 v1 pith:V5V3KO5H submitted 2025-06-10 cs.LG cs.CR

Jie Ren , Yue Xing , Yingqian Cui , Charu C. Aggarwal , Hui Liu This is my paper
classification cs.LGcs.CR
keywords machineunlearninglargelanguagemodelsgradientascentknowledgeremovalbehavioralsuppressionevaluationadversarialattacksrighttobeforgotten
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This systematization of knowledge paper argues that the field of LLM unlearning has been conflating two different goals: removing knowledge from the model's internals versus suppressing its expression in outputs. The authors establish that the most common removal-intended technique, gradient ascent and its variants, often fails at true removal and instead behaves like suppression: the model recognizes the forgetting data as a trigger and simulates ignorance. Evidence comes from adversarial prompts that recover supposedly forgotten answers and from fine-tuning that re-memorizes the data, alongside representation-space studies showing the model still distinguishes the forgetting cluster. The paper then uses this removal-versus-suppression distinction to organize a taxonomy of methods, review evaluation practices, and identify practical gaps such as sequential unlearning and utility preservation.

What carries the argument

The organizing device is an intention-based taxonomy that splits unlearning methods into removal-intended (gradient ascent and its variants, task arithmetic) and suppression-intended (fine-grained probability, rejection fine-tuning, input-space interventions, representation re-steering, logits difference, retrieval-based blocking). The explanation for why GA underperforms its intention rests on how the GA loss operates: it is a reversed next-token prediction loss that increases the model's error whenever content from the forgetting set appears in the prompt. This makes the forgetting data function as a trigger, so the model learns to behave as though it knows nothing rather than actually deleting the knowledge; representation-space findings that the forgetting cluster remains separated and dominates mixed prompts support this mechanism.

What would settle it

A controlled experiment that, after unlearning, fails to recover the target knowledge under any adversarial or fine-tuning probe—while the same probes are shown to work on models that still contain the data—and additionally shows internal representations revert to the pre-training state would falsify the claim that GA only suppresses.

Watch

Extended reading notes

Core claim

The paper's central claim is that GA-based unlearning, despite being designed to erase the training influence of forgetting data, does not actually remove internal knowledge; it treats the forgetting data as a signal that triggers unlearning behavior. When the signal appears, the model increases its output error and simulates ignorance, which is the same behavior produced by suppression-intended methods. The main evidence is that unlearned models still produce the target knowledge under adversarial prompting (for example, partial prompts or optimized suffixes), and that benign fine-tuning can bring the supposedly forgotten information back. The authors also describe representation-space observations in which the forgetting data remains a distinct cluster, indicating the model can still identify it. The paper concludes that current removal-intended methods suppress rather than erase, and that genuinely verifying removal would require new theoretical and empirical tools.

Load-bearing premise

The claim that unlearning only suppresses knowledge rests on assuming that attacks which recover the information after unlearning are probing genuine retention, not injecting the answer into the model.

Editorial extensions

If this is right

  • Evaluation of unlearning must treat adversarial prompting and post-unlearning fine-tuning as standard robustness tests, not optional extras, because they expose whether knowledge was truly removed.
  • Benchmarks should be redesigned to distinguish removal from suppression, using representation-level metrics and recovery probes rather than only greedy-decoding accuracy.
  • For many real-world deployments, behavioral suppression may be sufficient, so suppression-intended methods can be a pragmatic and even preferable choice.
  • Sequential unlearning and utility loss remain unsolved; retrieval-augmented approaches that update an index instead of parameters offer a path around repeated parameter edits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If GA unlearning is fundamentally suppression, then a concrete testable prediction follows: removing the identifiable trigger from a prompt (for example, paraphrasing forget-set content beyond syntactic similarity) should make the model answer as if unlearned, something current benchmarks rarely check.
  • The same logic implies that legal 'right to be forgotten' compliance may be impossible to demonstrate verifiably for LLMs, since no audit can prove the absence of internal traces without also ruling out attack-induced artifacts.
  • The paper's own mention that soft token attacks recover knowledge even from models that never saw the data suggests a methodological safeguard: recovery audits should always include never-seen control data to distinguish retention from attack artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper is a SoK survey of machine unlearning for large language models. It proposes a taxonomy based on the intention of unlearning methods, dividing them into removal-intended methods (GA-based variants and task arithmetic) and suppression-intended methods (operating on input, hidden representations, or output space). The paper argues that GA-based methods, despite their removal intention, functionally suppress rather than erase knowledge, citing adversarial-prompt recovery and fine-tuning recovery studies, and it offers a mechanism based on 'unlearning signals' and syntactic similarity. It further reviews evaluation metrics and benchmarks and discusses practical gaps such as sequential unlearning, scalability, poisoning risks, and interference with alignment.

Significance. The proposed intention-based taxonomy is a useful organizational lens, and the survey covers a broad and current literature, including evaluation limitations (e.g., Section 8.3 on greedy sampling and core-set effects) and deployment gaps (Section 9) that are not always treated together. The review of GA variants and suppression approaches is comprehensive, and the explicit listing of metrics and benchmarks with citations is valuable. If the Section 7 claim were fully established, it would reframe unlearning evaluation away from task-level metrics toward representation-level or control-based auditing. However, the central empirical claim is currently supported mainly by recovery probes whose reliability the paper itself questions, and the mechanism in Section 7.2 relies on a single same-group preprint. The significance is therefore conditional on the authors addressing these evidence gaps in revision.

major comments (3)
  1. [Section 7.1, Takeaway] The paper's central claim—that GA-based unlearning suppresses rather than erases internal representations—is inferred from two strands of evidence: adversarial recovery (Section 7.1.1) and fine-tuning recovery (Section 7.1.2). The paper itself states, citing Chen et al. [94], that soft token attacks cannot reliably audit unlearning because they elicit target knowledge even from models that never saw the data. This caveat is not applied to the partial-subsequence attacks of Schwarzschild et al. [84], the GCG attacks of Yuan et al. [85], or the fine-tuning recovery results of Hu et al. [87] and Deeb & Roger [88]. The authors should either report control experiments showing that these probes do not recover forget-set knowledge from a model never trained on the forget set, or explicitly restrict the claim to probes that are known to be valid. As written, the main takeaway overstates what the cited evidence establishes.
  2. [Section 7.2] The 'unlearning signal' mechanism—the claim that GA-unlearned models treat forgetting data as a trigger to suppress generation—is presented as a 'new understanding' and then summarized in a categorical Takeaway. The representation-space evidence for this mechanism comes from Ren et al. [62], a preprint with the same first author, with support from the position paper Thaker et al. [82]; the related poisoning discussion in Section 9 similarly relies on Ren et al. [99]. The paper should explicitly flag these as single-source, preliminary interpretations, state whether any independent replication or contradiction exists, and describe what evidence would falsify the mechanism. Without that, Section 7.2 moves from a plausible research hypothesis to an established conclusion too quickly.
  3. [Sections 3–4] Eq. (2) defines unlearning as approximating the retrained model's output distribution p*, while Section 4 defines removal-intended unlearning as 'genuinely eliminat[ing] the model's internal knowledge or training trace.' The paper never defines what counts as an internal knowledge trace or gives a criterion for removal that is distinct from behavioral suppression. Without such a criterion, the taxonomy is a classification of stated intentions rather than a verifiable distinction, and the only operational test offered—recovery probes—is precisely the probe whose reliability is questioned in Section 7.1.1. The paper should either formalize the removal/suppression distinction (e.g., invariance under a class of input transformations, or representation-level criteria) or explicitly frame the taxonomy as an intention-based organization of methods rather than an empirically testable dichotomy.
minor comments (5)
  1. [Section 1] The phrase 'read-world settings' should be 'real-world settings'.
  2. [Section 3] The sentence after Eq. (2), 'the task of unlearning is to find a p_u that can that can approximate', contains a duplicated 'that can' and should be corrected.
  3. [Section 6.4.2] The word 'forgertting' should be 'forgetting' in 'forgertting data'.
  4. [Section 7.2] The text 'this also explains the mechanist of unlearning signal' should read 'mechanism of the unlearning signal', and 'haven't fine a better way' should be 'haven't found a better way'.
  5. [Table 1] The entry 'Thie type of methods' should be 'This type of method' or 'These types of methods', and 'play-and-plug' should be 'plug-and-play'.

Circularity Check

1 steps flagged · score 4.0 of 10

The Section 7.2 'unlearning signal' mechanism is imported from same-group preprint [62] and then restated as the Takeaway; the broader superficiality claim still has independent external evidence.

  1. self citation load bearing [Section 7.2, paragraph 'From over-generalization to unlearning signals' and Takeaway]
    "Based on this observation, the unlearning mechanism of GA is further explained as the unlearning signal by [62]. ... As the authors conclude, this implies that, instead of removing the forgetting data, GA-unlearned models treat it as a unlearning signal to suppress the generation."

    The SoK's central mechanistic conclusion—'GA-based unlearning does not truly remove knowledge; instead, it treats the forgetting data as a signal to trigger unlearning behavior'—is the same claim as [62]'s conclusion, and [62] is a preprint by the present paper's own group (first author Ren; co-authors Xing and H. Liu also overlap). The paper presents [62]'s representation-space observation as the evidence for this mechanism without deriving or independently reproducing it; the 'new understanding' section therefore reduces to a self-citation loop. External attack/fine-tuning results [84,85,87,88] support the weaker claim that GA is superficial, but the specific 'unlearning signal' mechanism is not independently established.

full rationale

Most of this SoK is a taxonomy and literature review; the removal/suppression classification is definitional and not circular. The main interpretive claim that GA-based unlearning is superficial is supported by independent external studies (adversarial attacks and fine-tuning recovery [84,85,87,88,90]). However, the paper's strongest mechanistic formulation in Section 7.2—that GA detects an 'unlearning signal' and only simulates ignorance—is attributed to [62], a same-group preprint, and then elevated to a Takeaway. Because the paper does not reproduce [62]'s evidence or cite independent replication, this is load-bearing self-citation. The paper also cites [99], another same-group preprint, for the poisoning-risk gap in Section 9, but that is a secondary practical point. Finally, the paper's own admission in Section 7.1.1 that soft token attacks 'cannot reliably audit unlearning' [94] weakens the adversarial-recovery evidence, but that is a correctness/validity concern rather than a circularity loop. Overall, the central claim retains independent empirical content, so the circularity score is moderate.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No new physical or mathematical entities and no fitted parameters are introduced. The two listed axioms are non-empirical premises the survey relies on; neither is proven in the paper.

assumptions (2)
  • domain assumption Recovery of forgotten knowledge through adversarial prompts or fine-tuning demonstrates that the knowledge was never truly removed.
    Invoked throughout Section 7.1, where results from [84,85,86,87,88] are treated as evidence that GA methods suppress rather than erase. The paper itself cites [94] showing soft token attacks can induce outputs from models that never saw the data, which weakens this assumption and is not resolved.
  • domain assumption Conclusions from TOFU, MUSE, WMDP, and RWKU benchmarks generalize to practical LLM deployments.
    Section 8.2 presents these benchmarks as representative, and Sections 7 and 9 generalize from them to claims about unlearning in production settings. The paper notes differences in benchmark design but does not establish external validity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoK: Machine Unlearning for Large Language Models." pith.science (2026). https://pith.science/paper/V5V3KO5H

@misc{pith2026250609227,
  author       = {Pith},
  title        = {Pith review of: SoK: Machine Unlearning for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V5V3KO5H}},
  note         = {Machine review of arXiv:2506.09227}
}
read the original abstract

Large language model (LLM) unlearning has become a critical topic in machine learning, aiming to eliminate the influence of specific training data or knowledge without retraining the model from scratch. A variety of techniques have been proposed, including Gradient Ascent, model editing, and re-steering hidden representations. While existing surveys often organize these methods by their technical characteristics, such classifications tend to overlook a more fundamental dimension: the underlying intention of unlearning--whether it seeks to truly remove internal knowledge or merely suppress its behavioral effects. In this SoK paper, we propose a new taxonomy based on this intention-oriented perspective. Building on this taxonomy, we make three key contributions. First, we revisit recent findings suggesting that many removal methods may functionally behave like suppression, and explore whether true removal is necessary or achievable. Second, we survey existing evaluation strategies, identify limitations in current metrics and benchmarks, and suggest directions for developing more reliable and intention-aligned evaluations. Third, we highlight practical challenges--such as scalability and support for sequential unlearning--that currently hinder the broader deployment of unlearning methods. In summary, this work offers a comprehensive framework for understanding and advancing unlearning in generative AI, aiming to support future research and guide policy decisions around data removal and privacy.

Figures

Figures reproduced from arXiv: 2506.09227 by the authors.

Figure 1
Figure 1. Workflow and existing problems of unlearning [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

123 extracted references · 16 canonical work pages

  1. [62]

    A general framework to enhance fine-tuning-based llm unlearning,

    J. Ren, Z. Dai, X. Tang, H. Liu, J. Zeng, Z. Li, R. Goutam, S. Wang, Y . Xing, Q. He, and H. Liu, “A general framework to enhance fine-tuning-based llm unlearning,”arXiv preprint arXiv:2502.17823, 2025

    arXiv 2025

  2. [94]

    Soft Token Attacks Cannot Reliably Audit Unlearning in Large Language Models

    H. Chen, S. Szyller, W. Xu, and N. Himayat, “Soft token attacks cannot reliably audit unlearning in large language models,”arXiv preprint arXiv:2502.15836, 2025

    work page Pith review arXiv 2025

  3. [84]

    Rethinking llm memorization through the lens of adversarial com- pression,

    A. Schwarzschild, Z. Feng, P. Maini, Z. Lipton, and J. Z. Kolter, “Rethinking llm memorization through the lens of adversarial com- pression,”Advances in Neural Information Processing Systems, vol. 37, pp. 56 244–56 267, 2024

    2024

  4. [85]

    Towards robust knowledge unlearning: An adversarial framework for assess- ing and improving unlearning robustness in large language models,

    H. Yuan, Z. Jin, P. Cao, Y . Chen, K. Liu, and J. Zhao, “Towards robust knowledge unlearning: An adversarial framework for assess- ing and improving unlearning robustness in large language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 24, 2025, pp. 25 769–25 777

    2025

  5. [87]

    Jogging the memory of unlearned models through targeted relearning attacks,

    S. Hu, Y . Fu, S. Wu, and V . Smith, “Jogging the memory of unlearned models through targeted relearning attacks,” inICML 2024 Workshop on Foundation Models in the Wild, 2024

    2024

  6. [88]

    Do unlearning methods remove information from language model weights?

    A. Deeb and F. Roger, “Do unlearning methods remove information from language model weights?”arXiv preprint arXiv:2410.08827, 2024

    arXiv 2024

  7. [82]

    Po- sition: Llm unlearning benchmarks are weak measures of progress,

    P. Thaker, S. Hu, N. Kale, Y . Maurya, Z. S. Wu, and V . Smith, “Po- sition: Llm unlearning benchmarks are weak measures of progress,” arXiv preprint arXiv:2410.02879, 2024

    arXiv 2024

  8. [99]

    Keeping an eye on llm unlearning: The hidden risk and remedy,

    J. Ren, Z. Dai, X. Tang, Y . Xing, S. Zeng, H. Liu, J. Zeng, Q. Peng, S. Varshney, S. Wang, Q. He, C. C. Aggarwal, and H. Liu, “Keeping an eye on llm unlearning: The hidden risk and remedy,”

Show all 123 references
  1. [1]

    Nat- ural questions: a benchmark for question answering research,

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Leeet al., “Nat- ural questions: a benchmark for question answering research,”Trans- actions of the Association for Computational Linguistics, vol. 7, pp. 453–...

    2019

  2. [2]

    Neural machine translation: A review,

    F. Stahlberg, “Neural machine translation: A review,”Journal of Artificial Intelligence Research, vol. 69, pp. 343–418, 2020

    2020

  3. [3]

    Automatic text summarization: A comprehensive survey,

    W. S. El-Kassas, C. R. Salama, A. A. Rafea, and H. K. Mohamed, “Automatic text summarization: A comprehensive survey,”Expert systems with applications, vol. 165, p. 113679, 2021

    2021

  4. [4]

    Deep reinforcement learning for dialogue generation,

    J. Li, W. Monroe, A. Ritter, M. Galley, J. Gao, and D. Juraf- sky, “Deep reinforcement learning for dialogue generation,”arXiv preprint arXiv:1606.01541, 2016

    2016 arXiv

  5. [5]

    Regulating chatgpt and other large generative ai models,

    P. Hacker, A. Engel, and M. Mauer, “Regulating chatgpt and other large generative ai models,” inProceedings of the 2023 ACM Con- ference on Fairness, Accountability, and Transparency, 2023, pp. 1112–1123

    2023

  6. [6]

    Chatgpt: a case study on copyright challenges for generative artificial intelligence systems,

    N. Lucchi, “Chatgpt: a case study on copyright challenges for generative artificial intelligence systems,”European Journal of Risk Regulation, vol. 15, no. 3, pp. 602–624, 2024

    2024

  7. [7]

    M. M. Grynbaum and R. Mac. (2023) The times sues openai and microsoft over a.i. use of copyrighted work. [Online]. Available: https://www.nytimes.com/2023/12/27/business/ media/new-york-times-open-ai-microsoft-lawsuit.html

    2023

  8. [8]

    The eu general data protection regulation (gdpr),

    P. V oigt and A. V on dem Bussche, “The eu general data protection regulation (gdpr),”A Practical Guide, 1st Ed., Cham: Springer International Publishing, vol. 10, no. 3152676, pp. 10–5555, 2017

    2017

  9. [9]

    The right to be forgotten,

    J. Rosen, “The right to be forgotten,”Stan. L. Rev. Online, vol. 64, p. 88, 2011

    2011

  10. [10]

    Rethinking machine unlearning for large language models,

    S. Liu, Y . Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y . Yao, C. Y . Liu, X. Xu, H. Liet al., “Rethinking machine unlearning for large language models,”arXiv preprint arXiv:2402.08787, 2024

    2024 arXiv

  11. [11]

    TOFU: A task of fictitious unlearning for LLMs,

    P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter, “TOFU: A task of fictitious unlearning for LLMs,” in First Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id=B41hNBoWLo

    2024

  12. [12]

    Copyright protection in generative ai: A technical perspective,

    J. Ren, H. Xu, P. He, Y . Cui, S. Zeng, J. Zhang, H. Wen, J. Ding, P. Huang, L. Lyuet al., “Copyright protection in generative ai: A technical perspective,”arXiv preprint arXiv:2402.02333, 2024

    2024 arXiv

  13. [13]

    Large language model unlearning,

    Y . Yao, X. Xu, and Y . Liu, “Large language model unlearning,” arXiv preprint arXiv:2310.10683, 2023

    2023 arXiv

  14. [14]

    Negative preference opti- mization: From catastrophic collapse to effective unlearning,

    R. Zhang, L. Lin, Y . Bai, and S. Mei, “Negative preference opti- mization: From catastrophic collapse to effective unlearning,”arXiv preprint arXiv:2404.05868, 2024

    2024 arXiv

  15. [15]

    The wmdp benchmark: Measuring and reducing malicious use with unlearning,

    N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, G. Mukobiet al., “The wmdp benchmark: Measuring and reducing malicious use with unlearning,” inForty- first International Conference on Machine Learning, 2024

    2024

  16. [16]

    Offset unlearning for large language models,

    J. Y . Huang, W. Zhou, F. Wang, F. Morstatter, S. Zhang, H. Poon, and M. Chen, “Offset unlearning for large language models,”arXiv preprint arXiv:2404.11045, 2024

    2024 arXiv

  17. [17]

    In-context unlearn- ing: Language models as few shot unlearners,

    M. Pawelczyk, S. Neel, and H. Lakkaraju, “In-context unlearn- ing: Language models as few shot unlearners,”arXiv preprint arXiv:2310.07579, 2023

    2023 arXiv

  18. [18]

    Executive order 14110: Safe, secure, and trustworthy development and use of artificial intelligence,

    J. R. B. Jr., “Executive order 14110: Safe, secure, and trustworthy development and use of artificial intelligence,” https://www.federalregister.gov/documents/2023/11/01/2023-24283/ safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence, October 2023, 88 Fe...

    2023

  19. [19]

    Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence (artificial intelligence act),

    “Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence (artificial intelligence act),” https://eur-lex.europa.eu/eli/ reg/2024/1689/oj, 2024, article 53

    2024

  20. [20]

    europa.eu/eli/reg/2016/679/oj, 2016, article 17

    “Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (general data protection regulation),” https://eur-lex. europa.eu/...

    2016

  21. [21]

    Knowledge unlearning for mitigating privacy risks in language models,

    J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, “Knowledge unlearning for mitigating privacy risks in language models,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 14 389–14 408

    2023

  22. [22]

    Muse: Machine unlearning six-way evaluation for language models,

    W. Shi, J. Lee, Y . Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, and C. Zhang, “Muse: Machine unlearning six-way evaluation for language models,”arXiv preprint arXiv:2407.06460, 2024

    2024 arXiv

  23. [23]

    Machine unlearning of pre-trained large language models,

    J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, and X. Yue, “Machine unlearning of pre-trained large language models,” inPro- ceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), 2024, pp. 8403–8419

    2024

  24. [24]

    Llm surgery: Efficient knowledge unlearning and editing in large language models,

    A. K. Veldanda, S.-X. Zhang, A. Das, S. Chakraborty, S. Rawls, S. Sahu, and M. Naphade, “Llm surgery: Efficient knowledge unlearning and editing in large language models,”arXiv preprint arXiv:2409.13054, 2024

    2024 arXiv

  25. [25]

    Simplicity prevails: Rethinking negative preference optimization for llm unlearning,

    C. Fan, J. Liu, L. Lin, J. Jia, R. Zhang, S. Mei, and S. Liu, “Simplicity prevails: Rethinking negative preference optimization for llm unlearning,”arXiv preprint arXiv:2410.07163, 2024

    2024

  26. [26]

    Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate,

    Z. Bu, X. Jin, B. Vinzamuri, A. Ramakrishna, K.-W. Chang, V . Cevher, and M. Hong, “Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate,”arXiv preprint arXiv:2410.22086, 2024

    arXiv

  27. [27]

    Grail: Gradient- based adaptive unlearning for privacy and copyright in llms,

    K.-W. Kim, J.-H. Park, J.-M. Han, and S.-W. Lee, “Grail: Gradient- based adaptive unlearning for privacy and copyright in llms,”arXiv preprint arXiv:2504.12681, 2025

    2025 arXiv

  28. [28]

    Dualoptim: Enhancing efficacy and stability in machine unlearning with dual optimizers,

    X. Zhong, H. Luo, and C. Liu, “Dualoptim: Enhancing efficacy and stability in machine unlearning with dual optimizers,”arXiv preprint arXiv:2504.15827, 2025

    2025

  29. [29]

    Soul: Unlocking the power of second- order optimization for llm unlearning,

    J. Jia, Y . Zhang, Y . Zhang, J. Liu, B. Runwal, J. Diffenderfer, B. Kailkhura, and S. Liu, “Soul: Unlocking the power of second- order optimization for llm unlearning,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 4276–4292

    2024

  30. [30]

    A unified gradient-based framework for task-agnostic continual learning-unlearning,

    Z. Huang, X. Cheng, J. Zhang, J. Zheng, H. Wang, Z. He, T. Li, and X. Huang, “A unified gradient-based framework for task-agnostic continual learning-unlearning,”arXiv preprint arXiv:2505.15178, 2025

    2025 arXiv

  31. [31]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training,

    H. Liu, Z. Li, D. L. W. Hall, P. Liang, and T. Ma, “Sophia: A scalable stochastic second-order optimizer for language model pre-training,” inThe Twelfth International Conference on Learning Representations

  32. [32]

    To each (textual sequence) its own: Improving memorized-data unlearning in large language models,

    G.-O. B ˘arbulescu and P. Triantafillou, “To each (textual sequence) its own: Improving memorized-data unlearning in large language models,” inInternational Conference on Machine Learning. PMLR, 2024, pp. 3003–3023

    2024

  33. [33]

    Fine-grained pluggable gradient ascent for knowledge unlearning in language models,

    X. Feng, C. Chen, Y . Li, and Z. Lin, “Fine-grained pluggable gradient ascent for knowledge unlearning in language models,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 10 141–10 155

    2024

  34. [34]

    Selective forgetting: Advancing machine unlearning techniques and evaluation in language models,

    L. Wang, X. Zeng, J. Guo, K.-F. Wong, and G. Gottlob, “Selective forgetting: Advancing machine unlearning techniques and evaluation in language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 1, 2025, pp. 843–851

    2025

  35. [35]

    Editing models with task arithmetic,

    G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  36. [36]

    Negmerge: Consensual weight negation for strong machine unlearning,

    H. Kim, D. Han, and J. Choe, “Negmerge: Consensual weight negation for strong machine unlearning,”arXiv preprint arXiv:2410.05583, 2024

    2024 arXiv

  37. [37]

    When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers,

    H. Li, Y . Zhang, S. Zhang, P.-Y . Chen, S. Liu, and M. Wang, “When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers,” inThe Thirteenth International Conference on Learning Representations

  38. [38]

    Exact unlearning of finetuning data via model merging at scale,

    K. Kuo, A. Setlur, K. Srinivas, A. Raghunathan, and V . Smith, “Exact unlearning of finetuning data via model merging at scale,” arXiv preprint arXiv:2504.04626, 2025

    2025 arXiv

  39. [39]

    Towards robust and parameter-efficient knowledge unlearning for llms,

    S. Cha, S. Cho, D. Hwang, and M. Lee, “Towards robust and parameter-efficient knowledge unlearning for llms,” inThe Thir- teenth International Conference on Learning Representations

  40. [40]

    Obliviate: Efficient unmemorization for protecting intellectual property in large language models,

    M. Russinovich and A. Salem, “Obliviate: Efficient unmemorization for protecting intellectual property in large language models,”arXiv preprint arXiv:2502.15010, 2025

    2025 arXiv

  41. [41]

    Towards safer large language models through machine unlearning,

    Z. Liu, G. Dou, Z. Tan, Y . Tian, and M. Jiang, “Towards safer large language models through machine unlearning,”arXiv preprint arXiv:2402.10058, 2024

    2024 arXiv

  42. [42]

    Knowledge sanitization of large language models,

    Y . Ishibashi and H. Shimodaira, “Knowledge sanitization of large language models,”arXiv preprint arXiv:2309.11852, 2023

    2023 arXiv

  43. [43]

    Suv: Scalable large language model copyright compliance with regularized selective unlearning,

    T. Xu, X. Liu, F. Wu, X. Wang, and J. Gao, “Suv: Scalable large language model copyright compliance with regularized selective unlearning,”arXiv preprint arXiv:2503.22948, 2025

    2025

  44. [44]

    Who’s harry potter? approximate unlearning in llms,

    R. Eldan and M. Russinovich, “Who’s harry potter? approximate unlearning in llms,”arXiv preprint arXiv:2310.02238, 2023

    2023 arXiv

  45. [45]

    Revisiting who’s harry potter: Towards targeted unlearning from a causal intervention perspective,

    Y . Liu, Y . Zhang, T. Jaakkola, and S. Chang, “Revisiting who’s harry potter: Towards targeted unlearning from a causal intervention perspective,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 8708–8731

    2024

  46. [46]

    Relearn: Unlearning via learning for large language models,

    H. Xu, N. Zhao, L. Yang, S. Zhao, S. Deng, M. Wang, B. Hooi, N. Oo, H. Chen, and N. Zhang, “Relearn: Unlearning via learning for large language models,”arXiv preprint arXiv:2502.11190, 2025

    2025 arXiv

  47. [47]

    Answer when needed, forget when not: Language models pretend to forget via in-context knowledge unlearning,

    S. Takashiro, T. Kojima, A. Gambardella, Q. Cao, Y . Iwasawa, and Y . Matsuo, “Answer when needed, forget when not: Language models pretend to forget via in-context knowledge unlearning,” arXiv preprint arXiv:2410.00382, 2024

    arXiv

  48. [48]

    When machine unlearning meets retrieval-augmented generation (rag): Keep secret or forget knowledge?

    S. Wang, T. Zhu, D. Ye, and W. Zhou, “When machine unlearning meets retrieval-augmented generation (rag): Keep secret or forget knowledge?”arXiv preprint arXiv:2410.15267, 2024

    2024 arXiv

  49. [49]

    (de)-indexing and the right to be forgotten,

    S. Vilella and G. Ruffo, “(de)-indexing and the right to be forgotten,” arXiv preprint arXiv:2501.03989, 2025

    2025 arXiv

  50. [50]

    Alu: Agentic llm unlearning,

    D. Sanyal and M. Mandal, “Alu: Agentic llm unlearning,”arXiv preprint arXiv:2502.00406, 2025

    2025 arXiv

  51. [51]

    Large language model unlearning via embedding-corrupted prompts,

    C. Y . Liu, Y . Wang, J. Flanigan, and Y . Liu, “Large language model unlearning via embedding-corrupted prompts,”arXiv preprint arXiv:2406.07933, 2024

    2024 arXiv

  52. [52]

    On effects of steering latent representation for large language model unlearning,

    D. Huu-Tien, T.-T. Pham, H. Thanh-Tung, and N. Inoue, “On effects of steering latent representation for large language model unlearning,”arXiv preprint arXiv:2408.06223, 2024

    2024 arXiv

  53. [53]

    Lunar: Llm unlearning via neural activation redirection,

    W. F. Shen, X. Qiu, M. Kurmanji, A. Iacob, L. Sani, Y . Chen, N. Cancedda, and N. D. Lane, “Lunar: Llm unlearning via neural activation redirection,”arXiv preprint arXiv:2502.07218, 2025

    2025

  54. [54]

    Im- proving the robustness of representation misdirection for large lan- guage model unlearning,

    D. Huu-Tien, H. Thanh-Tung, L.-M. Nguyen, and N. Inoue, “Im- proving the robustness of representation misdirection for large lan- guage model unlearning,”arXiv preprint arXiv:2501.19202, 2025

    2025 arXiv

  55. [55]

    Falcon: Fine-grained activation manipulation by con- trastive orthogonal unalignment for large language model,

    J. Hu, Z. Huang, X. Yin, W. Ruan, G. Cheng, Y . Dong, and X. Huang, “Falcon: Fine-grained activation manipulation by con- trastive orthogonal unalignment for large language model,”arXiv preprint arXiv:2502.01472, 2025

    2025

  56. [56]

    Large scale knowledge washing,

    Y . Wang, R. Wu, Z. He, X. Chen, and J. McAuley, “Large scale knowledge washing,”arXiv preprint arXiv:2405.16720, 2024

    2024 arXiv

  57. [57]

    Applying sparse autoencoders to unlearn knowledge in language models,

    E. Farrell, Y .-T. Lau, and A. Conmy, “Applying sparse autoencoders to unlearn knowledge in language models,” inNeurips Safe Gener- ative AI Workshop 2024

    2024

  58. [58]

    Don’t forget it! conditional sparse autoencoder clamping works for unlearning,

    M. Khoriaty, A. Shportko, G. Mercier, and Z. Wood-Doughty, “Don’t forget it! conditional sparse autoencoder clamping works for unlearning,”arXiv preprint arXiv:2503.11127, 2025

    2025 arXiv

  59. [59]

    Sauce: Selective concept unlearning in vision-language models with sparse autoencoders,

    Q. Li, J. Geng, D. Zhu, F. Cai, C. Lyu, and F. Karray, “Sauce: Selective concept unlearning in vision-language models with sparse autoencoders,”arXiv preprint arXiv:2503.14530, 2025

    2025 arXiv

  60. [60]

    Saes can improve unlearning: Dynamic sparse autoencoder guardrails for precision unlearning in llms,

    A. Muhamed, J. Bonato, M. Diab, and V . Smith, “Saes can improve unlearning: Dynamic sparse autoencoder guardrails for precision unlearning in llms,”arXiv preprint arXiv:2504.08192, 2025

    2025 arXiv

  61. [61]

    Unlearn what you want to forget: Efficient unlearning for llms,

    J. Chen and D. Yang, “Unlearn what you want to forget: Efficient unlearning for llms,” inThe 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  62. [63]

    On large language model continual unlearning,

    C. Gao, L. Wang, K. Ding, C. Weng, X. Wang, and Q. Zhu, “On large language model continual unlearning,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  63. [64]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.”ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  64. [65]

    Dissecting language mod- els: Machine unlearning via selective pruning,

    N. Pochinkov and N. Schoots, “Dissecting language mod- els: Machine unlearning via selective pruning,”arXiv preprint arXiv:2403.01267, 2024

    2024 arXiv

  65. [66]

    Modality- aware neuron pruning for unlearning in multimodal large language models,

    Z. Liu, G. Dou, X. Yuan, C. Zhang, Z. Tan, and M. Jiang, “Modality- aware neuron pruning for unlearning in multimodal large language models,”arXiv preprint arXiv:2502.15910, 2025

    2025 arXiv

  66. [67]

    Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference,

    J. Ji, Y . Liu, Y . Zhang, G. Liu, R. R. Kompella, S. Liu, and S. Chang, “Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference,”arXiv preprint arXiv:2406.08607, 2024

    2024 arXiv

  67. [68]

    Guard: Generation-time llm unlearning via adaptive restriction and detection,

    Z. Deng, C. Y . Liu, Z. Pang, X. He, L. Feng, Q. Xuan, Z. Zhu, and J. Wei, “Guard: Generation-time llm unlearning via adaptive restriction and detection,”arXiv preprint arXiv:2505.13312, 2025

    2025 arXiv

  68. [69]

    Machine unlearn- ing,

    L. Bourtoule, V . Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, “Machine unlearn- ing,” in2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021, pp. 141–159

    2021

  69. [70]

    Continual learning and private unlearning,

    B. Liu, Q. Liu, and P. Stone, “Continual learning and private unlearning,” inConference on Lifelong Learning Agents. PMLR, 2022, pp. 243–254

    2022

  70. [71]

    Amnesiac machine learn- ing,

    L. Graves, V . Nagisetty, and V . Ganesh, “Amnesiac machine learn- ing,” inProceedings of the AAAI Conference on Artificial Intelli- gence, vol. 35, no. 13, 2021, pp. 11 516–11 524

    2021

  71. [72]

    Unrolling sgd: Understanding factors influencing machine unlearning,

    A. Thudi, G. Deza, V . Chandrasekaran, and N. Papernot, “Unrolling sgd: Understanding factors influencing machine unlearning,” in2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P). IEEE, 2022, pp. 303–319

    2022

  72. [73]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in Neural Information Process- ing Systems, vol. 36, 2024

    2024

  73. [74]

    In-context unlearning: Language models as few-shot unlearners,

    M. Pawelczyk, S. Neel, and H. Lakkaraju, “In-context unlearning: Language models as few-shot unlearners,” inForty-first Interna- tional Conference on Machine Learning

  74. [75]

    Effective skill unlearning through intervention and abstention,

    Y . Li, C.-E. Sun, and T.-W. Weng, “Effective skill unlearning through intervention and abstention,”arXiv preprint arXiv:2503.21730, 2025

    2025 arXiv

  75. [76]

    Sparse autoencoders find highly interpretable features in language models,

    H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey, “Sparse autoencoders find highly interpretable features in language models,”arXiv preprint arXiv:2309.08600, 2023

    2023 arXiv

  76. [77]

    A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models,

    D. Shu, X. Wu, H. Zhao, D. Rai, Z. Yao, N. Liu, and M. Du, “A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models,”arXiv preprint arXiv:2503.05613, 2025

    2025

  77. [78]

    Sparse autoencoder

    A. Nget al., “Sparse autoencoder.”

  78. [79]

    Mechanistic unlearning: Robust knowledge unlearning and edit- ing via mechanistic localization,

    P. Guo, A. Syed, A. Sheshadri, A. Ewart, and G. K. Dziugaite, “Mechanistic unlearning: Robust knowledge unlearning and edit- ing via mechanistic localization,”arXiv preprint arXiv:2410.12949, 2024

    2024 arXiv

  79. [80]

    Faithun: Toward faithful forgetting in language models by investigating the intercon- nectedness of knowledge,

    N. Yang, M. Kim, S. Yoon, J. Shin, and K. Jung, “Faithun: Toward faithful forgetting in language models by investigating the intercon- nectedness of knowledge,”arXiv preprint arXiv:2502.19207, 2025

    2025

  80. [81]

    Does localization inform unlearning? a rigorous examination of local parameter attribution for knowledge unlearning in language models,

    H. Lee, U. Hwang, H. Lim, and T. Kim, “Does localization inform unlearning? a rigorous examination of local parameter attribution for knowledge unlearning in language models,”arXiv preprint arXiv:2505.16252, 2025

    2025

  81. [83]

    Eight methods to evaluate robust unlearning in llms,

    A. Lynch, P. Guo, A. Ewart, S. Casper, and D. Hadfield-Menell, “Eight methods to evaluate robust unlearning in llms,”arXiv preprint arXiv:2402.16835, 2024

    2024 arXiv

  82. [86]

    Harry potter is still here! probing knowledge leakage in targeted unlearned large language models via automated adversarial prompting,

    B. T. T. To and T. Le, “Harry potter is still here! probing knowledge leakage in targeted unlearned large language models via automated adversarial prompting,”arXiv preprint arXiv:2505.17160, 2025

    2025 arXiv

  83. [89]

    Universal and transferable adversarial attacks on aligned lan- guage models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrik- son, “Universal and transferable adversarial attacks on aligned lan- guage models,”arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  84. [90]

    Dissecting fine-tuning unlearning in large language models,

    Y . Hong, Y . Zou, L. Hu, Z. Zeng, D. Wang, and H. Yang, “Dissecting fine-tuning unlearning in large language models,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 3933–3941

    2024

  85. [91]

    Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space,

    L. Schwinn, D. Dobre, S. Xhonneux, G. Gidel, and S. G ¨unnemann, “Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space,”Advances in Neural Information Processing Systems, vol. 37, pp. 9086–9116, 2024

    2024

  86. [92]

    Tex- tual unlearning gives a false sense of unlearning,

    J. Du, Z. Wang, J. Zhang, X. Pang, J. Hu, and K. Ren, “Tex- tual unlearning gives a false sense of unlearning,”arXiv preprint arXiv:2406.13348, 2024

    2024 arXiv

  87. [93]

    Improving align- ment and robustness with circuit breakers,

    A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, J. Z. Kolter, M. Fredrikson, and D. Hendrycks, “Improving align- ment and robustness with circuit breakers,” inThe Thirty-eighth An- nual Conference on Neural Information Processing Systems, 2024

    2024

  88. [95]

    Safe unlearning: A surprisingly effective and gener- alizable solution to defend against jailbreak attacks,

    Z. Zhang, J. Yang, P. Ke, S. Cui, C. Zheng, H. Wang, and M. Huang, “Safe unlearning: A surprisingly effective and gener- alizable solution to defend against jailbreak attacks,”arXiv preprint arXiv:2407.02855, vol. 1, no. 2, p. 3, 2024

    2024 arXiv

  89. [96]

    Upcore: Utility- preserving coreset selection for balanced unlearning,

    V . Patil, E. Stengel-Eskin, and M. Bansal, “Upcore: Utility- preserving coreset selection for balanced unlearning,”arXiv preprint arXiv:2502.15082, 2025

    2025 arXiv

  90. [97]

    Dusk: Do not unlearn shared knowledge,

    W. Jeung, S. Yoon, H. Hong, S. Kim, S. Han, Y . Yu, and A. No, “Dusk: Do not unlearn shared knowledge,”arXiv preprint arXiv:2505.15209, 2025

    2025 arXiv

  91. [98]

    Which retain set matters for llm unlearning? a case study on entity unlearning,

    H. Chang and H. Lee, “Which retain set matters for llm unlearning? a case study on entity unlearning,”arXiv preprint arXiv:2502.11441, 2025

    2025 arXiv

  92. [100]

    A closer look at machine unlearning for large language models,

    X. Yuan, T. Pang, C. Du, K. Chen, W. Zhang, and M. Lin, “A closer look at machine unlearning for large language models,”arXiv preprint arXiv:2410.08109, 2024

    2024 arXiv

  93. [101]

    Rwku: Benchmarking real-world knowledge unlearning for large language models,

    Z. Jin, P. Cao, C. Wang, Z. He, H. Yuan, J. Li, Y . Chen, K. Liu, and J. Zhao, “Rwku: Benchmarking real-world knowledge unlearning for large language models,”arXiv preprint arXiv:2406.10890, 2024

    2024 arXiv

  94. [102]

    Lume: Llm unlearning with multitask evaluations,

    A. Ramakrishna, Y . Wan, X. Jin, K.-W. Chang, Z. Bu, B. Vinzamuri, V . Cevher, M. Hong, and R. Gupta, “Lume: Llm unlearning with multitask evaluations,”arXiv preprint arXiv:2502.15097, 2025

    2025 arXiv

  95. [103]

    Privacy risk in machine learning: Analyzing the connection to overfitting,

    S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, “Privacy risk in machine learning: Analyzing the connection to overfitting,” in2018 IEEE 31st computer security foundations symposium (CSF). IEEE, 2018, pp. 268–282

    2018

  96. [104]

    Extracting training data from large language models,

    N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingssonet al., “Extracting training data from large language models,” in30th USENIX security symposium (USENIX Security 21), 2021, pp. 2633– 2650

    2021

  97. [105]

    Detecting pretraining data from large language models,

    W. Shi, A. Ajith, M. Xia, Y . Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, “Detecting pretraining data from large language models,”arXiv preprint arXiv:2310.16789, 2023

    2023 arXiv

  98. [106]

    Waterdrum: Watermarking for data-centric unlearning metric,

    X. Lu, X. Niu, G. K. R. Lau, B. T. C. Nhung, R. H. L. Sim, F. Wen, C.-S. Foo, S.-K. Ng, and B. K. H. Low, “Waterdrum: Watermarking for data-centric unlearning metric,”arXiv preprint arXiv:2505.05064, 2025

    2025

  99. [107]

    Rethinking llm unlearning objectives: A gradient perspec- tive and go beyond,

    Q. Wang, J. P. Zhou, Z. Zhou, S. Shin, B. Han, and K. Q. Wein- berger, “Rethinking llm unlearning objectives: A gradient perspec- tive and go beyond,”arXiv preprint arXiv:2502.19301, 2025

    2025 arXiv

  100. [108]

    Losing is for cherishing: Data valuation based on machine unlearn- ing and shapley value,

    L. Ma, S. Yang, Z. Wang, Y . Wang, L. Wang, T. Wei, and K. Zhang, “Losing is for cherishing: Data valuation based on machine unlearn- ing and shapley value,”arXiv preprint arXiv:2505.16147, 2025

    2025

  101. [109]

    A value for n-person games,

    L. S. Shapleyet al., “A value for n-person games,” 1953

    1953

  102. [110]

    Unlearning isn’t deletion: Investigating reversibility of machine unlearning in llms,

    X. Xu, X. Yue, Y . Liu, Q. Ye, H. Hu, and M. Du, “Unlearning isn’t deletion: Investigating reversibility of machine unlearning in llms,” arXiv preprint arXiv:2505.16831, 2025

    2025 arXiv

  103. [111]

    Erasing without remembering: Implicit knowledge forgetting in large language models,

    H. Wang, Y . Jing, H. Sun, Y . Wang, J. Wang, J. Liao, and D. Tao, “Erasing without remembering: Implicit knowledge forgetting in large language models,”arXiv preprint arXiv:2502.19982, 2025

    2025

  104. [112]

    R-tofu: Unlearning in large reason- ing models,

    S. Yoon, W. Jeung, and A. No, “R-tofu: Unlearning in large reason- ing models,”arXiv preprint arXiv:2505.15214, 2025

    2025 arXiv

  105. [113]

    Measuring massive multitask language under- standing,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language under- standing,”arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  106. [114]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in Neural Information Pro- cessing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  107. [115]

    A probabilistic per- spective on unlearning and alignment for large language models,

    Y . Scholten, S. G ¨unnemann, and L. Schwinn, “A probabilistic per- spective on unlearning and alignment for large language models,” arXiv preprint arXiv:2410.03523, 2024

    2024 arXiv

  108. [116]

    Not all data are unlearned equally,

    A. Krishnan, S. Reddy, and M. Mosbach, “Not all data are unlearned equally,”arXiv preprint arXiv:2504.05058, 2025

    2025 arXiv

  109. [117]

    An empir- ical study of catastrophic forgetting in large language models during continual fine-tuning,

    Y . Luo, Z. Yang, F. Meng, Y . Li, J. Zhou, and Y . Zhang, “An empir- ical study of catastrophic forgetting in large language models during continual fine-tuning,”arXiv preprint arXiv:2308.08747, 2023

    2023 arXiv

  110. [118]

    Llm unlearning reveals a stronger-than-expected coreset effect in current benchmarks,

    S. Pal, C. Wang, J. Diffenderfer, B. Kailkhura, and S. Liu, “Llm unlearning reveals a stronger-than-expected coreset effect in current benchmarks,”arXiv preprint arXiv:2504.10185, 2025

    2025 arXiv

  111. [119]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  112. [120]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications,

    B. Wei, K. Huang, Y . Huang, T. Xie, X. Qi, M. Xia, P. Mittal, M. Wang, and P. Henderson, “Assessing the brittleness of safety alignment via pruning and low-rank modifications,”arXiv preprint arXiv:2402.05162, 2024

    2024 arXiv

  113. [121]

    The unlocking spell on base llms: Rethinking alignment via in-context learning,

    B. Y . Lin, A. Ravichander, X. Lu, N. Dziri, M. Sclar, K. Chandu, C. Bhagavatula, and Y . Choi, “The unlocking spell on base llms: Rethinking alignment via in-context learning,”arXiv preprint arXiv:2312.01552, 2023

    2023 arXiv

  114. [122]

    Catastrophic failure of llm unlearning via quantization,

    Z. Zhang, F. Wang, X. Li, Z. Wu, X. Tang, H. Liu, Q. He, W. Yin, and S. Wang, “Catastrophic failure of llm unlearning via quantization,”arXiv preprint arXiv:2410.16454, 2024

    2024 arXiv

  115. [2025]

    Available: https://arxiv.org/abs/2506.00359

    [Online]. Available: https://arxiv.org/abs/2506.00359

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.