Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Unlearning methods suppress LLMs' ability to use forgotten facts when those facts appear in the prompt, and a KL-consistency term restores that contextual utility without weakening forgetting.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 09:00 UTC pith:S5GNWRZ4

load-bearing objection A genuinely useful diagnostic finding, but the fix's headline numbers look in-sample. the 4 major comments →

arxiv 2510.17620 v2 pith:S5GNWRZ4 submitted 2025-10-20 cs.CL

Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models

Yuefeng Peng , Parnian Afshar , Megan Ganji , Thomas Butler , Amir Houmansadr , Mingxian Wang , Dezhi Hong This is my paper
classification cs.CL
keywords LLM unlearningcontextual utilityin-context learningknowledge forgettingKL divergenceTOFU benchmarkcontext-aware unlearningretrieval-augmented generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies a blind spot in LLM unlearning evaluation: standard benchmarks measure whether the model forgot targeted facts (Direct QA) and whether it retains general utility, but not whether it can still use that same knowledge when it is explicitly supplied in the prompt—the situation in RAG and interactive assistants. Evaluating six unlearning methods on the TOFU benchmark across two instruction-tuned models, the authors find that unlearning consistently degrades this 'contextual utility,' sometimes to near zero even when the ground-truth answer is in the context. To fix this, they add a plug-in KL-divergence term that anchors the model's contextual predictions to the original pre-unlearned model. With this term added to RMU, NPO, and UNDIAL, Contextual QA scores rise to near-original levels (≥0.95 LLM-Judge) while forgetting and retain-set utility stay essentially unchanged. The paper argues that unlearning evaluation should include context-grounded use of forgotten facts and that the KL term is a general, easy-to-tune fix.

Core claim

The central discovery is that unlearning objectives that penalize the forget set also suppress the model's ability to condition on the same content when it is later provided as external evidence. Existing two-term objectives L(w) = −λ_f L_f(S_f,w) + λ_r L_r(S_r,w) optimize only a forget-versus-retain trade-off; the forget term reshapes internal representations or output distributions in a way that ripples into inference-time conditioning. The paper's fix adds a third term, λ_c C(S_ctx^f,w), a KL-consistency term that aligns the unlearned model's distribution on contextual queries (forget question + gold context) with the frozen original model's distribution. This separation is meant to enfor

What carries the argument

The load-bearing object is the context term C(S_ctx^f,w) = (1/|S_ctx^f|) Σ_{(q,a,c)∈S_ctx^f} KL( p_w(·|q,c) ∥ p_orig(·|q,c) ), a plug-in KL-divergence regularizer. It is added to any unlearning objective as J(w) = −λ_f L_f(S_f,w) + λ_r L_r(S_r,w) + λ_c C(S_ctx^f,w). The term anchors the unlearned model's predictive distribution on context-grounded forget queries to that of the frozen, pre-unlearned model, counteracting the representation-level suppression caused by the forget term. Its modularity is the point: it can be attached to gradient-ascent, NPO, RMU, or UNDIAL without redesigning the underlying method.

Load-bearing premise

The near-perfect restoration of Contextual QA is measured on context-augmented forget-set questions that are likely the same ones used to train the KL term; the paper does not describe a held-out contextual split, so if the evaluation set overlaps with S_ctx^f, the gains would be an in-sample artifact rather than evidence of restored general capability.

What would settle it

Partition the forget set into disjoint subsets: use one subset's context-augmented examples for the KL term, and evaluate Contextual QA on the other subset's questions with context. If Contextual QA gains on the held-out subset are much smaller than on the trained subset, the restoration is in-sample; if comparable, the mechanism generalizes.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Unlearning evaluation should adopt Contextual QA as a standard axis: a method that forgets well but cannot ground on supplied evidence is less deployable than current metrics suggest, especially in retrieval-augmented systems.
  • The plug-in KL term is a drop-in fix: any existing unlearning objective can be augmented with it, and the paper reports it is stable across a wide range of λ_c values, lowering the tuning burden.
  • The approach lets models satisfy both privacy/compliance goals and in-context usability: they refuse to volunteer forgotten facts from memory while correctly answering when the fact is given in the prompt.
  • Methods that suppress contextual use most (RMU, gradient ascent, GradDiff) benefit most; the gains on RMU are the largest (Contextual QA from ~0.00 to ≥0.97 LLM-Judge).
  • Robustness to paraphrased and reasoning-style contexts is demonstrated in a single case study, suggesting the restored ability is not just verbatim copying from the prompt.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the in-sample issue is resolved (held-out contextual split), the KL-consistency mechanism could transfer to other unlearning settings—e.g., forgetting outdated or copyrighted content—where the same 'forget but use if provided' behavior is desired. The paper does not test this beyond TOFU.
  • The KL term essentially prevents the unlearning objective from corrupting the conditional use of tokens that appear in context. A natural extension would be to measure whether the term also protects against adversarial relearning or membership-inference leakage, since it anchors to the original model's distribution.
  • For deployment, the paper's λ_c selection procedure (grid search with tolerance δ=0.06 on forgetting) is a validation-set-based selection; an independent held-out Contextual QA set would be needed to confirm that the reported near-perfect scores reflect general capability rather than overfitting to the training context examples.
  • The utility of the approach depends on the assumption that a user may legitimately reintroduce forgotten facts; in strictly regulated settings where even context-grounded use is prohibited, the KL term might be unnecessary or even counterproductive—a boundary case the paper does not discuss.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies 'contextual utility' in LLM unlearning: after unlearning on a forget set, models often fail to use that same information when it is reintroduced in the prompt as context. The authors evaluate six unlearning methods on TOFU with two instruction-tuned models, showing consistent degradation of Contextual QA when the ground-truth answer is provided in the prompt. They then propose a plug-in KL-divergence term added to existing unlearning objectives that aligns the unlearned model's contextual predictive distribution with the original pre-unlearned model on forget-set questions paired with ground-truth context. Experiments on NPO, RMU, and UNDIAL report that this context-aware variant restores Contextual QA to near-perfect levels while maintaining Direct QA forgetting and retain-set utility. The central claim is that this is a general, practical fix for the overlooked side effect of contextual suppression.

Significance. If the central claim held, the paper would make a useful contribution: it identifies a real and under-explored failure mode in unlearning evaluation and proposes a simple, modular remedy. The systematic evaluation of vanilla methods across models and forget ratios is a strength, and the qualitative examples clearly illustrate the phenomenon. The proposed KL-consistency term is a natural and easy-to-integrate addition to existing objectives. However, the evidence for the method's effectiveness is compromised by apparent overlap between the training distribution of the context term and the Contextual QA evaluation set, and by hyperparameter selection on the same metrics and examples. The only out-of-distribution evaluation is a single example, which is not sufficient to support the claim of restored general capability. The manuscript is therefore not yet ready for publication in its current form.

major comments (4)
  1. [§4.2, §5, Table 2] The Contextual QA evaluation appears to use the same (q, c) pairs as the training set S_ctx^f for the KL term. Section 5 states 'we use the same datasets... as described in Section 3,' and Section 3's Contextual QA template provides the ground-truth answer as context. No held-out contextual split is described in §5 or Appendix A.4. Thus the near-perfect Contextual QA scores in Table 2 may reflect in-sample imitation of the original model on memorized (q,c) pairs rather than a recovered general capability to use forgotten facts as context. The authors should evaluate on a held-out set of forget-set questions (e.g., different questions about the same authors, or paraphrased/perturbed contexts) and report those numbers separately.
  2. [Appendix A.4] The selection of λ_c is performed by grid search over six values, choosing the run with the highest Contextual QA LLM-Judge and utility among those within δ=0.06 of the vanilla Direct QA score. Because λ_c is selected on the same metric and the same examples used in the headline Table 2 results, the reported restoration is optimistic and possibly an artifact of selection. The authors should either use a separate validation split for λ_c selection or report results for all λ_c values without selection, showing that the improvement is robust rather than cherry-picked.
  3. [§6.2] The only out-of-distribution evidence for the context-aware method is a single qualitative example with paraphrased and reasoning-based contexts (Table 4). This is anecdotal and cannot support the claim that the method restores contextual utility generally. The paper should include a quantitative evaluation on a held-out set of context variants (e.g., paraphrases, reorderings, or semantically equivalent but differently worded contexts) for all methods and models, not just one RMU example on Gemma-2B-IT.
  4. [§4.2 and Table 2] The headline metric (Contextual QA LLM-Judge) measures agreement with the original model's answer on the same distribution that the KL term is trained against. This makes the improvement somewhat circular: the objective explicitly trains the model to match the original model's contextual distribution, and the evaluation rewards exactly that match. A meaningful test would measure performance on contextual inputs that were not used in training, or measure whether the model can use forgotten facts in novel reasoning or compositional settings.
minor comments (5)
  1. [Abstract / §1] Typo: 'direclty' in the Introduction should be 'directly'.
  2. [§4.2] Typo: 'eaisly intergrate' should be 'easily integrate'.
  3. [Table 1] The NPO output contains 'quqlity' (likely 'quality') and the table would benefit from consistent formatting of the provided context and question.
  4. [§3 / Figure 2] The captions for Figure 2 do not explain what the shaded regions or error bars represent (if any). Please clarify.
  5. [Appendix A.4] The description of the convergence criterion is somewhat involved; a pseudocode block or a more precise formal definition would improve reproducibility.

Circularity Check

2 steps flagged

Contextual QA restoration is trained and selected on the same examples and metric it is then reported on; the improvement is largely in-sample by construction.

specific steps
  1. fitted input called prediction [§4.2 (Eq. 4), §3 Evaluation Tasks, §5 Setup]
    "J(w) = −λ_f L_f(S_f,w) + λ_r L_r(S_r,w) + λ_c C(S_ctx^f, w), where S_ctx^f denotes the forget examples paired with their ground-truth context. ... C(S_ctx^f, w) = (1/|S_ctx^f|) Σ_{(q,a,c)∈S_ctx^f} KL(p_w(·|q,c) || p_orig(·|q,c)). ... We use the same datasets, models, metrics, and training settings as described in Section 3. ... Contextual QA: The input prompt explicitly provides the ground-truth answer to each question, allowing us to test the model’s ability to leverage externally supplied information."

    The context term trains the unlearned model to match the original model's distribution on (q,c) pairs drawn from the forget set with the ground-truth answer supplied as context. The Contextual QA evaluation uses the same datasets and the same kind of prompt—ground-truth answer in the context—so the Table 2 Contextual QA scores are measured on the training distribution of the added KL term. Near-perfect restoration therefore reflects how well the KL objective was optimized on those exact examples, not an independent prediction of a recovered general capability. No held-out contextual split is described in §5 or Appendix A.4, so the reported restoration is in-sample by construction.

  2. fitted input called prediction [Appendix A.4 (λ_c selection and convergence criterion)]
    "We then select the one with the highest Contextual QA score (LLM-Judge) and model utility jointly among those that match the vanilla model’s forgetting effectiveness—that is, Direct QA (LLM-Judge) within a tolerance δ of the vanilla baseline. Here, δ is the allowed slack in forgetting effectiveness to enable contextual improvements, which we set to 0.06 in our evaluation. ... For each method, we identified the convergence epoch using the rule described earlier."

    The reported Table 2 numbers are not a single pre-registered configuration: λ_c is grid-searched over six values and the run with the highest Contextual QA LLM-Judge and utility (within a forgetting slack) is selected. The convergence epoch is also chosen by requiring Contextual QA to be near its global best. Since Contextual QA LLM-Judge is precisely the headline claim being evaluated, the reported near-perfect restoration is partly a model-selection artifact—choosing the best of several runs on the same metric that is then presented as the method's outcome.

full rationale

The paper's diagnosis that vanilla unlearning methods suppress contextual utility is independently demonstrated by the systematic evaluation in §3 and Figure 2, and that part is not circular. However, the paper's central constructive claim—that the plug-in KL term 'restores contextual utility to near original levels'—is supported by Table 2 numbers that are effectively in-sample. The context term C(S_ctx^f,w) trains the model to match the original model on forget questions with ground-truth context, while Contextual QA evaluates the same datasets with the same ground-truth-context prompts; no held-out contextual split is reported. Additionally, the specific λ_c and convergence epoch are selected using the same Contextual QA LLM-Judge metric that constitutes the headline result. Thus the reported near-perfect restoration reduces largely to the training objective and the selection rule, rather than to evidence of a general recovered capability. The only out-of-distribution evidence is the single-example qualitative robustness test in §6.2, which is anecdotal. Overall, this is partial circularity in the central restoration claim, while the forgetting and retain-set utility findings remain largely independent.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. Its central claim rests on one ad hoc design choice (using the original model as the contextual teacher), one unstated and load-bearing assumption about evaluation-set disjointness, and two standard but unvalidated benchmark assumptions. It also introduces three fitted hyperparameters (λ_c per method/model, δ, ε) that directly shape the reported results.

free parameters (3)
  • λ_c (NPO, RMU, UNDIAL) = Gemma: 2.0, 0.01, 0.5; Qwen: 1.0, 0.5, 1.0
    Chosen by grid search over six values per method and model, selecting the run with highest Contextual QA and utility among those within δ=0.06 of vanilla Direct QA (Appendix A.4). This hyperparameter directly controls the strength of the term that produces the headline gains.
  • δ (forgetting slack in λ_c selection) = 0.06
    Tolerance used when selecting λ_c: the context-aware run may have Direct QA LLM-Judge up to 0.06 worse than vanilla. This slack means the reported 'no loss in forgetting' claim is only true within a 6-point judge-score tolerance.
  • ε (convergence tolerance) = 0.01
    Convergence epoch requires Direct QA, Contextual QA, and utility all to be within 0.01 of their respective global bests (Appendix A.4). The reported epoch per method is a post hoc selection and affects the numbers in Table 2.
axioms (4)
  • ad hoc to paper The original pre-unlearned model's output distribution on contextual prompts is the correct target for 'contextual utility'.
    C(S_ctx^f,w) in §4.2 is defined as KL(p_w(·|q,c) || p_orig(·|q,c)). This assumes the original model, which still encodes the forgotten facts, is exactly what a properly unlearned model should imitate when facts are supplied in context. This is a design choice, not a benchmark requirement.
  • domain assumption The Contextual QA evaluation set is independent of the S_ctx^f set used to train the context term, or any overlap has negligible effect.
    No held-out contextual split is described in §5 or Appendix A.4. If the same forget-set questions are used in both the loss and the metric, the reported gains are in-sample. The paraphrased/reasoning case study in §6.2 is the only out-of-distribution evidence and is anecdotal.
  • domain assumption LLM-Judge (Claude 3.5 Sonnet v2) binary factual-equivalence scores are a reliable measure of answer quality.
    All headline metrics rely on this judge (Appendix A.1.2). No inter-annotator agreement, calibration, or human validation is reported, and the judge is also used in the λ_c selection criterion.
  • domain assumption TOFU's synthetic author profiles are a valid testbed for conclusions about real-world sensitive or outdated knowledge removal.
    The paper generalizes to real deployments from a synthetic benchmark (Maini et al., 2024). Mechanisms on synthetic data may not fully capture unlearning behavior on real factual knowledge, though TOFU is a standard benchmark in the field.

pith-pipeline@v1.3.0-alltime-deepseek · 13856 in / 17177 out tokens · 141795 ms · 2026-08-04T09:00:06.142184+00:00 · methodology

0 comments
read the original abstract

Large language models may encode sensitive information or outdated knowledge that needs to be removed, to ensure responsible and compliant model responses. Unlearning has emerged as an efficient alternative to full retraining, aiming to remove specific knowledge while preserving overall model utility. Existing evaluations of unlearning methods focus on (1) the extent of forgetting of the target knowledge (forget set) and (2) maintaining performance on the retain set (i.e., utility). However, these evaluations overlook an important usability aspect: users may still want the model to leverage the removed information if it is re-introduced in the prompt. In a systematic evaluation of six state-of-the-art unlearning methods, we find that they consistently impair such contextual utility. To address this, we augment unlearning objectives with a plug-in term that preserves the model's ability to use forgotten knowledge when it is present in context. Extensive experiments demonstrate that our approach restores contextual utility to near original levels while still maintaining effective forgetting and retain-set utility.

Figures

Figures reproduced from arXiv: 2510.17620 by Amir Houmansadr, Dezhi Hong, Megan Ganji, Mingxian Wang, Parnian Afshar, Thomas Butler, Yuefeng Peng.

Figure 1
Figure 1. Figure 1: Overview of our settings. (a) Apply unlearning to remove the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Contextual QA performance across metrics (ROUGE-L, LLM-judge, and utility) for un [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Examples used in context-aware unlearning. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation on forget ratio for Gemma-2B-IT. For each ratio (1%, 5%, 10%), we report [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Template for Contextual QA, where the model is given both the context and the question [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Template for LLM-Judge, which evaluates whether the model answer matches the refer [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Direct QA results for the 5% forget set. Each row corresponds to a model ( [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Contextual QA results for Gemma-2B-IT at 1% and 10% forget ratios. Each row shows [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: λ-ablation on the 5% forget set. Each row corresponds to one unlearning method (top to bottom: NPO, RMU, UNDIAL). Within each row, the subplots report Direct QA performance, Contextual QA performance, and Model Utility. forgetting and overall utility, while substantially improving contextual utility toward the ideal level. For example, across all three methods, Contextual QA performance steadily increases … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning

    cs.LG 2026-05 unverdicted novelty 6.0

    Distinguishable Deletion unifies knowledge erasure and refusal for LLM unlearning via an energy index that enforces boundaries during training and enables refusal at inference.

Reference graph

Works this paper leans on

35 extracted references · 6 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Many-shot in-context learning

    Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37: 0 76930--76966, 2024

    2024

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  3. [3]

    Extracting training data from large language models

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pp.\ 2633--2650, 2021

    2021

  4. [4]

    Lift yourself up: Retrieval-augmented text generation with self-memory

    Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. Lift yourself up: Retrieval-augmented text generation with self-memory. Advances in Neural Information Processing Systems, 36, 2024

    2024

  5. [5]

    Machine unlearning doesn't do what you think: Lessons for generative ai policy, research, and practice

    A Feder Cooper, Christopher A Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, et al. Machine unlearning doesn't do what you think: Lessons for generative ai policy, research, and practice. arXiv preprint arXiv:2412.06966, 2024

    arXiv 2024

  6. [6]

    Undial: Self-distillation with adjusted logits for robust unlearning in large language models

    Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vuli \'c . Undial: Self-distillation with adjusted logits for robust unlearning in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)...

    2025

  7. [7]

    Openunlearning: Accelerating llm unlearning via unified benchmarking of methods and metrics

    Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zachary C Lipton, J Zico Kolter, and Pratyush Maini. Openunlearning: Accelerating llm unlearning via unified benchmarking of methods and metrics. arXiv preprint arXiv:2506.12618, 2025

    arXiv 2025

  8. [8]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp.\ arXiv--2407, 2024

    2024

  9. [9]

    Foundation models and fair use

    Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. Journal of Machine Learning Research, 24 0 (400): 0 1--79, 2023

    2023

  10. [10]

    Unlearning or obfuscating? jogging the memory of unlearned LLM s via benign relearning

    Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Unlearning or obfuscating? jogging the memory of unlearned LLM s via benign relearning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=fMNRYBvcQN

    2025

  11. [11]

    On information and sufficiency

    Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22 0 (1): 0 79--86, 1951

    1951

  12. [12]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 0 9459--9474, 2020

    2020

  13. [13]

    The wmdp benchmark: Measuring and reducing malicious use with unlearning

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. In International Conference on Machine Learning, pp.\ 28525--28550. PMLR, 2024

    2024

  14. [14]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  15. [15]

    Tofu: A task of fictitious unlearning for llms

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. In First Conference on Language Modeling, 2024

    2024

  16. [16]

    Zhang, and Nicolas Papernot

    Andrei Ioan Muresanu, Anvith Thudi, Michael R. Zhang, and Nicolas Papernot. Fast exact unlearning for in-context learning data for LLM s. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=TzNVZEsqTi

    2025

  17. [17]

    Scalable extraction of training data from aligned, production language models

    Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Florian Tram \`e r, and Katherine Lee. Scalable extraction of training data from aligned, production language models. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  18. [18]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  19. [19]

    In-context unlearning: Language models as few-shot unlearners

    Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few-shot unlearners. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine ...

    2024

  20. [20]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 0 53728--53741, 2023

    2023

  21. [21]

    A systematic survey of prompt engineering in large language models: Techniques and applications

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Smith, and Chiyuan Zhang

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE : Machine unlearning six-way evaluation for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=TArmA033BU

    2025

  23. [23]

    Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai

    Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai. arXiv preprint arXiv:2407.00106, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Rethinking LLM unlearning objectives: A gradient perspective and go beyond

    Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, and Kilian Q Weinberger. Rethinking LLM unlearning objectives: A gradient perspective and go beyond. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  27. [27]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

    Pith/arXiv arXiv 2025

  28. [28]

    Exploring criteria of loss reweighting to enhance LLM unlearning

    Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. Exploring criteria of loss reweighting to enhance LLM unlearning. In Forty-second International Conference on Machine Learning, 2025 b

    2025

  29. [29]

    Large language model unlearning

    Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 37: 0 105425--105475, 2024

    2024

  30. [30]

    Negative preference optimization: From catastrophic collapse to effective unlearning

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling, 2024 a

    2024

  31. [31]

    Raft: Adapting language model to domain specific rag

    Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. Raft: Adapting language model to domain specific rag. arXiv preprint arXiv:2403.10131, 2024 b

    Pith/arXiv arXiv 2024

  32. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  33. [33]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  34. [34]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  35. [35]

    The full name of the author born in Kuwait City, Kuwait on the 8th of September, 1956 is Basil Mahfouz Al-Kuwaiti

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 1956