Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Existing Large Language Model Unlearning Evaluations Are Inconclusive

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that standard evaluations of LLM unlearning are inconclusive because they can inject new information, depend on question format, and rely on spurious correlations.

desk verdict The task-format dependence evidence is solid enough to support 'inconclusive', but the paper's flagship bit-counting argument for information injection is unsound and should be reworked before publication. read the letter →

arxiv 2506.00688 v1 pith:3VKE5EKL submitted 2025-05-31 cs.LG cs.AIcs.CL

Zhili Feng , Yixuan Even Xu , Alexander Robey , Robert Kirk , Xander Davies , Yarin Gal , Avi Schwarzschild , J. Zico Kolter This is my paper
classification cs.LGcs.AIcs.CL
keywords machineunlearninglargelanguagemodelsevaluationadversarialattacksinformationinjectionmultiple-choiceaccuracymemorizationdetectiontasksensitivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current benchmarks for checking whether a large language model has truly unlearned sensitive knowledge do not support the conclusions drawn from them. It identifies three failure modes: some evaluation procedures feed new information into the model during testing, results swing with the task format, and datasets contain spurious correlations that let models appear to relearn knowledge they never saw. Because of these problems, the same unlearned model can look successfully erased under one protocol and completely unerased under another. The paper proposes two principles for future evaluations: minimal information injection and downstream task awareness. If the argument holds, existing claims about shallow unlearning, and also claims about successful unlearning, need to be re-examined.

What carries the argument

The central machinery is the evaluation protocol itself, examined through three representative instruments: finetuning attacks, which modify weights on a small number of samples; input-space attacks, represented by Enhanced GCG, which optimizes a 100-token prefix prepended to forget-set prompts; and the adversarial compression ratio, the ratio of a string's length to the shortest prompt that makes the model generate it. A bit-capacity calculation compares the prefix's roughly 1,500 bits with the roughly 1,430 bits needed to encode 55 percent of the WMDP-Bio multiple-choice answers, motivating the charge that input-space attacks can inject, not merely elicit, information. These instruments are used to argue that no current evaluation cleanly separates pre-existing knowledge from evaluation artifacts.

What would settle it

Optimize an Enhanced GCG prefix on a handful of WMDP-Bio samples and evaluate it on held-out questions from the same dataset that were never used during optimization: if the prefix's accuracy gain largely disappears on those held-out questions, the injection account is supported, whereas if the gain persists, the prefix is eliciting latent knowledge and the information-injection violation is not established for input-space attacks.

Watch

Extended reading notes

Core claim

The paper's central claim is that current evaluation protocols may both overstate and understate unlearning success. It reaches this conclusion by showing three things: finetuning attacks and input-space attacks can inject substantial new information into the model during testing; evaluation outcomes shift with task format, such as multiple-choice letter probability, multiple-choice text probability, and open-ended generation; and benchmarks such as TOFU contain spurious correlations that allow models to generalize from retain data to unseen forget data. The paper therefore rejects the conclusiveness of standard adversarial unlearning evaluations and proposes two principles for future evaluations, minimal information injection and downstream task awareness, validated by targeted experiments showing how violations of each produce misleading conclusions.

Load-bearing premise

The paper's information-injection critique of input-space attacks rests on the premise that the accuracy gained from an optimized 100-token prefix must be explained by information encoded in that prefix, measured by counting its bits, rather than by the prefix eliciting latent knowledge the model already possesses.

Editorial extensions

If this is right

  • If finetuning attacks can inject information, then a model that fails such an attack may simply have been re-taught the forget set during testing, so the failure does not prove the knowledge was retained.
  • If multiple-choice accuracy depends on whether answers are scored by letter probability or text probability, then benchmark numbers from different papers are not directly comparable, and reported unlearning success rates carry an unstated format assumption.
  • If TOFU-type benchmarks contain spurious correlations, then relearning on retain data improving forget-set accuracy does not establish that unlearning failed, since the improvement can come from benchmark artifacts rather than recovered knowledge.
  • If conclusions flip across tasks, then unlearning should be reported as a cross-format leakage matrix rather than a single accuracy number, and a model that leaks in any format has not truly forgotten.
  • If a memorization detector gives a positive result after unlearning, that is strong evidence the information is still present, making memorization detectors a useful high bar even though a negative result remains inconclusive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bit-capacity estimate is best read as a rough upper bound: a single shared 100-token prefix optimized on fewer than ten samples cannot store roughly 1,430 sample-specific answer bits, so the information-injection charge against input-space attacks would be stronger if paired with a per-sample transfer test on held-out questions.
  • The two principles imply that future benchmarks should report an explicit injection budget and a cross-format leakage matrix, which would make unlearning claims more actionable for regulators and deployment decisions.
  • If task-format sensitivity is as large as reported, unlearning success should be evaluated over a distribution of downstream tasks rather than a single multiple-choice score, and forget sets should be constructed to be independent of retain-set features to avoid spurious generalization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper argues that current adversarial LLM unlearning evaluations are inconclusive, identifying three failure modes: (i) finetuning and input-space attacks can inject new information into the model rather than elicit forgotten knowledge; (ii) evaluation outcomes depend heavily on the task format (MCQ letter probability, MCQ text probability, open-ended generation); and (iii) benchmarks such as TOFU contain spurious correlations that may allow relearning of the forget set from retain data. The authors propose two principles for future evaluations, minimal information injection and downstream task awareness, and support their critique with experiments on WMDP and TOFU using RMU and NPO unlearned models, Enhanced GCG attacks, and adversarial compression ratio (ACR) memorization detectors.

Significance. The task-format-dependence results, especially the ACR comparisons in Figure 7 and the Enhanced GCG open-ended versus MCQ contrast in Figure 5, are a useful and likely robust contribution to the unlearning evaluation literature. If the spurious-correlation claim about TOFU holds, it would complicate the interpretation of a widely used benchmark. The paper also proposes concrete reporting recommendations that could improve future evaluations. However, the information-injection claim for input-space attacks is not currently established: the bit-capacity argument in Section 4.2 is invalid as stated, and the supporting prefix-transfer experiment is ambiguous. The broader 'inconclusive' thesis is defensible through the task-dependence evidence alone, but the paper's first stated pitfall needs either a rigorous repair or a narrower restatement.

major comments (4)
  1. [§4.2, bit-capacity calculation] The computation 1300 × log2(4) × 0.55 ≈ 1430 bits is not the information required for a single shared 100-token prefix to reach 55% accuracy on WMDP-Bio. Because the prefix is fixed across all test questions, it cannot store per-question answer identities; the relevant quantity is the mutual information between the prefix and the labels achievable by the optimization procedure, which is bounded by the size of the optimization set (fewer than ten samples) and the complexity of the hypothesis class, not by the test-set answer entropy. The comparison with 100 × log2(32000) ≈ 1500 bits therefore does not establish that the observed accuracy gain can be explained by information injected through the prefix. This is load-bearing for the paper's first claimed pitfall, so the argument should be replaced or removed.
  2. [§4.2, overfitting experiment] The experiment that finetunes Llama-3.2-1B on WMDP-Bio to overfitting, optimizes an Enhanced GCG prefix on seven samples, and then reports improved accuracy on the original base model does not demonstrate that the prefix injects new knowledge. The base model's parameters are unchanged, so the accuracy gain from 39.9% to 53.8% must be explained either by latent knowledge already present in the base model or by a generic answer-selection heuristic induced by the prefix; both alternatives are consistent with the reported numbers, and the latter is exactly the possibility the paper intends to rule out. A control on questions whose answers were never present in any model used during optimization is needed, along with an analysis of what the optimized prefix encodes.
  3. [§4.1, Figure 2] The TOFU-MCQ finetuning experiment is presented as evidence that TOFU contains spurious correlations that let retain-set finetuning reintroduce forget-set knowledge. However, finetuning on a subset of TOFU-MCQ and evaluating on held-out questions about different fictitious authors may simply improve the model's ability to answer MCQs in the benchmark's format; it does not demonstrate that specific forget-set facts were relearned. The experiment should be embedded in an actual unlearning setup with disjoint forget and retain authors, and it should include a control that isolates format learning (e.g., finetuning on an unrelated MCQ dataset and measuring the same held-out accuracy gain). This does not invalidate the task-dependence findings in §4.3, but it weakens the first claimed pitfall.
  4. [§4.3 and Appendix B, ACR thresholds] The success thresholds for the ACR evaluations in Figure 7 are selected after inspecting the optimization behavior on the same data: Appendix B states that thresholds are chosen such that further increasing them does not boost the success probability significantly. This data-dependent threshold choice makes the reported success rates difficult to interpret, and the qualitative comparison across tasks should be accompanied by a sensitivity analysis over threshold values or by a pre-specified thresholding rule defined on a held-out set. Without this, the conclusion that the relative effectiveness of RMU and NPO reverses across tasks rests on an arbitrary element of the evaluation protocol.
minor comments (6)
  1. [Figures 2, 4, 5, 6] These figures report point estimates without error bars or seed counts; adding error bars or stating that each curve is a single run would clarify the reliability of the comparisons, especially in Figure 4 where the curves are close.
  2. [§3.1, Eq. (1)] The notation Pr(x||y; M_U) is confusing; it appears to denote the probability that the model generates the target knowledge y given the prompt prefix x, but the conditional dependence is written backwards. Please clarify the notation.
  3. [§3.1, Eq. (2)] The definition of ACR says M(x)=y under greedy decoding, which is a deterministic equality, but §4.3 describes the ACR evaluation as finding a suffix that maximizes the probability of generating the correct answer. These two characterizations are inconsistent and should be reconciled.
  4. [Abstract and §5] The abstract says the principles are 'validated' through experiments, but the paper does not implement a downstream-aware evaluation protocol; it demonstrates pitfalls and gives recommendations. Consider softening 'validated' to 'illustrated' or 'motivated'.
  5. [§5.1, Recommendation 1] The proposed 'injection budget' is described only heuristically, and the paper's own bit-based measure is shown to be unreliable. A more concrete definition of how an injection budget would be computed and enforced would strengthen this recommendation.
  6. [Table 1] The table reports 40%, 50%, and 60% percentiles of ACR, but no explanation is given for why these particular percentiles are chosen; a brief justification would help the reader interpret the summary.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the central critique rests on independent experiments, with only minor self-citations and heuristic supporting arguments that do not reduce the conclusion to its inputs.

full rationale

The paper's central claim that current unlearning evaluations are inconclusive is supported by empirical demonstrations that are not derived from the paper's own assumptions. The task-format dependence results in Figures 5-7 compare MCQ scoring modes and open-ended generation on the same models and datasets; these are self-contained measurements. The TOFU spurious-generalization experiment in Figure 2 is a direct finetune-and-holdout measurement, and the finetuning-attack format-dependence experiments in Figures 4 and 6 are also empirical comparisons. None of these steps fit the enumerated circularity patterns: no parameter is fitted and then renamed as a prediction, no cited theorem is invoked to forbid alternatives, and no result is equivalent by construction to its input. The ACR metric [7] and TOFU dataset [28] are self-citations by the current authors, but they are used as tools rather than as load-bearing justification; the paper's conclusions do not reduce to the correctness of these cited works. The Section 4.2 information-injection argument for input-space attacks is heuristic: comparing the bit capacity of a 100-token shared prefix (roughly 1500 bits) with a per-question answer-entropy estimate (roughly 1430 bits) is not a derivation, and the supporting overfitting experiment does not cleanly separate elicitation from injection. This is a correctness or statistical-validity limitation, not circularity in the paper's own equations. The paper also explicitly acknowledges its limited experimental scope in the Broader Impact section, noting the small number of unlearning methods and academic-style benchmarks. Accordingly, no circular step is established; the score of 2 reflects only the presence of minor self-citations and the heuristic nature of one supporting argument, while the central claim retains independent content. A score of 0 would also be defensible if the minor self-citations are disregarded entirely, as they are not load-bearing for the main conclusion.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. It relies on existing benchmarks (TOFU, WMDP), existing metrics (ACR), and standard assumptions about LLM capacity and attack hierarchies. The main hand-chosen parameter is the ACR success threshold, which affects conclusions in Figure 7.

free parameters (1)
  • ACR success threshold (suffix length limit) = task-specific, chosen so that further increasing does not boost success probability (Appendix B)
    The ACR evaluation in Figure 7 classifies a sample as successfully generated if the minimal suffix length is below a hand-picked threshold. Thresholds are selected based on observed saturation, which can affect the relative ranking of methods.
assumptions (3)
  • domain assumption LLMs are universal sequence approximators
    Invoked in Section 4.2 to argue that a well-optimized prompt could in principle inject new information, citing [52].
  • domain assumption ACR is a valid memorization detector and a high bar for unlearning evaluation
    The paper adopts adversarial compression ratio from Schwarzschild et al. [7] (self-cited) and recommends it as a yardstick in Recommendation 3.
  • domain assumption Finetuning attacks establish an upper bound on success of other adversarial evaluations
    Cited from Che et al. [5] in Section 3.1 to justify focusing on finetuning and input-space attacks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Existing Large Language Model Unlearning Evaluations Are Inconclusive." pith.science (2026). https://pith.science/paper/3VKE5EKL

@misc{pith2026250600688,
  author       = {Pith},
  title        = {Pith review of: Existing Large Language Model Unlearning Evaluations Are Inconclusive},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3VKE5EKL}},
  note         = {Machine review of arXiv:2506.00688}
}
read the original abstract

Machine unlearning aims to remove sensitive or undesired data from large language models. However, recent studies suggest that unlearning is often shallow, claiming that removed knowledge can easily be recovered. In this work, we critically examine standard unlearning evaluation practices and uncover key limitations that shake our trust in those findings. First, we show that some evaluations introduce substantial new information into the model, potentially masking true unlearning performance by re-teaching the model during testing. Second, we demonstrate that evaluation outcomes vary significantly across tasks, undermining the generalizability of current evaluation routines. Finally, we find that many evaluations rely on spurious correlations, making their results difficult to trust and interpret. Taken together, these issues suggest that current evaluation protocols may both overstate and understate unlearning success. To address this, we propose two principles for future unlearning evaluations: minimal information injection and downstream task awareness. We validate these principles through a series of targeted experiments, showing how violations of each can lead to misleading conclusions.

Figures

Figures reproduced from arXiv: 2506.00688 by the authors.

Figure 1
Figure 1. The evaluation cycle of unlearned models. Adversarial unlearning evaluations seek to determine whether an unlearned model retains sensitive data after unlearning. In this paper, we show that existing evaluations offer inconclusive results and we propose two principles to improve reliability and ensure future evaluations more accurately reflect true forgetting. for the diversity of model use cases, and rely on spurio… view at source ↗
Figure 2
Figure 2. Evidence of spurious generalization. Finetuning attacks significantly improve accuracy on TOFU’s test set. This finding indicates that it may be possible to spuriously generalize between the retain and forget sets in finetuning-based un￾learning evaluations. Learning Path Model Init Base Model Unlearning Path Unlearned Model Relearning Path Relearned Model Subopt. Relearned Model [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
Figure 5
Figure 5. Enhanced GCG task dependence. The effectiveness of Enhanced GCG varies widely across downstream tasks. Whereas this method nearly recovers the base model’s ac￾curacy for maximum letter probability gener￾ation, it offers little improvement in accuracy for maximum text generation tasks. formats match. This implies that successful evaluations may require prior knowledge about the original unlearning algorithm. 4.2 Inpu… view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Finetuning attacks across MCQ formats. While finetuning attacks computed for maximum letter probability generation recover the base model’s performance, analogous attacks on the maximum text probability do not yield similar accuracy recovery. we are given a model M, an…
Figure 7
Figure 7. Figure 7: Inconclusive ACR evaluations on WMDP-Bio. Success rates of different models on different datasets and tasks. The error bars show the standard errors. For each task, the conclusion about the relative effectiveness of unlearning methods differ. Top: CHOOSE task. The succ…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A new method, Partial Model Collapse, iteratively fine-tunes an LLM on its own self-generated responses to conditionally collapse its output distribution on forget queries, removing private answers without the true la...

Reference graph

Works this paper leans on

59 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Are aligned neural networks adversarially aligned?Advances in Neural Information Processing Systems, 36, 2024

    Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned?Advances in Neural Information Processing Systems, 36, 2024

    2024

  2. [2]

    Rethinking machine unlearning for large language models.Nature Machine Intelligence, pages 1–14, 2025

    Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models.Nature Machine Intelligence, pages 1–14, 2025

    work page 2025

  3. [3]

    The wmdp benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218, 2024

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218, 2024

    arXiv 2024

  4. [4]

    Negative preference optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868, 2024

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868, 2024

    arXiv 2024

  5. [5]

    Model manipulation attacks enable more rigorous evaluations of llm capabilities

    Zora Che, Stephen Casper, Anirudh Satheesh, Rohit Gandikota, Domenic Rosati, Stewart Slocum, Lev E McKinney, Zichu Wu, Zikui Cai, Bilal Chughtai, et al. Model manipulation attacks enable more rigorous evaluations of llm capabilities. InNeurips Safe Generative AI Workshop, 2024

    work page 2024

  6. [6]

    An adversarial perspective on machine unlearning for ai safety.arXiv preprint arXiv:2409.18025, 2024

    Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for ai safety.arXiv preprint arXiv:2409.18025, 2024

    arXiv 2024

  7. [7]

    Rethinking llm memorization through the lens of adversarial compression.arXiv preprint arXiv:2404.15146, 2024

    Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Rethinking llm memorization through the lens of adversarial compression.arXiv preprint arXiv:2404.15146, 2024

    arXiv 2024

  8. [8]

    Machine unlearning: Solutions and challenges

    Jie Xu, Zihan Wu, Cong Wang, and Xiaohua Jia. Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024

    2024

Show all 59 references
  1. [9]

    On the necessity of auditable algorithmic definitions for machine unlearning

    Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In31st USENIX security symposium (USENIX Security 22), pages 4007–4022, 2022

    2022

  2. [10]

    Arcane: An efficient architecture for exact machine unlearning

    Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. Arcane: An efficient architecture for exact machine unlearning. InIJCAI, volume 6, page 19, 2022

    2022

  3. [11]

    Certified data removal from machine learning models.arXiv preprint arXiv:1911.03030, 2019

    Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models.arXiv preprint arXiv:1911.03030, 2019

    1911 arXiv

  4. [12]

    Amnesiac machine learning

    Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516–11524, 2021

    2021

  5. [13]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  6. [14]

    Position: Llm unlearning benchmarks are weak measures of progress.arXiv preprint arXiv:2410.02879, 2024

    Pratiksha Thaker, Shengyuan Hu, Neil Kale, Yash Maurya, Zhiwei Steven Wu, and Virginia Smith. Position: Llm unlearning benchmarks are weak measures of progress.arXiv preprint arXiv:2410.02879, 2024

    2024 arXiv

  7. [15]

    A probabilistic perspective on unlearning and alignment for large language models.arXiv preprint arXiv:2410.03523, 2024

    Yan Scholten, Stephan Günnemann, and Leo Schwinn. A probabilistic perspective on unlearning and alignment for large language models.arXiv preprint arXiv:2410.03523, 2024

    2024 arXiv

  8. [16]

    Regulation (eu) 2016/679 of the european parliament and of the council

    European Union. Regulation (eu) 2016/679 of the european parliament and of the council. Official Journal of the European Union, 2016

    2016

  9. [17]

    UK General Data Protection Regulation (UK GDPR), 2021

    UK Government. UK General Data Protection Regulation (UK GDPR), 2021. URL https: //www.gov.uk/data-protection. Post-Brexit adaptation of the EU GDPR, enforced by the UK Information Commissioner’s Office

    2021

  10. [18]

    Ccpa regulations: Final regulation text.Office of the Attorney General, California Department of Justice, 2021

    CA OAG. Ccpa regulations: Final regulation text.Office of the Attorney General, California Department of Justice, 2021

    2021

  11. [19]

    Bill C-27: Digital Charter Implementation Act, 2022 – Consumer Privacy Protection Act (CPPA), 2022

    Government of Canada. Bill C-27: Digital Charter Implementation Act, 2022 – Consumer Privacy Protection Act (CPPA), 2022. URL https://www.parl.ca/DocumentViewer/en/ 44-1/bill/C-27/first-reading . Proposed legislation to modernize Canadian privacy law, including rights to erasu...

    2022

  12. [20]

    Machine unlearning via algorithmic stability

    Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. InConference on Learning Theory, pages 4126–4142. PMLR, 2021

    2021

  13. [21]

    Machine unlearning

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE, 2021

    2021

  14. [22]

    Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations

    Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. InComputer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI...

    2020

  15. [23]

    Making ai forget you: Data deletion in machine learning.Advances in neural information processing systems, 32, 2019

    Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning.Advances in neural information processing systems, 32, 2019

    2019

  16. [24]

    The algorithmic foundations of differential privacy.Founda- tions and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014

    Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy.Founda- tions and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014

    2014

  17. [25]

    Approximate data deletion from machine learning models

    Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. InInternational Conference on Artificial Intelligence and Statistics, pages 2008–2016. PMLR, 2021

    2008

  18. [26]

    Remember what you want to forget: Algorithms for machine unlearning.Advances in Neural Information Processing Systems, 34:18075–18086, 2021

    Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning.Advances in Neural Information Processing Systems, 34:18075–18086, 2021

    2021

  19. [27]

    Who’s harry potter? approximate unlearning in llms

    Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023

    2023 arXiv

  20. [28]

    Tofu: A task of fictitious unlearning for llms.arXiv preprint arXiv:2401.06121, 2024

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms.arXiv preprint arXiv:2401.06121, 2024

    2024 arXiv

  21. [29]

    Benchmarking vision language model unlearning via fictitious facial identity dataset, 2024

    Yingzi Ma, Jiongxiao Wang, Fei Wang, Siyuan Ma, Jiazhao Li, Xiujun Li, Furong Huang, Lichao Sun, Bo Li, Yejin Choi, Muhao Chen, and Chaowei Xiao. Benchmarking vision language model unlearning via fictitious facial identity dataset, 2024. URL https://arxiv.org/abs/ 2411.03554

    2024 arXiv

  22. [30]

    Muse: Machine unlearning six-way evaluation for language models.arXiv preprint arXiv:2407.06460, 2024

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models.arXiv preprint arXiv:2407.06460, 2024. 11

    2024 arXiv

  23. [31]

    Rwku: Benchmarking real-world knowledge unlearning for large language models.arXiv preprint arXiv:2406.10890, 2024

    Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models.arXiv preprint arXiv:2406.10890, 2024

    2024 arXiv

  24. [32]

    Open problems in machine unlearning for ai safety.arXiv preprint arXiv:2501.04952, 2025

    Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O’Gara, Robert Kirk, Ben Bucknall, Tim Fist, et al. Open problems in machine unlearning for ai safety.arXiv preprint arXiv:2501.04952, 2025

    2025 arXiv

  25. [33]

    Jailbreakbench: An open robustness benchmark for jailbreaking large language models

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv pre...

    2024 arXiv

  26. [34]

    Simplicity prevails: Rethinking negative preference optimization for llm unlearning.arXiv preprint arXiv:2410.07163, 2024

    Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for llm unlearning.arXiv preprint arXiv:2410.07163, 2024

    2024

  27. [35]

    Improving alignment and robustness with circuit breakers

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  28. [36]

    Fast yet effective machine unlearning.IEEE Transactions on Neural Networks and Learning Systems, 2023

    Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning.IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  29. [37]

    Self- destructing models: Increasing the costs of harmful dual uses of foundation models

    Peter Henderson, Eric Mitchell, Christopher Manning, Dan Jurafsky, and Chelsea Finn. Self- destructing models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 287–296, 2023

    2023

  30. [38]

    Tamper-resistant safeguards for open-weight llms.arXiv preprint arXiv:2408.00761, 2024

    Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, et al. Tamper-resistant safeguards for open-weight llms.arXiv preprint arXiv:2408.00761, 2024

    2024 arXiv

  31. [39]

    Do unlearning methods remove information from language model weights?arXiv preprint arXiv:2410.08827, 2024

    Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights?arXiv preprint arXiv:2410.08827, 2024

    2024 arXiv

  32. [40]

    Eight methods to evaluate robust unlearning in llms.arXiv preprint arXiv:2402.16835, 2024

    Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms.arXiv preprint arXiv:2402.16835, 2024

    2024 arXiv

  33. [41]

    Llm defenses are not robust to multi-turn human jailbreaks yet.arXiv preprint arXiv:2408.15221, 2024

    Nathaniel Li, Ziwen Han, Ian Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, and Summer Yue. Llm defenses are not robust to multi-turn human jailbreaks yet.arXiv preprint arXiv:2408.15221, 2024

    2024 arXiv

  34. [42]

    Scalable extraction of training data from (production) language models.arXiv preprint arXiv:2311.17035, 2023

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ip- polito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models.arXiv preprint arXiv:2311....

    2023 arXiv

  35. [43]

    Preventing generation of verba- tim memorization in language models gives a false sense of privacy

    Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing generation of verba- tim memorization in language models gives a false sense of privacy. InProceedings of the 16th Inter...

    2023

  36. [44]

    Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning

    Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning. InProceedings of the International Conference on Learning Representations (ICLR), 2025. URL https://arxiv. org/abs/2406.13356. a...

    2025 arXiv

  37. [45]

    Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419, 2023

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419, 2023. 12

    2023 arXiv

  38. [46]

    Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

    Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  39. [47]

    Jailbreaking llm-controlled robots.arXiv preprint arXiv:2410.13691, 2024

    Alexander Robey, Zachary Ravichandran, Vijay Kumar, Hamed Hassani, and George J Pappas. Jailbreaking llm-controlled robots.arXiv preprint arXiv:2410.13691, 2024

    2024 arXiv

  40. [48]

    Flrt: Fluent student-teacher redteaming.arXiv preprint arXiv:2407.17447, 2024

    T Ben Thompson and Michael Sklar. Flrt: Fluent student-teacher redteaming.arXiv preprint arXiv:2407.17447, 2024

    2024 arXiv

  41. [49]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment, 2023

    2023

  42. [50]

    Textbooks are all you need ii: phi-1.5 technical report, 2023

    Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023. URL https://arxiv.org/ abs/2309.05463

    2023 arXiv

  43. [51]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  44. [52]

    Are transformers universal approximators of sequence-to-sequence functions?arXiv preprint arXiv:1912.10077, 2019

    Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions?arXiv preprint arXiv:1912.10077, 2019

    1912 arXiv

  45. [53]

    Inside-out: Hidden factual knowledge in llms.arXiv preprint arXiv:2503.15299, 2025

    Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. Inside-out: Hidden factual knowledge in llms.arXiv preprint arXiv:2503.15299, 2025

    2025 arXiv

  46. [54]

    Information-theoretic probing for linguistic structure.arXiv preprint arXiv:2004.03061, 2020

    Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-theoretic probing for linguistic structure.arXiv preprint arXiv:2004.03061, 2020

    2004 arXiv

  47. [55]

    Quantifying emergence in large language models.arXiv preprint arXiv:2405.12617, 2024

    Hang Chen, Xinyu Yang, Jiaying Zhu, and Wenya Wang. Quantifying emergence in large language models.arXiv preprint arXiv:2405.12617, 2024

    2024 arXiv

  48. [56]

    Measuring and modifying factual knowledge in large language models

    Pouya Pezeshkpour. Measuring and modifying factual knowledge in large language models. In2023 International Conference on Machine Learning and Applications (ICMLA), pages 831–838. IEEE, 2023

    2023

  49. [57]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.arXiv preprint arXiv:2310.11324, 2023

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.arXiv preprint arXiv:2310.11324, 2023

    arXiv

  50. [58]

    Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882, 2023

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882, 2023

    2023 arXiv

  51. [59]

    Does prompt formatting have any impact on llm performance?arXiv preprint arXiv:2411.10541, 2024

    Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. Does prompt formatting have any impact on llm performance?arXiv preprint arXiv:2411.10541, 2024. 13 A Additional Experiments In this section, we present additional results in the set of e...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.