Pith. sign in

REVIEW 5 major objections 6 minor 4 cited by

Tool Unlearning for Tool-Augmented LLMs

T0 review · 5 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ToolDelete removes a tool-augmented LLM's ability to call specific APIs while preserving other knowledge and skills.

desk verdict Genuine new task, plausible first method, but the deletion guarantee is not yet established and the experimental reporting needs tightening. read the letter →

arxiv 2502.01083 v2 pith:G3M4JCWQ submitted 2025-02-03 cs.LG cs.AIcs.CL

Jiali Cheng , Hadi Amiri This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords toolunlearningmachinetool-augmentedlargelanguagemodelsparametricknowledgedeletiontaskarithmeticmembershipinferenceattackAPIsafetysequential
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Tool-augmented large language models learn to call external tools and APIs, and this paper argues that selectively removing that learned ability—tool unlearning—is a missing and practically needed machine-unlearning capability. Existing unlearning methods forget individual samples, not skills, so the paper introduces ToolDelete, which trains the model to imitate a tool-free version of itself on queries for the deleted tool while keeping its responses on retained tools and adding back general knowledge through task arithmetic. On multiple tool-learning datasets and 7B-scale tool-augmented LLMs, ToolDelete is reported to outperform adapted general and LLM-specific unlearning baselines on forgetting, retention, and general capability, while saving about 75% of retraining time. The paper also contributes LiRA-Tool, a membership-inference-style probe that uses synthesized shadow samples to test whether tool knowledge has actually been removed.

What carries the argument

The load-bearing mechanism is the use of a tool-free reference model $f_0$ as a deletion target: instead of destroying specific training pairs, the unlearned model is supervised to reproduce $f_0$'s answers on the queries of the forgotten tool, which defines 'no tool knowledge' in a measurable way. Around that, Equation (5) combines deletion and retention losses, and Equation (6) adds the task-arithmetic vector $\theta_0 - \theta_R$ to preserve general capabilities. Evaluation is carried by LiRA-Tool, which builds shadow distributions of tool queries with GPT-4-generated samples and applies a likelihood-ratio test to see whether a tool is still detectable in the model.

What would settle it

Generate a held-out set of queries for a forgotten tool that deliberately differ in wording, argument passing, and API-call style from both the training demonstrations and the GPT-4 shadow samples, then measure how often the unlearned model still emits a valid tool call. If the tool-call rate on these out-of-distribution queries remains substantial while the reported forget-set accuracy is low, the deletion is incomplete.

Watch

Extended reading notes

Core claim

The paper's central claim is that tool knowledge can be deleted from an LLM's parameters by enforcing three properties: Tool Knowledge Deletion ($g(f_0, t_i) - g(f', t_i) \geq 0$), Tool Knowledge Retention ($g(f, t_m) - g(f', t_m) \approx 0$), and General Capability Retention through task arithmetic. Concretely, ToolDelete optimizes the unlearned model $\theta'$ so that on forget-tool demonstrations the output matches the tool-free model $f_0$'s responses, on retained-tool demonstrations the output matches the original tool-augmented model $f$'s responses, and then applies the shift $\theta'^* = \theta' + \alpha(\theta_0 - \theta_R)$. The authors report that this yields effective forgetting and retention for randomly selected and class-specific tools at a fraction of retraining cost, and they argue that existing sample-level unlearning methods fail on this task because they overfit to training prompts rather than removing the underlying tool-using skill.

Load-bearing premise

The method assumes that matching the tool-free model $f_0$ on the forget-tool demonstrations and shadow samples is sufficient to erase the tool; if $f_0$ itself retains partial tool ability, or if unseen queries use the tool in a different style, the model can still call the tool while passing the reported metrics.

Editorial extensions

If this is right

  • Tool-augmented LLMs can be updated to drop a compromised or deprecated API without retraining from scratch, cutting update time by roughly 75 percent.
  • Sequential deletion requests can be processed incrementally, so newly discovered unsafe tools can be removed as they arise without rebuilding the model.
  • Unlearning can proceed without full access to original training data, using synthesized shadow samples as substitutes.
  • The same three-property formulation works with SFT, DPO, and LoRA, giving deployment options under different compute budgets.
  • LiRA-Tool offers an audit signal for whether tool knowledge persists beyond the exact training prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because deletion is defined relative to $f_0$'s knowledge, any tool ability already present in the base model sets a lower bound on what 'fully unlearned' means; future work should define deletion against a model with no tool-related pretraining.
  • The task-arithmetic step that restores general ability could also re-introduce parts of the forgotten tool if the general-knowledge vector encodes tool-usage patterns, so per-tool reweighting of that vector is a testable extension.
  • The dependence on GPT-4-generated shadow samples means the evaluation inherits the sampling biases of that generator; building shadow distributions from real API documentation or user traffic would test how the method behaves under broader query variation.
  • The framework suggests an automatic API lifecycle manager: as tool providers deprecate versions, an assistant could unlearn the old version and learn the new one in place, without retaining historical data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper defines the task of tool unlearning for tool-augmented LLMs, where the goal is to remove the ability to use specific tools from the parametric knowledge of a model while preserving other tools and general capabilities. The proposed method, ToolDelete, combines three components: matching the tool-free model f0 on forget-tool demonstrations, fine-tuning on retained-tool demonstrations, and task arithmetic for general capability retention. The authors also propose LiRA-Tool, an adaptation of LiRA using GPT-4-generated shadow samples to evaluate tool-level forgetting. Experiments on ToolAlpaca, ToolBench, and API-Bench with three 7B models compare against retraining and seven unlearning baselines. The main claims are that ToolDelete outperforms baselines on forget-tool accuracy while preserving retained and general performance, supports sequential unlearning, operates without access to training data when needed, and is efficient.

Significance. The paper addresses a timely and practically relevant problem: removing a tool or API from a model that was trained to call it. The decomposition into deletion, retention, and general-capability preservation is sensible, and the ablation studies support the contribution of each component. The adaptation of LiRA to a tool-level membership attack is a reasonable first step, and the coverage of multiple tool-augmented models and datasets is a strength. However, the unlearning guarantee is only as strong as the coverage of the forget-set demonstrations and the shadow samples, and the reported membership-attack results show only marginal improvement over retraining. If the robustness and statistical issues are addressed, the paper would be a solid contribution to the unlearning literature.

major comments (5)
  1. [§3.1, Eq. (1); §2] The deletion target in Definition 3.1 is defined as the unlearned model having no more knowledge than the tool-free model f0, but Section 2 explicitly concedes that f0 may already possess some tool-using capabilities (e.g., basic arithmetic). Because Eq. (1) and the training objective in Eq. (5) use f0's outputs as the supervision signal, the method cannot guarantee true tool removal if f0 itself can produce tool-like completions. The paper should either use a reference model verified to be tool-free for the target tools or report f0's residual tool-call rate on a held-out query set to establish that the reference is indeed free of the forgotten tool.
  2. [§5, Fig. 2; §3.5] The reported LiRA-Tool results show that ToolDelete-DPO achieves TPR=0.14 at FPR=0.01, only 0.01 lower than Retrain, and ToolDelete-SFT performs comparably to Retrain. Given that the paper's central claim is that ToolDelete provides stronger unlearning than retraining, this 0.01 difference is not convincing without error bars or multiple seeds. Please report confidence intervals across at least 5 seeds and show the full ROC curves, or justify why TPR@FPR=0.01 is the appropriate operating point for this threat model.
  3. [Tables 1–7 and Figures 2–4] No error bars, standard deviations, or significance tests are reported anywhere in the experimental results. Several headline comparisons are small (e.g., 0.3–1.3 absolute points on TT, 0.3–3.3 on Tr, and the 0.01 TPR difference), so a single run cannot support the claimed improvements over baselines. At a minimum, the main table should report mean ± std over at least 5 random tool subsets with different random seeds.
  4. [Abstract vs. Table 1] The abstract states that ToolDelete outperforms existing general and LLM-specific unlearning algorithms by 12.5 accuracy points on forget tools and 9.1 points on retain tools. In Table 1, the largest Tf improvement over a baseline is 9.8 (ToolDelete-DPO vs. Retrain) and the Tr improvement over the best baseline is only 0.3 (ToolDelete-DPO vs. ICUL). The 12.5/9.1 numbers do not correspond to the reported table and should either be corrected or the source of those numbers clearly identified.
  5. [§5 (without-access variant) and §3.5] The no-access-to-training-data variant in Section 5 says ToolDelete 'can generate pseudo-samples for tools using the shadow samples technique developed for LiRA-Tool,' but Section 3.5 describes shadow samples as an evaluation technique for LiRA-Tool, not as a training-data generator. If the same GPT-4-generated samples are used both to train the no-access variant and to evaluate it with LiRA-Tool, the evaluation becomes circular. Please specify exactly how the pseudo-samples for training are generated and confirm that the training samples and the LiRA-Tool evaluation samples are disjoint.
minor comments (6)
  1. [Throughout] There are several typos and formatting inconsistencies: 'Furthremore' in Section 5, 'in the sue of' in Section 3.5, 'wining responses' in Section 3.4, inconsistent spacing in 'TOOL DELETE ', and inconsistent use of 'Tt' vs. 'TT' in Table 1 and the text.
  2. [Appendix D, Table 3] Table 3 lists the arrows for Tr and Tf incorrectly as Tr(↓) and Tf(↑), which contradicts the metric definitions in Section 4 where Tr should be ↑ and Tf should be ↓.
  3. [Appendix D, Tables 5 and 6] Tables 5 and 6 are identical, which appears to be a copy-paste error; one of them was presumably meant to report results on a different dataset or model (e.g., Gorilla/API-Bench).
  4. [§3.1, Eq. (1)] The knowledge function g in Definition 3.1 is never precisely defined; the later statement that outputs of LLMs are used as their knowledge is vague. Please specify how g is computed (e.g., exact-match on API-call tokens, likelihood of tool invocation) and how it is aggregated across demonstrations.
  5. [§3.3, Eq. (3)] The set of general tasks TG in Definition 3.3 is not enumerated concretely, and the evaluation in Section 4 uses MMLU, BBH, IFEval, and factual knowledge, but the paper does not justify why these tasks are the right instantiation of TG for all three tool-augmented models.
  6. [Appendix E] The shadow-sample generation procedure uses a single prompt template with one in-context example, yet the text claims the samples are diverse in 'prompt format, intention, and difficulty requirements.' The described procedure does not support that claim; please provide details on how multiple templates, difficulty levels, or query phrasings are varied.

Circularity Check

1 steps flagged · score 2.0 of 10

One self-definitional step: the TKD criterion in Eq. (1) is literally the first term of the training objective in Eq. (5); the main empirical claims, however, are supported by held-out and GPT-4-shadow-sample evaluations, so the paper is otherwise self-contained.

  1. self definitional [Section 3.1 Definition 3.1 / Eq. (1) and Section 3.4 Eq. (5)]
    "The unlearned model f′ satisfies tool knowledge deletion if: E ti∈Tf [g(f0, ti) − g(f′, ti)] ≥ 0. (1) ... θ′∗ = arg min θ′ Eti∈Tf [g(f0, ti) − g(f′, ti)] + Etm∈Tr [g(f, tm) − g(f′, tm)] , (5)"

    Definition 3.1 equates successful tool-knowledge deletion with E[g(f0,ti)-g(f',ti)] ≥ 0, and Eq. (5) sets the first term of the training objective to the same expectation, minimized over θ′. Therefore any model that reaches the objective satisfies TKD by construction; the property is the loss function renamed as a criterion rather than an independently measured result. The central empirical claims nevertheless rest on separate held-out Tf/Tr/TT/TG metrics and on LiRA-Tool evaluation with GPT-4 shadow samples, which are not defined as the training loss, so the headline results are not forced by this step.

full rationale

The only genuine construction-level circularity is formal: Definition 3.1 defines successful tool knowledge deletion as E[g(f0,ti)-g(f',ti)] ≥ 0, and Eq. (5) trains f' by minimizing exactly that quantity (plus a retention term). Thus satisfying TKD is tautological once the loss is minimized; it is not an independently measured result. This is a definitional choice rather than a fitted prediction, and it does not by itself drive the paper's stated contributions. The headline comparisons in Table 1 and Tables 5-6 use test/forget/retain sets from ToolAlpaca, ToolBench, and API-Bench that are separate from the demonstrations used in unlearning, and general-capability numbers come from MMLU, BBH, IFEval, etc. LiRA-Tool uses GPT-4-generated shadow samples that are external to the training distribution; the no-access variant repurposes the same generation technique, a limitation the paper does not fully disentangle, but its reported MIA figure is for the with-access setting. I found no load-bearing self-citation: the authors' prior GNNDelete/MultiDelete/Mu-Bench work is cited for similarity or benchmarking, not to justify the tool-unlearning mechanism. No uniqueness theorem or imported ansatz forces the choice of ToolDelete. The acknowledged limitations (f0 may already have some tool-using ability; GPT-4 shadow samples may not cover the true tool-usage space) are generalization risks, not circularity. Overall circularity score: 2.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several domain assumptions about how to measure and erase tool knowledge; no fitted constants beyond training hyperparameters are introduced, and no new physical or conceptual entities are postulated. LiRA-Tool's shadow-sample distribution is an evaluation construct rather than an invented entity.

free parameters (3)
  • alpha (task arithmetic scaling) = not reported
    Controls the magnitude of the (theta0 - thetaR) general-knowledge vector added in Equation 6; chosen by hand and no value is given in the paper.
  • retained-tool subset size for TKR = not reported
    Section 3.2 says the model is fine-tuned on a subset of Tr proportional to Tf for efficiency, but the exact proportion is not specified; it affects retention performance.
  • learning rate = 1e-5
    Global training hyperparameter reported in Appendix C; standard, but part of the recipe needed to reproduce results.
assumptions (5)
  • domain assumption Prompting-based output similarity measures tool knowledge (Definition 3.1).
    The paper equates knowledge about a tool with the LLM's responses to probe queries (Section 3.1); if this measure misses tool-calling behavior, the TKD objective may not capture real deletion.
  • domain assumption f0 is a valid tool-free reference for every forgotten tool.
    The paper acknowledges f0 may already have some tool-using capabilities (Section 2), but still uses f0's outputs as the deletion target; partial prior knowledge would make 'as if never learned' unattainable.
  • domain assumption Task arithmetic addition preserves general capabilities.
    Equation 6 assumes theta0 - thetaR encodes general knowledge whose addition to unlearned weights does not disturb deletion or retention; based on cited task arithmetic work (Ilharco et al., 2023).
  • domain assumption Fine-tuning on a subset of retained demonstrations preserves all retained-tool knowledge.
    Section 3.2 assumes a subset of Tr proportional to Tf suffices for Tool Knowledge Retention; no ablation varying the subset size is given.
  • domain assumption Standard evaluation sets (MMLU, BBH, IFEval, tool-specific success) reliably measure general and tool capabilities.
    The paper relies on benchmark metrics defined by prior datasets; these are standard but not validated for unlearning-specific degradation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tool Unlearning for Tool-Augmented LLMs." pith.science (2026). https://pith.science/paper/G3M4JCWQ

@misc{pith2026250201083,
  author       = {Pith},
  title        = {Pith review of: Tool Unlearning for Tool-Augmented LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3M4JCWQ}},
  note         = {Machine review of arXiv:2502.01083}
}
read the original abstract

Tool-augmented large language models (LLMs) are often trained on datasets of query-response pairs, which embed the ability to use tools or APIs directly into the parametric knowledge of LLMs. Tool-augmented LLMs need the ability to forget learned tools due to security vulnerabilities, privacy regulations, or tool deprecations. However, ``tool unlearning'' has not been investigated in unlearning literature. We introduce this novel task, which requires addressing distinct challenges compared to traditional unlearning: knowledge removal rather than forgetting individual samples, the high cost of optimizing LLMs, and the need for principled evaluation metrics. To bridge these gaps, we propose ToolDelete, the first approach for unlearning tools from tool-augmented LLMs. It implements three key properties to address the above challenges for effective tool unlearning and introduces a new membership inference attack (MIA) model for effective evaluation. Extensive experiments on multiple tool learning datasets and tool-augmented LLMs show that ToolDelete effectively unlearns randomly selected tools, while preserving the LLM's knowledge on non-deleted tools and maintaining performance on general tasks.

Figures

Figures reproduced from arXiv: 2502.01083 by the authors.

Figure 1
Figure 1. Tool Unlearning and the proposed TOOLDELETE approach. (a): Illustration of tool learning and tool unlearning. Learned tools may be requested to be unlearned due to many reasons, such as tools being insecure, restricted, or deprecated. (b): Differences between tool unlearning and traditional sample unlearning, in terms of objective and training data. (c): Proposed method TOOLDELETE. We encourage the unlearned model f… view at source ↗
Figure 2
Figure 2. Measuring tool unlearning with LiRA-Tool. Comparison to LLM-specific unlearning methods Ex￾isting LLM unlearning methods, despite effective in sample￾level unlearning, are prone to under-performing in tool unlearning. Both TOOLDELETE-SFT and TOOLDELETE￾DPO outperforms ICUL, SGA, and TAU on TT , Tr, Tf and TG. The only exception is ICUL, which outperforms TOOLDELETE-SFT on Tr by 2.7 absolute points, but is out￾perfor… view at source ↗
Figure 3
Figure 3. Performance of sequential unlearning on ToolAlpaca. We unlearn 2%, 5%, 10%, 20% of tools in a sequential manner. ments on ToolAlpaca show that TOOLDELETE-LoRA can achieve 97.7%, 99.6%, 84.5%, and 84.3% of the perfor￾mance of TOOLDELETE with full parameter on TT , Tr, Tf , TG on average across SFT and DPO, see [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training time of TOOLDELETE, which saves 74.8% of time on average. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Understanding Machine Unlearning Through the Lens of Mode Connectivity

    cs.LG 2026-07 unverdicted novelty 6.0 of 10

    Unlearned models usually connect to their originals by smooth low-loss paths, and the smoothness of that path can predict how hard the unlearning task was.

  2. A Numerical PDEs Approach to Evolution Equations in Shape Analysis Based on Regularized Morphoelasticity

    math.NA 2026-04 unverdicted novelty 4.0 of 10

    Regularized morphoelasticity yields a high-order elliptic system for continuous shape evolution that is solved by mixed finite elements in FEniCSx within an LDDMM-style optimal-control growth model.

  3. A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A survey and framework that categorizes generative model unlearning by point-wise versus concept-wise objectives, parameter-based versus non-parametric methods, and completeness/utility/efficiency evaluation.

  4. Speech Unlearning

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Existing machine unlearning methods perform poorly on speech tasks, and a SuperLoss-based structured forgetting strategy improves forgetting but is reported with almost no experimental detail.

Reference graph

Works this paper leans on

77 extracted references · 41 canonical work pages · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

    work page

  2. [2]

    API - BLEND : A comprehensive corpora for training and benchmarking API LLM s

    Basu, K., Abdelaziz, I., Chaudhury, S., Dan, S., Crouse, M., Munawar, A., Austel, V., Kumaravel, S., Muthusamy, V., Kapanipathi, P., and Lastras, L. API - BLEND : A comprehensive corpora for training and benchmarking API LLM s. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational L...

    2024

  3. [3]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.,...

    1901

  4. [4]

    and Triantafillou, P

    B a rbulescu, G.-O. and Triantafillou, P. To each ( T extual sequence) its own: Improving memorized-data unlearning in large language models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machi...

    work page 2024

  5. [5]

    Membership inference attacks from first principles

    Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramèr, F. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp.\ 1897--1914, 2022. doi:10.1109/SP46214.2022.9833649

    work page arXiv 2022

  6. [6]

    and Amiri, H

    Cheng, J. and Amiri, H. MultiDelete for multimodal machine unlearning. In European Conference on Computer Vision (ECCV), pp.\ 165--184. Springer, 2024 a

    work page 2024

  7. [7]

    and Amiri, H

    Cheng, J. and Amiri, H. Mu-bench: A multitask multimodal benchmark for machine unlearning. arXiv preprint arXiv:2406.14796, 2024 b

    arXiv 2024

  8. [8]

    and Amiri, H

    Cheng, J. and Amiri, H. Understanding machine unlearning through the lens of mode connectivity. arXiv preprint arXiv:2504.06407, 2025

    arXiv 2025

Show all 77 references
  1. [9]

    GNND elete: A general strategy for unlearning in graph neural networks

    Cheng, J., Dasoulas, G., He, H., Agarwal, C., and Zitnik, M. GNND elete: A general strategy for unlearning in graph neural networks. In The Eleventh International Conference on Learning Representations, 2023

    2023

  2. [10]

    Towards efficient machine unlearning with data augmentation: Guided loss-increasing (gli) to prevent the catastrophic model utility drop

    Choi, D., Choi, S., Lee, E., Seo, J., and Na, D. Towards efficient machine unlearning with data augmentation: Guided loss-increasing (gli) to prevent the catastrophic model utility drop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR...

    2024

  3. [11]

    S., Tarun, A

    Chundawat, V. S., Tarun, A. K., Mandal, M., and Kankanhalli, M. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security, 18: 0 2345--2354, 2023. doi:10.1109/TIFS.2023.3265506

    2023

  4. [12]

    Evaluating the Ripple Effects of Knowledge Editing in Language Models

    Cohen, R., Biran, E., Yoran, O., Globerson, A., and Geva, M. Evaluating the Ripple Effects of Knowledge Editing in Language Models . Transactions of the Association for Computational Linguistics, 12: 0 283--298, 04 2024. ISSN 2307-387X. doi:10.1162/tacl_a_00644

    2024 doi

  5. [13]

    Gpt3.int8(): 8-bit matrix multiplication for transformers at scale

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3.int8(): 8-bit matrix multiplication for transformers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 3...

    2022

  6. [14]

    and Russinovich, M

    Eldan, R. and Russinovich, M. Who's harry potter? approximate unlearning in llms, 2023

    2023

  7. [15]

    Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation

    Fan, C., Liu, J., Zhang, Y., Wong, E., Wei, D., and Liu, S. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In The Twelfth International Conference on Learning Representations, 2024

    2024

  8. [16]

    Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond

    Fan, C., Jia, J., Zhang, Y., Ramakrishna, A., Hong, M., and Liu, S. Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond. arXiv preprint arXiv:2502.05374, 2025 a

    2025 arXiv

  9. [17]

    Challenging forgets: Unveiling the worst-case forget sets in machine unlearning

    Fan, C., Liu, J., Hero, A., and Liu, S. Challenging forgets: Unveiling the worst-case forget sets in machine unlearning. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vision -- ECCV 2024, pp.\ 278--297, Cham, 2025 b . Sprin...

    2024

  10. [18]

    Erasing concepts from diffusion models

    Gandikota, R., Materzy\'nska, J., Fiotto-Kaufman, J., and Bau, D. Erasing concepts from diffusion models. In Proceedings of the 2023 IEEE International Conference on Computer Vision, 2023

    2023

  11. [19]

    Pal: program-aided language models

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  12. [20]

    Eternal sunshine of the spotless net: Selective forgetting in deep networks

    Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020

    2020

  13. [21]

    Factual or contextual? disentangling error types in entity description generation

    Goyal, N., Nenkova, A., and Daum \'e III, H. Factual or contextual? disentangling error types in entity description generation. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volu...

    2023 doi

  14. [22]

    Amnesiac machine learning

    Graves, L., Nagisetty, V., and Ganesh, V. Amnesiac machine learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35 0 (13): 0 11516--11524, May 2021. doi:10.1609/aaai.v35i13.17371

    2021 doi

  15. [23]

    Model editing can hurt general abilities of large language models

    Gu, J.-C., Xu, H.-X., Ma, J.-Y., Lu, P., Ling, Z.-H., Chang, K.-W., and Peng, N. Model editing can hurt general abilities of large language models. arXiv preprint arXiv:2401.04700, 2024

    2024 arXiv

  16. [24]

    Towards a unified view of parameter-efficient transfer learning

    He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., and Neubig, G. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022

    2022

  17. [25]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  18. [26]

    Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection

    Hoang, T., Rana, S., Gupta, S., and Venkatesh, S. Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.\ 4819--4828, January 2024

    2024

  19. [27]

    J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  20. [28]

    Unlearning or obfuscating? jogging the memory of unlearned LLM s via benign relearning

    Hu, S., Fu, Y., Wu, S., and Smith, V. Unlearning or obfuscating? jogging the memory of unlearned LLM s via benign relearning. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  21. [29]

    T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A

    Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023

    2023

  22. [30]

    Knowledge unlearning for mitigating privacy risks in language models

    Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., and Seo, M. Knowledge unlearning for mitigating privacy risks in language models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational...

    2023 doi

  23. [31]

    R., Liu, S., and Chang, S

    Ji, J., Liu, Y., Zhang, Y., Liu, G., Kompella, R. R., Liu, S., and Chang, S. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference. arXiv preprint arXiv:2406.08607, 2024

    2024 arXiv

  24. [32]

    Model sparsity can simplify machine unlearning

    Jia, J., Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y., Sharma, P., and Liu, S. Model sparsity can simplify machine unlearning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  25. [33]

    SOUL : Unlocking the power of second-order optimization for LLM unlearning

    Jia, J., Zhang, Y., Zhang, Y., Liu, J., Runwal, B., Diffenderfer, J., Kailkhura, B., and Liu, S. SOUL : Unlocking the power of second-order optimization for LLM unlearning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical M...

    2024 doi

  26. [34]

    Rwku: Benchmarking real-world knowledge unlearning for large language models

    Jin, Z., Cao, P., Wang, C., He, Z., Yuan, H., Li, J., Chen, Y., Liu, K., and Zhao, J. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint arXiv:2406.10890, 2024

    2024 arXiv

  27. [35]

    Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models

    Kassem, A., Mahmoud, O., and Saad, S. Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lang...

    2023 doi

  28. [36]

    Ablating concepts in text-to-image diffusion models

    Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang, R., and Zhu, J.-Y. Ablating concepts in text-to-image diffusion models. In Proceedings of the 2023 IEEE International Conference on Computer Vision, 2023

    2023

  29. [37]

    Towards unbounded machine unlearning

    Kurmanji, M., Triantafillou, P., Hayes, J., and Triantafillou, E. Towards unbounded machine unlearning. In Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 1957--1987. Curran As...

    1957

  30. [38]

    Machine unlearning for image-to-image generative models

    Li, G., Hsu, H., Chen, C.-F., and Marculescu, R. Machine unlearning for image-to-image generative models. In The Twelfth International Conference on Learning Representations, 2024 a

    2024

  31. [39]

    API -bank: A comprehensive benchmark for tool-augmented LLM s

    Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., and Li, Y. API -bank: A comprehensive benchmark for tool-augmented LLM s. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin...

    2023 doi

  32. [40]

    D., Dombrowski, A.-K., Goel, S., Phan, L., et al

    Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Phan, L., et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024 b

    2024 arXiv

  33. [41]

    Liu, H., Li, Z., Hall, D. L. W., Liang, P., and Ma, T. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In The Twelfth International Conference on Learning Representations, 2024

    2024

  34. [42]

    Muter: Machine unlearning on adversarially trained models

    Liu, J., Xue, M., Lou, J., Zhang, X., Xiong, L., and Qin, Z. Muter: Machine unlearning on adversarially trained models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 4892--4902, October 2023

    2023

  35. [43]

    An adversarial perspective on machine unlearning for AI safety

    ucki, J., Wei, B., Huang, Y., Henderson, P., Tram \`e r, F., and Rando, J. An adversarial perspective on machine unlearning for AI safety. Transactions on Machine Learning Research, 2025. ISSN 2835-8856

    2025

  36. [44]

    Eight methods to evaluate robust unlearning in llms

    Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield-Menell, D. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024

    2024 arXiv

  37. [45]

    The era of 1-bit llms: All large language models are in 1.58 bits

    Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764, 2024

    2024 arXiv

  38. [46]

    C., and Kolter, J

    Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. TOFU : A task of fictitious unlearning for LLM s. In First Conference on Language Modeling, 2024

    2024

  39. [47]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  40. [48]

    Talm: Tool augmented language models

    Parisi, A., Zhao, Y., and Fiedel, N. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022

    2022 arXiv

  41. [49]

    G., Zhang, T., Wang, X., and Gonzalez, J

    Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023

    2023 arXiv

  42. [50]

    In-context unlearning: Language models as few-shot unlearners

    Pawelczyk, M., Neel, S., and Lakkaraju, H. In-context unlearning: Language models as few-shot unlearners. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine ...

    2024

  43. [51]

    Tool LLM : Facilitating large language models to master 16000+ real-world API s

    Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L., Tian, R., Xie, R., Zhou, J., Gerstein, M., dahai li, Liu, Z., and Sun, M. Tool LLM : Facilitating large language models to master 16000+ real-world API s. In The Tw...

    2024

  44. [52]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  45. [53]

    Toolformer: Language models can teach themselves to use tools

    Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  46. [54]

    Adversarial unlearning: Reducing confidence along adversarial directions

    Setlur, A., Eysenbach, B., Smith, V., and Levine, S. Adversarial unlearning: Reducing confidence along adversarial directions. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022

    2022

  47. [55]

    S., Wei, J., Chung, H

    Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature, 620 0 (7972): 0 172--180, 2023

    2023

  48. [56]

    Exploring the impact of model scaling on parameter-efficient tuning

    Su, Y., Chan, C.-M., Cheng, J., Qin, Y., Lin, Y., Hu, S., Yang, Z., Ding, N., Sun, X., Xie, G., Liu, Z., and Sun, M. Exploring the impact of model scaling on parameter-efficient tuning. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empiri...

    2023 doi

  49. [57]

    W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., and Wei, J

    Suzgun, M., Scales, N., Sch \"a rli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., and Wei, J. Challenging BIG -bench tasks and whether chain-of-thought can solve them. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the...

    2023 doi

  50. [58]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases

    Tang, Q., Deng, Z., Lin, H., Han, X., Liang, Q., Cao, B., and Sun, L. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023

    2023 arXiv

  51. [59]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  52. [60]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  53. [61]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  54. [62]

    A., and Arora, R

    Ullah, E., Mai, T., Rao, A., Rossi, R. A., and Arora, R. Machine unlearning via algorithmic stability. In Belkin, M. and Kpotufe, S. (eds.), Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pp.\ 4126--4142. PML...

    2021

  55. [63]

    Vuong, Q. H. Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57 0 (2): 0 307--333, 1989. ISSN 00129682, 14680262

    1989

  56. [64]

    KGA : A general machine unlearning framework based on knowledge gap alignment

    Wang, L., Chen, T., Yuan, W., Zeng, X., Wong, K.-F., and Yin, H. KGA : A general machine unlearning framework based on knowledge gap alignment. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational L...

    2023 doi

  57. [65]

    Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples

    Wei, S., Zhang, M., Zha, H., and Wu, B. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  58. [66]

    Knowledge conflicts for llms: A survey

    Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., and Xu, W. Knowledge conflicts for llms: A survey. arXiv preprint arXiv:2403.08319, 2024

    2024 arXiv

  59. [67]

    Machine unlearning of pre-trained large language models

    Yao, J., Chien, E., Du, M., Niu, X., Wang, T., Cheng, Z., and Yue, X. Machine unlearning of pre-trained large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2024 doi

  60. [68]

    Large language model unlearning

    Yao, Y., Xu, X., and Liu, Y. Large language model unlearning. arXiv preprint arXiv:2310.10683, 2023

    2023 arXiv

  61. [69]

    Evaluating large language models at evaluating instruction following

    Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024

    2024

  62. [70]

    Towards certified unlearning for deep neural networks

    Zhang, B., Dong, Y., Wang, T., and Li, J. Towards certified unlearning for deep neural networks. In Forty-first International Conference on Machine Learning, 2024 a

    2024

  63. [71]

    Negative preference optimization: From catastrophic collapse to effective unlearning

    Zhang, R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling, 2024 b

    2024

  64. [72]

    E., and Stoica, I

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, ...

    2023

  65. [73]

    Lima: Less is more for alignment

    Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

    2024

  66. [74]

    Instruction-following evaluation for large language models

    Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  67. [75]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  68. [76]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  69. [77]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.