Pith. sign in

REVIEW 5 major objections 5 minor 51 references

Stochastic Meta-Unlearning claims that a language-backbone initialization meta-learned with VLM-level feedback lets a few text-only unlearning steps remove a target from the recomposed vision-language model, even when image evidence is supp

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:47 UTC pith:IX3TV7V5

load-bearing objection A sound and useful bilevel meta-unlearning method for VLMs, with honest limitations, but the quantitative claims need error bars and a less cherry-picked baseline comparison before they are taken at face value. the 5 major comments →

arxiv 2607.18615 v1 pith:IX3TV7V5 submitted 2026-07-21 cs.CL cs.LG

Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

Zijie Liu , Jinhao Duan , Gaowen Liu , Sijia Liu , Tianlong Chen This is my paper
classification cs.CL cs.LG
keywords machine unlearningvision-language modelsstochastic meta-learningmultimodal forgettinglanguage backbonebilevel optimizationtransferable unlearningmeme classification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that language-backbone unlearning can be made to stick in a full vision-language model (VLM) — even when images are present at inference — if the backbone begins from an initialization that was meta-learned with VLM-level feedback. The motivating observation is a mismatch: a target forgotten by the standalone text model can reappear once image information is supplied to the recomposed VLM, so text-only objectives are insufficient. To close that gap, the proposed Stochastic Meta-Unlearning (SMU) runs a few unlearning steps on the language backbone in an inner loop, then reinserts the updated backbone into the frozen VLM and evaluates forget/retain losses in an outer loop, using the resulting meta-gradient to adjust the starting point. The paper reports that SMU improves the forget–retain trade-off over three direct VLM-unlearning baselines on two VLMs and two meme datasets, and that the learned initialization transfers to new deletion targets and to different unlearning operators. If true, this matters because deployment-time unlearning could then be done cheaply from the text side only, without touching the visual encoder.

Core claim

The paper's central claim is that the failure of text-side unlearning to persist after multimodal recomposition can be repaired by changing the initialization rather than the unlearning procedure. SMU meta-learns an 'unlearning-ready' set of language-backbone parameters such that, after a few steps of ordinary text-only unlearning, the recomposed VLM forgets the target under both text-only and image-conditioned queries while retaining utility. The decisive mechanism is the outer loop: the updated backbone is reinserted into the frozen VLM, scored on multimodal forget/retain batches, and the meta-gradient is backpropagated through the unrolled inner steps. On two VLMs and two meme datasets, S

What carries the argument

The central object is the meta-learned 'unlearning-ready' initialization θ*_T — the language-backbone parameters or trainable adapters that serve as the starting point for deployment-time unlearning. The carrying mechanism is a bilevel meta-unlearning loop: the inner loop applies K text-only unlearning steps (using GradDiff, NPO, or RMU objectives) to the backbone; the outer loop recomposes the updated backbone with the frozen VLM and computes a weighted loss over multimodal and text forget/retain batches; the meta-gradient with respect to the initialization flows back through the unrolled inner steps. Additionally, SMU stochastically samples the environment (forgetting target, inner operato

Load-bearing premise

The central claim rests on the cluster-based forget/retain split isolating a single coherent target concept and on label accuracy faithfully measuring its removal; if either fails, the meta-learned initialization may be exploiting split structure or abstention rather than genuinely unlearning.

What would settle it

Re-run SMU with the forget set replaced by a randomly sampled subset of the training data (removing cluster coherence) and compare its forget-retain trade-off against the three baselines; if the advantage collapses, the method's transfer comes from the split's structure rather than from a generally unlearning-ready initialization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At deployment, unlearning a VLM requires only text-side gradients on the language backbone; the visual encoder and projector are never updated, cutting multimodal backprop cost.
  • A single meta-training phase yields an initialization usable for many future deletion requests: SMU transfers to unseen targets in the experiments.
  • The initialization is not hard-wired to one unlearning recipe; swapping the inner operator to RMU or NPO at meta-test time still gives low Forget accuracy, so users can pick the operator after the fact.
  • Because image-side unlearning is shown to be unstable (Table 2), the combination of backbone-side updates with VLM-level feedback emerges as the more reliable intervention point for modular VLMs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A broader principle suggests itself: in any modular model where an edit must stay local but behavior is judged globally, meta-learning the edit's starting point against system-level feedback could reconcile local updates with global outcomes — a recipe that may extend beyond language backbones to adapters, LoRA modules, or other modality pairs.
  • The advantage may depend on the forget set being a semantically coherent cluster; a natural stress test is to replace the cluster-based split with a random sample and see whether SMU's margin over baselines persists.
  • Because the paper measures accuracy rather than adversarial recovery, the method's resilience to rephrased prompts, image-only queries, or other cross-modal probes remains open; the authors themselves note there is no formal guarantee against stronger adversarial inputs.
  • The non-monotonic Forget accuracy during meta-training (Fig. 6) hints that the transferable initialization is a specific region of parameter space, not simply the endpoint of a descent; identifying that region could guide stopping criteria and even suggest a theory of when meta-unlearning generalizes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper addresses machine unlearning in vision-language models (VLMs). It observes that applying text-only unlearning to the language backbone of a VLM does not guarantee that the target is forgotten once the backbone is recomposed with the frozen visual components; image-conditioned queries can still elicit the target. To overcome this, the authors propose Stochastic Meta-Unlearning (SMU), a bilevel meta-learning framework. In the inner loop, a few text-side unlearning steps are applied to the language backbone; in the outer loop, the updated backbone is recomposed with the frozen VLM and the outer loss is evaluated on multimodal forget/retain batches. The outer loss is backpropagated through the inner unlearning steps to update the initialization. Experiments on Qwen2.5-VL and InternVL3 with Hateful Memes and COVID Memes report that SMU achieves a better forget-retain trade-off than three direct VLM-side baselines (GradDiff, NPO, RMU), and that the learned initialization transfers to new forgetting targets and to different meta-test unlearning operators. The paper also includes explicit limitations about computation, frozen visual components, and lack of formal guarantees.

Significance. If the reported results hold, the paper makes a useful conceptual contribution: it shows that language-backbone-only unlearning can be made robust under multimodal recomposition by meta-learning the initialization with VLM-level feedback. This is an interesting and non-obvious direction, and the bilevel formulation is well motivated by the component-system mismatch the authors identify. The paper also explicitly states limitations and does not overclaim formal guarantees. However, the empirical support is currently incomplete in several load-bearing places: the experimental design relies on an unvalidated cluster-based target split, no variance or multiple-seed results are reported, the sampling distribution used in stochastic meta-training is not specified, and no quantitative comparison is made to the text-side-only unlearning baseline that motivates the method. These issues do not invalidate the idea, but they must be addressed before the central empirical claims can be accepted.

major comments (5)
  1. [§5, Table 1] No error bars, standard deviations, or multiple-seed results are reported for any of the quantitative claims. The headline numbers (10.52-point Forget reduction, 20.10/17.01-point utility improvements) are point estimates from a single run. Since the differences between methods are sometimes large but the table has no uncertainty information, I cannot tell whether the reported ordering is stable. Please report means and variances over at least three seeds, or justify why single-run evaluation is appropriate.
  2. [§5, Appendix C.2] The cluster-based forget/retain split is load-bearing but unvalidated. KMeans (k=10, seed 42) is applied to MiniLM text embeddings of meme text, and the forget set is 'selected clusters'. The paper claims these form 'semantically coherent' targets, but no evidence is given: no cluster-size table, no label distribution per cluster, no human evaluation or concept-name inspection. Since RQ2/RQ3 switch between these clusters and call them 'new forgetting targets', the transfer claim could reflect transfer between trivially separable text clusters rather than between semantically meaningful unlearning targets. Please add a validation of the clusters, report which clusters are used for meta-training versus held out, and show that the targets correspond to identifiable concepts.
  3. [§4, §5] The core stochastic meta-training protocol is underspecified. The environment distribution p(E) over (z, m, c) is never defined: I could not find the number of meta-training targets, the number of sampled environments S, the distribution over inner-loop operators, the distribution over query conditions, or the mapping from clusters to training/test targets. Without this information, the RQ2/RQ3 claims of transfer are not reproducible and it is unclear whether meta-test targets are truly disjoint from meta-training targets. This is not a minor implementation detail; the paper's central contribution is the stochastic meta-learning design, so the exact protocol must be specified.
  4. [§1, §5] The paper's motivation is that text-only unlearning of the standalone backbone fails to generalize to the recomposed VLM, yet the main experiments do not include a text-side-only unlearning baseline. Table 1 compares SMU only with direct VLM-side baselines (GradDiff, NPO, RMU), which themselves receive VLM-level feedback. A comparison with the same three unlearning objectives applied to the language backbone using text-only data, without VLM-level outer feedback, is needed to isolate the contribution of VLM-level feedback. Figure 1 is anecdotal and does not quantify the failure or the effect of SMU relative to the text-side-only version.
  5. [§3, §5.1] The evaluation metric 'Forget accuracy' on the forget set is a classification accuracy on target-cluster examples. Lower accuracy is interpreted as forgetting, but it may equally reflect misclassification or a collapse to a non-target class (e.g., always predicting 'not hateful'). The paper does not report abstention rates, entropy, or any probe of target knowledge (e.g., open-ended generation or membership-style tests). Given the central claim is that SMU 'removes' the target, additional metrics are needed to distinguish genuine erasure from label flipping. The authors acknowledge in Appendix D that no formal guarantee exists, but the empirical metric should still be strengthened.
minor comments (5)
  1. [Table 1] The GradDiff rows contain formatting errors: '2.0232.09 41.20' and '0.8731.16 40.20' lack spacing between the Forget and Retain entries, making the table hard to read.
  2. [§4] The sentence 'a bilevel framework that aligns language-backbone unlearning with VLM-level forgetting(Figure 2.' has a missing closing parenthesis. Please fix the typo.
  3. [Abstract, §5.1] The claim 'compared with the strongest baseline for each metric' should be defined precisely. I reproduced the 10.52-point Forget reduction by averaging each baseline over all 8 settings and comparing to SMU, but the text does not say this. Please state the exact aggregation procedure.
  4. [Appendix C.2] The appendix says MiniLM embeddings are used 'if this encoder is unavailable' and otherwise TF-IDF is used. It is unclear which representation was actually used for the reported results. This affects reproducibility and should be stated explicitly.
  5. [§5, RQ2/RQ3] The transfer results are presented only as figures (Fig. 4, Fig. 5) with no numeric tables. Given that these figures support two of the three main claims, please include the underlying numbers, ideally in a supplementary table, so readers can assess the magnitude of the effects.

Circularity Check

0 steps flagged

No significant circularity: SMU's gains are held-out empirical comparisons; only minor non-load-bearing self-citations appear in related work.

full rationale

No load-bearing circular step is present. SMU's inner/outer equations (Eqs. 1-3) define an optimization procedure; the evaluation in Table 1 reports accuracy on held-out sets, not the optimized losses themselves, so the forget-retain improvements are not identity-with-input predictions. The transfer results (RQ2/RQ3) use a different meta-test target/operator than the meta-training target/operator, so they are not fitted-input predictions. The KMeans cluster split in Appendix C.2 is a data-construction choice; its semantic validity is a separate concern, not a circular reduction. The paper cites prior work by its own authors (e.g., refs [18], [19], [30], [38]) only in related-work/background positions; none of these citations supplies a load-bearing uniqueness theorem, ansatz, or definition of the target, and the central method is justified by the paper's own equations and experiments. The Limitations section explicitly states there is no formal guarantee of forgetting, consistent with an empirical claim. Thus the paper is self-contained against external benchmarks and receives a low score reflecting only the presence of non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

6 free parameters · 3 axioms · 0 invented entities

SMU introduces no new model component or entity; it introduces a learned initialization (a parameter state). The key free choices are algorithmic (loss weights, meta-training distribution p(E), inner-loop configuration) and data-split choices. The most consequential unexamined assumption is that label-accuracy forgetting equals knowledge removal.

free parameters (6)
  • Inner-loop steps K = 4
    Chosen hyperparameter for the meta-unlearning loop (Appendix C.4); controls how many text-side unlearning steps the initialization is optimized for.
  • Inner/meta learning rates α, β = 1e-5 each
    Chosen step sizes for inner unlearning and outer meta-update (Appendix C.4).
  • Outer loss weights λf, λr = 1.0, 1.0
    Default outer forget/retain weights (Appendix C.4). No ablation or sensitivity analysis is reported.
  • KMeans k = 10
    Cluster count for constructing forget vs. retain sets (Appendix C.2). Changes the size/coherence of the forget set.
  • Cluster selection for forget set = selected target clusters
    The choice of which cluster becomes the forget set is a data-split decision that affects all reported numbers; no cross-validation or stability analysis.
  • Fine-tuning epochs / LoRA rank = 3 epochs / rank 8
    Starting checkpoint is fine-tuned on full train set; this influences how much unimodal unlearning can remove.
axioms (3)
  • domain assumption Standard backpropagation through K inner gradient steps is computationally feasible for 7B-8B backbone parameters with LoRA; gradients through the frozen VLM do not destabilize training.
    The SMU meta-update (Eq. 3) requires differentiating through the recomposed VLM computation graph; no convergence or stability analysis is given.
  • domain assumption The cluster-based PCA/KMeans split yields a semantically coherent target whose removal is a valid unlearning request.
    Section 5 and Appendix C.2: forget set is the union of selected clusters. If clusters do not align with a meaningful concept, the task dissociates from actual unlearning.
  • domain assumption Forgetting measured by label accuracy on meme classification reflects genuine removal of target knowledge.
    The evaluation protocol (Appendix C.3) uses label accuracy only; the authors themselves note that this gives no formal guarantee against adversarial recovery (Appendix D). The assumption that accuracy drop equals knowledge removal is load-bearing for the claimed 'forgetting'.

pith-pipeline@v1.3.0-alltime-deepseek · 14931 in / 6739 out tokens · 49707 ms · 2026-08-01T14:47:23.524876+00:00 · methodology

0 comments
read the original abstract

Machine unlearning for vision-language models (VLMs) remains underexplored. Unlike language models, VLMs combine a language backbone with visual components, which makes unlearning more complex. There is a surprising phenomenon when moving from single-modality unlearning to VLM unlearning: a target forgotten by the standalone language backbone can still be recovered when image information is given to the full VLM. This shows that text-only feedback is not enough for reliable VLM unlearning. Motivated by this observation, we propose Stochastic Meta-Unlearning (SMU), a bilevel framework that uses VLM-level feedback to learn an unlearning-ready initialization. In the inner loop, SMU applies a few unlearning steps to the language backbone using text data. In the outer loop, SMU recomposes the updated backbone with the frozen VLM and evaluates forgetting and utility at the VLM level. This design makes the unlearning update aware of the final multimodal behavior, while still keeping the update local to the language backbone. Experiments on two VLMs, two multimodal meme datasets, and three baselines show that SMU achieves the best overall forget-retain trade-off. Compared with the strongest baseline for each metric, SMU reduces average Forget accuracy by 10.52 points and improves average Retain and Test accuracy by 20.10 and 17.01 points, respectively. More importantly, SMU also transfers to new forgetting targets and to different meta-test unlearning methods. These results suggest that VLM-level feedback can make language-backbone unlearning more reliable and more transferable for VLMs.

Figures

Figures reproduced from arXiv: 2607.18615 by Gaowen Liu, Jinhao Duan, Sijia Liu, Tianlong Chen, Zijie Liu.

Figure 1
Figure 1. Figure 1: Text-side unlearning alone does not guarantee multi￾modal forgetting. Across Qwen and InternVL on Covid and Hateful Memes, LM-side unlearning can reduce Forget Text accuracy, but the target often remains recoverable under image-conditioned VLM queries. Based on this formulation, we pro￾pose Stochastic Meta-Unlearning (SMU). SMU meta-learns a lan￾guage backbone initialization such that, after K steps of uni… view at source ↗
Figure 2
Figure 2. Figure 2: SMU aligns text-side unlearning with VLM-level for￾getting. The inner loop updates only the language backbone through text-side unlearning, while the outer loop recomposes the updated back￾bone with frozen visual components and uses VLM-level feedback to meta-update the initialization. Each meta-training episode samples an environment e = (z, m, c) ∼ p(E), where z is the forgetting target, m is the inner-l… view at source ↗
Figure 3
Figure 3. Figure 3: Cluster-based splits produce coherent forget targets. We visualize PCA-projected training examples for Covid Memes and Hateful Memes; colors denote clusters, from which target clusters are selected as forget sets. The base meta-unlearning objective can over￾fit to a fixed deletion setting. For example, it may learn an initialization that works for one forgetting target, one unlearning operator, or one quer… view at source ↗
Figure 4
Figure 4. Figure 4: SMU transfers to new forgetting targets. When the meta-test forget target differs from the meta-training target, SMU usually preserves low Forget accuracy while maintaining Retain and Test utility across models and datasets. Forget and retain split construction. For each dataset, we construct the for￾get set using a cluster-based proce￾dure. We first represent the training examples in a feature space and a… view at source ↗
Figure 5
Figure 5. Figure 5: SMU transfers across meta-test unlearning operators. After meta-training with GradDiff, SMU remains effective when the meta-test operator is replaced by RMU or NPO, maintaining low Forget accuracy and high Retain/Test accuracy. SMU takes a different approach: it performs the inner unlearning update on the language backbone while using the recomposed VLM for outer-loop forget-retain feed￾back. We compare SM… view at source ↗
Figure 6
Figure 6. Figure 6: Operator transfer improves with meta-training. Evaluating SMU at different checkpoints shows that Forget accuracy is non￾monotonic, while later checkpoints achieve stronger forgetting without large Retain/Test degradation. To evaluate this, we meta-train SMU on one target and then change the forget target at meta-test time. Starting from the learned initialization, we perform the same text-side unlearning … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 16 linked inside Pith

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Soft prompting for unlearning in large language models

    Karuna Bhaila, Minh-Hao Van, and Xintao Wu. Soft prompting for unlearning in large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers), pages 4046–4056, 2025

    2025

  3. [3]

    Machine unlearning

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In2021 IEEE symposium on security and privacy (SP), pages 141–159. IEEE, 2021

    2021

  4. [4]

    Rwku: Benchmarking real-world knowledge unlearning for large language models.Advances in Neural Information Processing Systems, 37:98213–98263, 2024

    Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, Jun Zhao, et al. Rwku: Benchmarking real-world knowledge unlearning for large language models.Advances in Neural Information Processing Systems, 37:98213–98263, 2024

    2024

  5. [5]

    Towards making systems forget with machine unlearning

    Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015

    2015

  6. [6]

    Safeeraser: Enhancing safety in multimodal large language models through multimodal machine unlearning

    Junkai Chen, Zhijie Deng, Kening Zheng, Yibo Yan, Shuliang Liu, PeiJun Wu, Peijie Jiang, Jia Liu, and Xuming Hu. Safeeraser: Enhancing safety in multimodal large language models through multimodal machine unlearning. InFindings of the Association for Computational Linguistics: ACL 2025, pages 14194–14224, 2025

    2025

  7. [7]

    Understanding the generalizability of hateful memes detection models against covid-19-related hateful memes

    Keyan Cuo, Wentai Zhao, Vishant Vishwamitra, Ziming Zhao, Hongxin Hu, et al. Understanding the generalizability of hateful memes detection models against covid-19-related hateful memes. InInternational Conference on Machine Learning and Applications, 2022

    2022

  8. [8]

    Unlearning personal data from a single image.arXiv preprint arXiv:2407.12069, 2024

    Thomas De Min, Massimiliano Mancini, Stéphane Lathuilière, Subhankar Roy, and Elisa Ricci. Unlearning personal data from a single image.arXiv preprint arXiv:2407.12069, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Clear: Character unlearning in 9 textual and visual modalities

    Alexey Dontsov, Dmitrii Korzh, Alexey Zhavoronkin, Boris Mikheev, Denis Bobkov, Aibek Alanov, Oleg Rogov, Ivan Oseledets, and Elena Tutubalina. Clear: Character unlearning in 9 textual and visual modalities. InFindings of the Association for Computational Linguistics: ACL 2025, pages 20582–20603, 2025

    2025

  10. [10]

    Does unlearning truly unlearn? a black box evaluation of llm unlearning methods.arXiv preprint arXiv:2411.12103, 2024

    Jai Doshi and Asa Cooper Stickland. Does unlearning truly unlearn? a black box evaluation of llm unlearning methods.arXiv preprint arXiv:2411.12103, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Who’s harry potter? approximate unlearning in llms, 2023

    Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms, 2023. URL https://arxiv. org/abs/2310.02238, 1(2):8, 2024

    Pith/arXiv arXiv 2023

  12. [12]

    Model-agnostic meta-learning for fast adap- tation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. InInternational conference on machine learning, pages 1126–1135. PMLR, 2017

    2017

  13. [13]

    Meta- unlearning on diffusion models: Preventing relearning unlearned concepts

    Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijie Deng, and Min Lin. Meta- unlearning on diffusion models: Preventing relearning unlearned concepts. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2131–2141, 2025

    2025

  14. [14]

    Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning.arXiv preprint arXiv:2406.13356, 2024

    Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning.arXiv preprint arXiv:2406.13356, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Learning to unlearn for robust machine unlearning

    Mark He Huang, Lin Geng Foo, and Jun Liu. Learning to unlearn for robust machine unlearning. InEuropean conference on computer vision, pages 202–219. Springer, 2024

    2024

  16. [16]

    Mmunlearner: Reformulating multimodal machine unlearning in the era of multimodal large language models

    Jiahao Huo, Yibo Yan, Xu Zheng, Yuanhuiyi Lyu, Xin Zou, Zhihua Wei, and Xuming Hu. Mmunlearner: Reformulating multimodal machine unlearning in the era of multimodal large language models. InFindings of the Association for Computational Linguistics: ACL 2025, pages 7190–7206, 2025

    2025

  17. [17]

    Knowledge unlearning for mitigating privacy risks in language models

    Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 14389–14408, 2023

    2023

  18. [18]

    Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference.Advances in Neural Information Processing Systems, 37:12581–12611, 2024

    Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana R Kompella, Sijia Liu, and Shiyu Chang. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference.Advances in Neural Information Processing Systems, 37:12581–12611, 2024

    2024

  19. [19]

    Soul: Unlocking the power of second-order optimization for llm unlearning

    Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for llm unlearning. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4276–4292, 2024

    2024

  20. [20]

    How can we know what language models know?Transactions of the Association for Computational Linguistics, 8:423–438, 2020

    Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know?Transactions of the Association for Computational Linguistics, 8:423–438, 2020

    2020

  21. [21]

    Pulse: Practical evaluation scenarios for large multimodal model unlearning.arXiv preprint arXiv:2507.01271, 2025

    Tatsuki Kawakami, Kazuki Egashira, Atsuyuki Miyai, Go Irie, and Kiyoharu Aizawa. Pulse: Practical evaluation scenarios for large multimodal model unlearning.arXiv preprint arXiv:2507.01271, 2025

    arXiv 2025

  22. [22]

    The hateful memes challenge: Detecting hate speech in multimodal memes.Advances in neural information processing systems, 33:2611–2624, 2020

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes.Advances in neural information processing systems, 33:2611–2624, 2020

    2020

  23. [23]

    Towards un- bounded machine unlearning.Advances in neural information processing systems, 36:1957– 1987, 2023

    Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards un- bounded machine unlearning.Advances in neural information processing systems, 36:1957– 1987, 2023

    1957

  24. [24]

    Single image unlearning: Efficient machine unlearning in multimodal large language models.Advances in Neural Information Processing Systems, 37:35414–35453, 2024

    Jiaqi Li, Qianshan Wei, Chuanyi Zhang, Guilin Qi, Miaozeng Du, Yongrui Chen, Sheng Bi, and Fan Liu. Single image unlearning: Efficient machine unlearning in multimodal large language models.Advances in Neural Information Processing Systems, 37:35414–35453, 2024. 10

    2024

  25. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  26. [26]

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-V oss, Cort B Breuer, Andy Z...

    2024

  27. [27]

    Sauce: Selective concept unlearning in vision-language models with sparse autoencoders.arXiv preprint arXiv:2503.14530, 2025

    Qing Li, Jiahui Geng, Derui Zhu, Fengyu Cai, Chenyang Lyu, and Fakhri Karray. Sauce: Selective concept unlearning in vision-language models with sparse autoencoders.arXiv preprint arXiv:2503.14530, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Large language model unlearn- ing via embedding-corrupted prompts.Advances in Neural Information Processing Systems, 37:118198–118266, 2024

    Chris Y Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearn- ing via embedding-corrupted prompts.Advances in Neural Information Processing Systems, 37:118198–118266, 2024

    2024

  29. [29]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  30. [30]

    Rethinking machine unlearning for large language models.Nature Machine Intelligence, 7(2):181–194, 2025

    Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models.Nature Machine Intelligence, 7(2):181–194, 2025

    2025

  31. [31]

    Protecting privacy in multimodal large language models with mllmu-bench

    Zheyuan Liu, Guangyao Dou, Mengzhao Jia, Zhaoxuan Tan, Qingkai Zeng, Yongle Yuan, and Meng Jiang. Protecting privacy in multimodal large language models with mllmu-bench. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers), pages...

    2025

  32. [32]

    Modality-aware neuron pruning for unlearning in multimodal large language models

    Zheyuan Liu, Guangyao Dou, Xiangchi Yuan, Chunhui Zhang, Zhaoxuan Tan, and Meng Jiang. Modality-aware neuron pruning for unlearning in multimodal large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 5913–5933, 2025

    2025

  33. [33]

    Towards transfer unlearning: empirical evidence of cross-domain bias mitigation.arXiv preprint arXiv:2407.16951, 2024

    Huimin Lu, Masaru Isonuma, Junichiro Mori, and Ichiro Sakata. Towards transfer unlearning: empirical evidence of cross-domain bias mitigation.arXiv preprint arXiv:2407.16951, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Benchmarking vision language model unlearning via fictitious facial identity dataset.arXiv preprint arXiv:2411.03554, 2024

    Yingzi Ma, Jiongxiao Wang, Fei Wang, Siyuan Ma, Jiazhao Li, Jinsheng Pan, Xiujun Li, Furong Huang, Lichao Sun, Bo Li, et al. Benchmarking vision language model unlearning via fictitious facial identity dataset.arXiv preprint arXiv:2411.03554, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Tofu: A task of fictitious unlearning for llms.arXiv preprint arXiv:2401.06121, 2024

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms.arXiv preprint arXiv:2401.06121, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Descent-to-delete: Gradient-based methods for machine unlearning

    Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. InAlgorithmic Learning Theory, pages 931–962. PMLR, 2021

    2021

  37. [37]

    On first-order meta-learning algorithms

    Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018

    Pith/arXiv arXiv 2018

  38. [38]

    Unlearning sensitive information in multimodal llms: Benchmark and attack-defense evaluation.arXiv preprint arXiv:2505.01456, 2025

    Vaidehi Patil, Yi-Lin Sung, Peter Hase, Jie Peng, Tianlong Chen, and Mohit Bansal. Unlearning sensitive information in multimodal llms: Benchmark and attack-defense evaluation.arXiv preprint arXiv:2505.01456, 2025. 11

    Pith/arXiv arXiv 2025

  39. [39]

    Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2463–2473, 2019

    2019

  40. [40]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  41. [41]

    Data selection for transfer unlearning.arXiv preprint arXiv:2405.10425, 2024

    Nazanin Mohammadi Sepahvand, Vincent Dumoulin, Eleni Triantafillou, and Gintare Karolina Dziugaite. Data selection for transfer unlearning.arXiv preprint arXiv:2405.10425, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Muse: Machine unlearning six-way evaluation for language models.arXiv preprint arXiv:2407.06460, 2024

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models.arXiv preprint arXiv:2407.06460, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Bertnesia: Investigating the capture and forgetting of knowledge in bert

    Jonas Wallat, Jaspreet Singh, and Avishek Anand. Bertnesia: Investigating the capture and forgetting of knowledge in bert. InProceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 174–183, 2020

    2020

  44. [44]

    Umu-bench: Closing the modality gap in multimodal unlearning evaluation

    Chengye Wang, Yuyuan Li, XiaoHua Feng, Chaochao Chen, Xiaolin Zheng, and Jianwei Yin. Umu-bench: Closing the modality gap in multimodal unlearning evaluation. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025

    2025

  45. [45]

    Specify privacy yourself: Assessing inference- time personalized privacy preservation ability of large vision-language models

    Xingqi Wang, Xiaoyuan Yi, Xing Xie, and Jia Jia. Specify privacy yourself: Assessing inference- time personalized privacy preservation ability of large vision-language models. InProceedings of the 33rd ACM International Conference on Multimedia, pages 12304–12313, 2025

    2025

  46. [46]

    Machine unlearning of pre-trained large language models

    Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 8403–8419, 2024

    2024

  47. [47]

    Large language model unlearning.Advances in Neural Information Processing Systems, 37:105425–105475, 2024

    Yuanshun Yao and Xiaojun Xu. Large language model unlearning.Advances in Neural Information Processing Systems, 37:105425–105475, 2024

    2024

  48. [48]

    Large language model unlearning.arXiv preprint arXiv:2310.10683, 2023

    Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning.arXiv preprint arXiv:2310.10683, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    A closer look at machine unlearning for large language models.arXiv preprint arXiv:2410.08109, 2024

    Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. A closer look at machine unlearning for large language models.arXiv preprint arXiv:2410.08109, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Negative preference optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868, 2024

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    not hateful

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava- video: Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024. A Algorithms For completeness, Algorithms 1 and 2 provide pseudocode for stochastic meta-training and meta-test unlearning, respectively. They are deferred from the main paper bec...

    Pith/arXiv arXiv 2024