Pith. sign in

REVIEW 5 major objections 4 minor 44 references

Unlearning in a vision-language model is asymmetric and incomplete: text unlearning transfers to images, but image unlearning leaves text vulnerabilities that typographic attacks recover.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:56 UTC pith:KIRMQ3MI

load-bearing objection Cross-modal unlearning diagnosis is worth reading; the CrossInf mitigation numbers as reported are not to be trusted because of test-set selection. the 5 major comments →

arxiv 2607.16442 v1 pith:KIRMQ3MI submitted 2026-07-17 cs.CV cs.CLcs.CR

One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models

Sudharshan Balaji , Yili Ren , Guangjing Wang , Yimin Chen , Ning Wang This is my paper
classification cs.CV cs.CLcs.CR
keywords machine unlearningvision-language modelscross-modal transfertypographic attacksinfluence functionsasymmetric unlearningshallow unlearningmodel safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether safety unlearning applied in one input modality of a vision-language model—text or image—also suppresses semantically equivalent harmful inputs in the other modality. It finds the transfer is real but asymmetric and incomplete: text unlearning sharply reduces harmful responses to images, while image unlearning leaves harmful text responses partly alive (13.9% LlamaGuard attack rate on LLaVA, 9.7% on IDEFICS, 2.8% on InstructBLIP). It also finds the unlearning is shallow: rendering the forbidden text as an image via typographic attacks recovers 58–69% of the supposedly erased behaviors across all three architectures. To close the gap, the paper proposes CrossInf, which uses block-level influence scores to identify the transformer blocks most responsible for cross-modal coupling and applies unlearning only there; this cuts the LLaVA visual-to-text attack rate to 1.4% and typographic recovery to near zero across all three models, without needing multimodal unlearning data. The stakes are practical: single-modality unlearning is the current default, and these results say it gives a false sense of safety for multimodal deployments.

Core claim

Gradient-ascent unlearning in one modality propagates unevenly through the shared backbone: text-to-visual transfer is near-complete (cross-modal attack success below 0.5% on LlamaGuard-4), while visual-to-text transfer leaves residual risk and is strongly mediated by fusion design. The Q-Former bottleneck of InstructBLIP transfers almost fully; the MLP projector of LLaVA leaves a large gap. Typographic attacks that re-encode forbidden text as pixels recover 58–69% of unlearned behaviors, showing the knowledge was suppressed, not erased. The central claim is that CrossInf, which selects top-k transformer blocks by cross-modal influence, closes most of the gap: visual-to-text LlamaGuard ASR d

What carries the argument

The central object is a block-level influence function over LoRA adapters: for each (layer, adapted-module) block, the paper computes IF_b = -G_cross (regularized Fisher)^{-1} G_f^T between the same-modality forget set and a small cross-modal probe set (DataInf's 2-D gradient features per LoRA matrix), aggregates the absolute entries into a scalar score, and applies the three-term unlearning loss (gradient ascent, refusal, KL-to-reference) only to the top-k% blocks, freezing the rest. The probe set is never used as an unlearning target; it only identifies which blocks couple the modalities.

Load-bearing premise

The cross-modal probe set used to score blocks and to choose the target k is the same set used to measure success, so the reported mitigation benefit assumes those 72 text probes and the 90 typographic images are representative of the attacks that will actually be deployed.

What would settle it

Take a new VLM from one of the three families, run CrossInf with top-k chosen on one held-out set of cross-modal probes, then evaluate on a disjoint set of text probes for the same 12 concepts and on typographic images that were never used in scoring or selection. If the visual-to-text LlamaGuard ASR does not stay near 1.4% (or the typographic ASR at 0%), the headline mitigation is an artifact of selection on the evaluation set.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Deployers who unlearn in one modality should assume the other modality is still partially vulnerable and test both directions plus typographic re-encoding.
  • The LLM blocks are the necessary and sufficient intervention point for cross-modal transfer; vision encoders and fusion modules can be left untouched.
  • Typographic attacks are a cheap, no-optimization bypass for gradient-ascent unlearning, so unlearning evaluations should include them as a standard stress test.
  • A small cross-modal probe set can substitute for large multimodal unlearning datasets when the unlearning is targeted to high-influence blocks.
  • Architecture choice matters: bottlenecks (Q-Former) saturate and transfer on their own; distributed cross-attention (IDEFICS) needs targeted selection; loose MLP projection (LLaVA) can only be partially rescued.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The numbers for CrossInf's headline reductions and the per-model top-k choice both come from the same 72-item text-probe set, so the 13.9%→1.4% and 58–69%→0% results are likely optimistic relative to a deployment attack distribution the method never saw.
  • Inference: The asymmetry suggests a cheap pre-deployment audit: measure cross-modal ASR in the direction opposite to the unlearning modality; if the gap is large, either the fusion architecture cannot support transfer or the unlearning update is too shallow.
  • Inference: If the block-selection mechanism is right, the same influence scores could predict whether a given VLM will transfer any safety intervention, and could be used to choose fusion architectures that are inherently more unlearning-friendly.
  • Inference: A testable extension would run CrossInf's influence scoring on one probe set and evaluate on a disjoint held-out set of probes and new typographic images; if the gains persist, the method is a genuine transfer enhancer rather than a selection artifact.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper reports a bidirectional measurement of cross-modal unlearning transfer in three VLMs (LLaVA-1.5, InstructBLIP, IDEFICS) with different fusion mechanisms. It finds that text-to-visual unlearning transfer is strong (cross-modal ASR below 5% across models), while visual-to-text transfer is weaker and architecture-dependent (text-probe ASR of 13.9% for LLaVA vs. 3% for InstructBLIP after visual unlearning). It further shows that typographic attacks re-encode harmful text as images and recover 58–69% of unlearned behaviors. To close the transfer gap, it proposes CrossInf, an influence-guided block-selection method that concentrates the unlearning loss on the transformer blocks most influential for cross-modal generalization, reporting reductions in LlamaGuard-4 ASR from 13.9% to 1.4% for LLaVA in the visual-to-text direction and to 0% under typographic attacks across all three architectures, while preserving utility. Human evaluation with three annotators (Fleiss κ=0.77) is used to validate the automated metrics, and CKA analysis is used to relate transfer to representational change.

Significance. If the empirical claims hold, this is a useful systematic study. The diagnostic contributions — asymmetry of cross-modal transfer, the role of the fusion architecture, and the fragility of gradient-ascent unlearning under typographic attacks — are novel in scope and directly relevant to VLM safety. The paper also transparently releases code, data probes, and configurations (Appendix F), and it reports both target-string and LlamaGuard metrics alongside a human evaluation, which is good practice. The proposed CrossInf mitigation is potentially valuable, but as presented its quantitative claims are substantially weakened by selection-on-the-evaluation-set issues, the absence of a random-block control, and the reliance on LlamaGuard-4, which the paper itself shows underreports post-unlearning unsafe outputs. The diagnostic findings may survive revision; the mitigation claims are not yet established.

major comments (5)
  1. [§6.2, Table 3] The per-model top-k is selected on the same Exp 1 cross-modal text probes that constitute the headline evaluation set. Table 3 reports Cross-modal ASR for k∈{10,30,50,100} and §6.2 then cites the k=50 (LLaVA) and k=30 (InstructBLIP/IDEFICS) numbers as the CrossInf result. Since the same 72-probe set is used both to pick k and to report the final 13.9%→1.4% reduction, this is selection on the evaluation set. The improvement is therefore inflated. A held-out probe set (or nested selection) is required to support the claim, or the paper should explicitly frame the result as a training-set result rather than a deployment prediction.
  2. [§4.2 and §6.2] The provenance of D_cross is not stated precisely. §4.2 describes D_cross as a 'small cross-modal probe set' used only to compute influence scores; §6.2 says the overhead measurement used '50 cross-modal samples (matching our experimental setup)'. The only cross-modal text probe set described in §5.2 is the 72-item custom set used for Exp 1 evaluation. If D_cross is a subset of that 72-item pool, the influence scores have access to the evaluation distribution, and the block selection is not independent of the test set. Please state explicitly where D_cross is drawn from; if it is the same pool, the 'no multimodal unlearning data' claim in the introduction and abstract is misleading, because a labeled probe set from the target modality was available for model selection.
  3. [§6.3, Table 5] The near-zero ASR claims for CrossInf rest on LlamaGuard-4, which the paper's own Table 5 shows has 51% recall overall and a false-negative rate of 74% on post-unlearning outputs. The target-string ASR after CrossInf is still 43.1% for LLaVA in Exp 1 and 47.1% under typographic attacks. The paper attributes the TS/LG gap to degenerate non-canonical responses (Appendix D), but the human evaluation in §6.5 is not reported for the CrossInf outputs or the typographic-attack outputs specifically, so it does not validate that all TS positives are benign. Without human labels on a stratified sample of the CrossInf TS-positive responses, the claim that the TS metric is a blind spot rather than residual unsafe behavior is unsupported.
  4. [§4.2, Eq. (4)] No control condition separates the contribution of the influence scores from the effect of freezing most blocks. The reported comparison is vanilla unlearning (k=100) versus influence-selected top-k; there is no random-block selection with the same k and the same training budget. The observed improvement could come from the k sweep or from the implicit regularization of updating fewer blocks. A random-block or magnitude-based baseline is necessary to attribute the improvements to Eq. (4). Additionally, the block-level influence score in Eq. (4) is an ad hoc construction (sum of absolute entries of IF_b) whose validity as an estimator of causal contribution is simply assumed; some validation of the score (e.g., correlation with held-out transfer improvement) would be needed.
  5. [§5.1.2, §5.2, Tables 8–10] The evaluation is small-N, single-run, with no uncertainty quantification. Exp 1 cross-modal evaluation uses 72 text probes; Exp 3 uses 90 typographic images. The reported differences are on the order of a handful of items (13.9% is about 10/72; 1.4% is about 1/72; 0% is 0/90). No seeds, multiple runs, or confidence intervals are reported. The architecture-dependence claims (e.g., LLaVA 13.9% vs InstructBLIP 2.8%) may be sensitive to a few responses. At minimum, per-item results, bootstrap CIs, or multiple unlearning runs with different random seeds should be provided for the headline numbers.
minor comments (4)
  1. [Throughout] Notation is inconsistent: 'LlamaGuard-4' and 'Llama-Guard-4' are used interchangeably; 'CrossInfapplies' and similar missing spaces occur after macro names; 'we did a study over the selection of k' should be rephrased. A careful copyedit is needed.
  2. [Figures 7 and 8] The figure captions say 'Bars = target-string ASR; diamonds = LlamaGuard ASR,' but the legends list 'Baseline (LG) Post (LG) Baseline (TS) Post (TS)' with no explicit mapping to marker shapes. The current presentation makes it difficult to tell which series is which; please align the legend order and marker shapes, and define them in the caption.
  3. [Table 3] The header row 'Probe set Pre-Unlearning Vanilla (k=100) k=10 k=30 k=50' is confusing because 'Probe set' is not actually a column that varies; use row grouping labels for each model block. Also define the 'TS/LG' slash notation in the caption.
  4. [Table 10 and §6.4] The intervention names are written differently in the table ('Vision only', 'Fusion only', 'LLM only', 'Fusion + LLM') and in the text ('vision-only', 'fusion-only', 'LLM-only', 'fusion+LLM'). Please standardize the notation for consistency.

Circularity Check

2 steps flagged

CrossInf's headline ASR reductions are minima of k sweeps on the same evaluation sets used to report them; the transfer-diagnosis claims themselves are not circular.

specific steps
  1. fitted input called prediction [§6.2 (Table 3), Exp 1 visual→text cross-modal ASR]
    "we select the top- k% most influential blocks and apply the vanilla unlearning loss using the best k configuration per model. We did a study over the selection of k. Table 3 reports the supporting block-selection sweep across k∈{10, 30, 50, 100}. ... In our experiments, we adopt k=50 for LLaVA and k=30 for InstructBLIP and IDEFICS. ... Cross-modal ASR 73.6/36.1 40.3/13.9 61.1/1.4 47.2/5.6 43.1/1.4"

    The per-model top-k is selected from a sweep over k∈{10,30,50,100} using the Exp 1 cross-modal text-probe ASR rows in Table 3 — the same 72-probe evaluation set on which the headline 'reduces the cross-modal ASR from 13.9% to 1.4%' is computed. The reported CrossInf LG-4 ASR (LLaVA 1.4%; InstructBLIP/IDEFICS 0%) is therefore the minimum of the hyperparameter sweep on the evaluation set, not an independent prediction. Without a held-out probe set or a control that separates k-selection from influence-score quality, the improvement cannot be attributed to the influence guidance; the top-k parameter is effectively fitted to the test metric and then presented as the method's achieved result. The paper also does not state that the 50 cross-modal samples used for the D_cross influence-score over

  2. fitted input called prediction [§6.3, Table 4 and Figure 10 (Exp 3 typographic-attack robustness)]
    "comparing vanilla unlearning to CrossInf at the best k configuration per model (Table 4). ... Table 4: Experiment 3 typographic attack ASR by model and k. Values are target-string/LG-4 percentages on the 90 typographic attack images. The best (lowest target-string) CrossInf configuration per model is in bold."

    The Exp 3 headline robustness numbers (LG-4 ASR 0% for all three architectures; target-string 47.1/13.5/3.8) are obtained by selecting, from Table 4, the k value with the lowest target-string ASR on the very 90 typographic attack images that constitute the Exp 3 evaluation set. Reporting these selected minima as 'CrossInf decreases the ASR substantially ... all three architectures drop to 0%' is selection on the test set: the method's robustness is the best of the k sweep on the evaluation data, not a held-out estimate. The discrepancy with the uniform k adopted in §6.2 (LLaVA k=50, InstructBLIP/IDEFICS k=30) versus the bolded best k per model in Table 4 (LLaVA k=30, InstructBLIP k=10, IDEFICS k=50) confirms that the reported Exp 3 numbers were chosen as the minima on the same attack set u

full rationale

The core bidirectional transfer measurements (Exp 0, Exp 1, Exp 2, the CKA and LoRA-magnitude analyses, and the human evaluation) are empirical observations with stated datasets and metrics; they do not reduce to their own inputs, and there is no load-bearing self-citation chain of the enumerated kinds. However, the two central CrossInf mitigation claims — visual-to-text LG-4 ASR dropping from 13.9% to 1.4% (or 0% in two models) and typographic-attack LG-4 ASR dropping to 0% — are computed on the same probe sets used to select the per-model top-k hyperparameter. Table 3 sweeps k over the Exp 1 text probes and Table 4 selects the lowest-ASR k on the 90 typographic images; reporting those minima as the achieved ASR is test-set fitting, so the 'improvement' is fitted rather than predicted. The paper provides no held-out validation split for k and no control separating influence-score quality from the k-selection effect, and it does not establish that the D_cross samples used for influence scoring are disjoint from the Exp 1 evaluation probes. The paper's own Table 5 (LG-4 recall 51%, false-negative rate 49%, worsening to 74% on post-unlearning outputs) further weakens the literal 'near-zero LG-4' framing, but that is a metric-validity issue rather than circularity. Overall, the transfer diagnosis is non-circular; the mitigation headline is partially circular by construction through selection on the evaluation set, yielding a score of 6.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The central mitigation depends on one fitted hyperparameter (k), an unvalidated block-influence score, and the assumption that the 50-sample cross-modal probe set is representative and not the same distribution as the 72-item eval set. The diagnostic results additionally assume small custom probe sets and LlamaGuard-4 judgments are adequate. No new physical entities are introduced.

free parameters (2)
  • top-k block selection fraction = LLaVA k=50; InstructBLIP k=30; IDEFICS k=30
    Swept over {10, 30, 50, 100} in Table 3 and selected per model to minimize cross-modal ASR on the same Exp 1 probe set used for the headline 13.9% to 1.4% improvement. This is a fitted hyperparameter for the main mitigation claim.
  • per-layer damping lambda_l in influence score = not reported
    Appears in Eq. 3-4 as a per-layer regularizer in the empirical Fisher; no values or selection procedure are given, yet it controls the influence-score ranking.
axioms (6)
  • domain assumption D_cross (50 cross-modal samples) is representative of future cross-modal attacks and is independent of the evaluation probes.
    Used to compute influence scores in §4.2; if it overlaps with the 72-probe eval set, the reported improvements are fitted to the test set.
  • ad hoc to paper The block-level influence score in Eq. 4 is a valid estimator of a block's causal contribution to cross-modal generalization.
    Introduced without derivation; no random-block or gradient-sorting control validates this quantity.
  • domain assumption Gradient-ascent unlearning with the three-term loss of Eq. 1 and QLoRA is a representative instantiation of single-modality unlearning for VLMs.
    Used for all experiments; alternative methods (representation misdirection, preference optimization) may transfer differently, as the paper concedes in §8.
  • domain assumption LlamaGuard-4 unsafe/safe verdicts can serve as the headline safety metric despite an observed 49% FNR overall and 74% FNR post-unlearning.
    The 13.9% to 1.4% and 'near-zero' claims are LG-4 numbers; target-string ASR does not show the same improvements.
  • domain assumption Custom 72 text probes and 90 typographic images are concept-aligned with the forget sets and representative of attack distributions.
    They are in-house artifacts; no external validation beyond three-annotator agreement on aggregate labels.
  • domain assumption The three open-source VLMs (LLaVA, InstructBLIP, IDEFICS) span the fusion design space sufficiently to support architectural conclusions.
    All share LLaMA-family backbones; larger or natively multimodal models may differ, as the paper notes.

pith-pipeline@v1.3.0-alltime-deepseek · 26875 in / 15525 out tokens · 131217 ms · 2026-08-01T20:56:27.166656+00:00 · methodology

0 comments
read the original abstract

Machine unlearning is widely used to remove hazardous knowledge from large language models. Modern Vision-Language Models (VLMs), however, process both text and visual inputs, raising a fundamental security question: does unlearning in one modality transfer to the other? We present the first systematic, bidirectional study of cross-modal unlearning transfer across three VLM architectures: LLaVA-1.5 (MLP projection), InstructBLIP (Q-Former), and IDEFICS (gated cross-attention). We find that unlearning transfers across modalities, but the transfer is asymmetric and incomplete. In some cases, text unlearning strongly transfers to vision. However, this robustness is not preserved under typographic attacks that manipulate the visual presentation of text. Under such attacks, previously unlearned knowledge can be readily recovered, indicating shallow unlearning. To address the transfer gap and shallow robustness, we propose \textsc{CrossInf}, an influence-guided mitigation strategy. Motivated by the observation that different model components contribute unequally to cross-modal transfer, \textsc{CrossInf} focuses unlearning on transformer blocks that most influence cross-modal generalization. It reduces the transfer gap by more than half in architectures with strong fusion, while preserving model utility. It also improves robustness under typographic attacks, reducing the attack success rate to near zero. We further conduct human evaluation with three annotators ($\kappa{=}0.77$) to validate our findings. Finally, we analyze shallow unlearning using Centered Kernel Alignment (CKA), providing insights into the observed transfer behavior and robustness limitations.

Figures

Figures reproduced from arXiv: 2607.16442 by Guangjing Wang, Ning Wang, Sudharshan Balaji, Yili Ren, Yimin Chen.

Figure 1
Figure 1. Figure 1: In contrast, image-based unlearning does not fully general [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: Left: Single-modality unlearning creates a cross [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the CrossInf experimental framework. Each of the three VLM architectures is evaluated in its baseline state and after vanilla unlearning and CrossInf unlearning, under both same-modal and cross-modal conditions. Experiments span two transfer directions, an intervention-point ablation, and typographic attack recovery. CrossInf: Influence-Guided Selection Forget Set FForget Set orget Set Retain S… view at source ↗
Figure 3
Figure 3. Figure 3: CrossInf design. DataInf scores identify the most influential blocks; unlearning is applied only to the top-𝑘% while the rest are frozen. sample), and C (𝑏) = 1 𝑛𝑓 (G (𝑏) 𝑓 ) ⊤G (𝑏) 𝑓 +𝜆I is the regularized empirical Fisher computed entirely from D𝑓 . The probe set Dcross enters only through G (𝑏) cross, which is a one-pass forward-backward read used to score blocks; it is never used as an unlearning targe… view at source ↗
Figure 4
Figure 4. Figure 4: Three VLM fusion designs studied in this work: MLP [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Six representative typographic attack images from [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Cross-modal transfer results. Top: Exp 0 (text [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cross-modal ASR under CrossInf. Gray = vanilla, green = improved, red = worsened. LLaVA InsBLIP IDEFICS 0 2 4 6 8 10 12 14 Utility Acc. (%) 8.0 8.5 2.5 3.0 3.0 4.5 Exp 0: TruthfulQA LLaVA InsBLIP IDEFICS 0 20 40 60 80 75.374.4 79.379.7 31.7 11.5 Exp 1: VQA-v2 Vanilla Preserved / higher Lower [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Typographic attack robustness (Exp 3) of Vanilla [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Ablation profiles across four interventions in Exp 2, [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Interpretability analysis. Top: LoRA magnitude per layer (Exp 0 red, Exp 1 blue; [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Disagreement resolution on the 175-item shared [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗
Figure 15
Figure 15. Figure 15: Component-level LoRA magnitudes by interven [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 2 canonical work pages

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    2022

  2. [2]

    Hadi Askari, Shivanshu Gupta, Fei Wang, Anshuman Chhabra, and Muhao Chen. 2025. LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions. arXiv:2505.23811 [cs.CL] https://arxiv.org/abs/2505.23811

    arXiv 2025

  3. [3]

    Yinzhi Cao and Junfeng Yang. 2015. Towards Making Systems Forget with Machine Unlearning. InProceedings of the 2015 IEEE Symposium on Security and Privacy (SP ’15). IEEE Computer Society, USA, 463–480. doi:10.1109/SP.2015.35

    doi:10.1109/sp.2015.35 2015

  4. [4]

    Abu-Ghazaleh, M

    Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael B. Abu-Ghazaleh, M. Salman Asif, Yue Dong, Amit Roy-Chowdhury, and Chengyu Song. 2024. Can Textual Unlearning Solve Cross-Modality Safety Alignment?. InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mo- hit Bansal, and Yun-Nung Chen (Eds.). Association for Comp...

    doi:10.18653/v1/2024.findings-emnlp.574 2024

  5. [5]

    Junkai Chen, Zhijie Deng, Kening Zheng, Yibo Yan, Shuliang Liu, PeiJun Wu, Peijie Jiang, Jia Liu, and Xuming Hu. 2025. SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Ta...

    2025

  6. [6]

    Jiali Cheng and Hadi Amiri. 2024. MultiDelete for Multimodal Machine Unlearn- ing. arXiv:2311.12047 [cs.AI] https://arxiv.org/abs/2311.12047

    Pith/arXiv arXiv 2024

  7. [7]

    Minseok Choi, Kyunghyun Min, and Jaegul Choo. 2024. Cross-Lingual Unlearn- ing of Selective Knowledge in Multilingual Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 10732–10747. doi:10.18653/v...

    work page doi:10.18653/v1/2024.findings-emnlp.630 2024

  8. [8]

    Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales.Educational and Psychological Measurement20 (1960), 37 – 46. https://api.semanticscholar. org/CorpusID:15926286

    1960

  9. [9]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500 [cs.CV] https://arxiv.org/abs/2305.06500

    Pith/arXiv arXiv 2023

  10. [10]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314 [cs.LG] https: //arxiv.org/abs/2305.14314

    Pith/arXiv arXiv 2023

  11. [11]

    Alexey Dontsov, Dmitrii Korzh, Alexey Zhavoronkin, Boris Mikheev, Denis Bobkov, Aibek Alanov, Oleg Rogov, Ivan Oseledets, and Elena Tutubalina. 2025. CLEAR: Character Unlearning in Textual and Visual Modalities. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehva...

    doi:10.18653/v1/ 2025

  12. [12]

    Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. 2024. SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. arXiv:2310.12508 [cs.LG] https://arxiv.org/abs/2310.12508

    Pith/arXiv arXiv 2024

  13. [13]

    Joseph L. Fleiss. 1971. Measuring Nominal Scale Agreement Among Many Raters. Psychological Bulletin76, 5 (1971), 378–382. doi:10.1037/h0031619

    doi:10.1037/h0031619 1971

  14. [14]

    Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau

  15. [15]

    Jiahui Geng, Qing Li, Herbert Woisetschlaeger, Zongxiong Chen, Fengyu Cai, Yuxia Wang, Preslav Nakov, Hans-Arno Jacobsen, and Fakhri Karray. 2025. A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models. arXiv:2503.01854 [cs.CL] https://arxiv.org/abs/2503.01854

    Pith/arXiv arXiv 2025

  16. [16]

    Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. 2025. FigStep: jailbreaking large vision- language models via typographic visual prompts. InProceedings of the Thirty- Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligenc...

    2025

  17. [17]

    Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2019. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering.Int. J. Comput. Vision 127, 4, 398–414. doi:10.1007/s11263-018-1116-0

    doi:10.1007/s11263-018-1116-0 2019

  18. [18]

    Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability. arXiv:2402.08679 [cs.LG] https://arxiv.org/abs/2402.08679

    Pith/arXiv arXiv 2024

  19. [19]

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. (2023). arXiv:2312.06674 [cs.CL] https://arxiv.org/abs/2312.06674

    Pith/arXiv arXiv 2023

  20. [20]

    Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset. InAdvances in Neural Information Processing Systems. arXiv:2307.04657 [cs.CL] https://arxiv. org/abs/2307.04657

    arXiv 2023

  21. [21]

    Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. 2025. WAGLE: Strategic Weight Attribution for Effective and Modu- lar Unlearning in Large Language Models. InAdvances in Neural Information Processing Systems. arXiv:2410.17509 [cs.LG] https://arxiv.org/abs/2410.17509

    Pith/arXiv arXiv 2025

  22. [22]

    Pang Wei Koh and Percy Liang. 2017. Understanding Black-box Predictions via Influence Functions. InProceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 1885–1894. https://proceedings.mlr. press/v70/koh17a.html

    2017

  23. [23]

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of Neural Network Representations Revisited. InProceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 3519–3529. https://proceedings.mlr.press/v97...

    2019

  24. [24]

    Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. 2024. DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models. InThe Twelfth International Conference on Learning Representations. https://arxiv.org/ abs/2310.00902

    Pith/arXiv arXiv 2024

  25. [25]

    Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh

    Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. 2023. OBELICS: an open web-scale filtered dataset of interleaved image-text documents. InProceedings of the 37th International Conference on Neural Information Proces...

    2023

  26. [26]

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-Voss, Cort B Breuer, Andy Zo...

    2024

  27. [27]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computa- tional Linguistics, Dublin, Ireland, 3214–3...

    doi:10.18653/v1/2022.acl-long.229 2022

  28. [28]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved Baselines with Visual Instruction Tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26286–26296. https://arxiv.org/abs/ 2310.03744

    Pith/arXiv arXiv 2024

  29. [29]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual In- struction Tuning. InAdvances in Neural Information Processing Systems. https: //arxiv.org/abs/2304.08485

    Pith/arXiv arXiv 2023

  30. [30]

    Zheyuan Liu, Guangyao Dou, Mengzhao Jia, Zhaoxuan Tan, Qingkai Zeng, Yongle Yuan, and Meng Jiang. 2025. Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L...

    work page doi:10.18653/v1/2 2025

  31. [31]

    Zheyuan Liu, Guangyao Dou, Xiangchi Yuan, Chunhui Zhang, Zhaoxuan Tan, and Meng Jiang. 2025. Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher ...

    2025

  32. [32]

    Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. 2024. JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks. arXiv:2404.03027 [cs.CR] https: //arxiv.org/abs/2404.03027

    Pith/arXiv arXiv 2024

  33. [33]

    Lipton, and J

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. TOFU: A Task of Fictitious Unlearning for LLMs. arXiv:2401.06121 [cs.LG] https://arxiv.org/abs/2401.06121

    Pith/arXiv arXiv 2024

  34. [34]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Balaji et al

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Balaji et al. Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions ...

    2022

  35. [35]

    Karl Pearson. 1895. VII. Note on regression and inheritance in the case of two parents.Proceedings of the Royal Society of London58, 347-352 (12 1895), 240–242. arXiv:https://royalsocietypublishing.org/rspl/article-pdf/58/347- 352/240/263745/rspl.1895.0041.pdf doi:10.1098/rspl.1895.0041

    arXiv

  36. [36]

    Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. 2024. Visual adversarial examples jailbreak aligned large language models. InProceedings of the Thirty-Eighth AAAI Conference on Ar- tificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on...

    doi:10.1609/aaai.v38i19.30150 2024

  37. [37]

    Yiting Qu, Michael Backes, and Yang Zhang. 2025. Bridging the gap in vision language models in identifying unsafe concepts across modalities. InProceedings of the 34th USENIX Conference on Security Symposium(Seattle, WA, USA)(SEC ’25). USENIX Association, USA, Article 50, 20 pages

    2025

  38. [38]

    Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. 2024. Jailbreak in Pieces: Compositional Adversarial Attacks on Multi-Modal Language Models. InThe Twelfth International Conference on Learning Representations. https://arxiv.org/ abs/2307.14539

    Pith/arXiv arXiv 2024

  39. [39]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How Does LLM Safety Training Fail? arXiv:2307.02483 [cs.LG] https://arxiv.org/abs/ 2307.02483

    Pith/arXiv arXiv 2023

  40. [40]

    Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. Large Language Model Unlearn- ing. arXiv:2310.10683 [cs.CL] https://arxiv.org/abs/2310.10683

    Pith/arXiv arXiv 2024

  41. [41]

    Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. 2025. Catastrophic Failure of LLM Un- learning via Quantization. InThe Thirteenth International Conference on Learning Representations. https://arxiv.org/abs/2410.16454

    Pith/arXiv arXiv 2025

  42. [42]

    Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. 2024. Safety fine-tuning at (almost) no cost: a baseline for vision large language models. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 2604, 25 pages

    2024

  43. [43]

    I cannot

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043 [cs.CL] https://arxiv.org/abs/2307.15043 One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models 63% 30% 7% (a) Resolution Unanimous (...

    Pith/arXiv arXiv 2023

  44. [2023]

    InProceedings of the IEEE/CVF International Conference on Computer Vision

    Erasing Concepts from Diffusion Models. InProceedings of the IEEE/CVF International Conference on Computer Vision. 2426–2436. https://arxiv.org/abs/ 2303.07345

    Pith/arXiv arXiv