Pith. sign in

REVIEW 3 major objections 5 minor 16 references

PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PRMU claims that realistic person-centric unlearning in multimodal models must be evaluated without original training corpora, with neighbor-based locality, and that current methods fail this test.

desk verdict A genuinely useful corpus-free multimodal unlearning benchmark, but the NKS filter is calibrated to Qwen2.5-VL-32B, so the forgetting scores for smaller models partly measure suppression of knowledge they never had. read the letter →

arxiv 2608.11149 v1 pith:MKSR5AYI submitted 2026-08-11 cs.CV

Huafeng Chen , Yueming Lyu , Ziyuan Chen , Wenda Tan , Chenyang Si , Liucheng Guo , Caifeng Shan This is my paper · ORCID
classification cs.CV
keywords machineunlearningmultimodallargelanguagemodelsperson-centricknowledgecorpus-freebenchmarklocalitypreservationreactivationPRMUSGPE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that machine unlearning of a specific person from a multimodal large language model should be testable in the realistic situation where the requester supplies only the person's name and a reference image, not the original training data. To make that test possible it builds PRMU, a benchmark of 1,080 public figures with 50,649 textual and 41,303 visual probes, all filtered to knowledge that a strong reference model already holds. The benchmark adds a Neighbor Set of related individuals so that collateral damage to nearby knowledge is measured separately from general utility loss. Running four existing unlearning methods and a new lightweight baseline, SGPE, the paper reports that current methods sacrifice related knowledge for forgetting, that vision-language probes expose residual knowledge text probes miss, and that larger batches of deletion requests worsen both problems.

What carries the argument

The load-bearing objects are the PRMU pipeline and SGPE. PRMU uses a Native Knowledge Score (NKS) filter: a reference multimodal model must answer a probe with Token-F1 above 0.5, so only knowledge the model family already possesses enters the Forget, Neighbor, and Other Sets. The Neighbor Set is built from a composite score over five person-level signals—occupation match, nationality match, birth-year distance, popularity distance, and CLIP visual similarity—so locality is measured against people who resemble the target, not just random facts. SGPE carries the argument that unlearning can be localized: it derives displacement anchors from model-generated biographies, facts, and QA pairs, projects edits into the orthogonal complement of a target-blind activation basis, and activates the edit through a similarity gate comparing target prototypes to background prototypes. That gate is what converts a global parameter update into an input-conditioned unlearning operator.

What would settle it

Measure each target model's own pre-unlearning accuracy on the PRMU Forget Set. If a substantial fraction of Forget Set probes are not answerable by that model—for instance, LLaVA-1.5-7B's 'Before' QA(text) score is 0.630 in the batch setting—then the benchmark's forgetting scores conflate unlearning with refusal of questions the model never knew, and the NKS filter should be per-model rather than reference-model-based.

Watch

Extended reading notes

Core claim

On the paper's own terms, PRMU is the first person-centric multimodal unlearning benchmark that is corpus-free: unlearning methods receive only the original model and a target specification, while a target-specific proxy corpus generated by the model itself stands in for inaccessible training data. The benchmark's central discovery is a systematic trade-off: gradient ascent, rejection tuning, preference optimization, and direct preference optimization all reduce target answers, but at the cost of larger drops on the Neighbor Set than on unrelated knowledge, and their residual scores are consistently higher on vision-language probes than on text-only probes. PRMU attributes this to multimodal associations that reactivate target knowledge, and to the absence of locality-aware evaluation in prior benchmarks. Its SGPE baseline, which writes edits into a protected parameter subspace and gates them by target-relevant similarity, is offered as evidence that a corpus-free method can suppress target knowledge while keeping most neighbor and unrelated knowledge intact.

Load-bearing premise

The benchmark assumes that a probe judged 'native knowledge' by the reference model Qwen2.5-VL-32B is also knowledge possessed by each evaluated smaller model; if a model cannot answer a Forget Set probe before unlearning, then measuring a drop in that probe counts suppression of absent knowledge, not unlearning.

Editorial extensions

If this is right

  • If PRMU's findings hold, text-only evaluations of MLLM unlearning understate residual knowledge; vision-language probes should be part of any deletion test.
  • Neighbor-based locality scores should accompany forgetting metrics, because collateral damage concentrates on related individuals rather than unrelated knowledge.
  • Corpus-free deletion protocols are feasible: model-generated proxy corpora can substitute for inaccessible forget and retain data, and SGPE shows a parameter-editing approach can exploit them.
  • Batch deletion requests introduce cross-target interference, so scalable unlearning needs methods that treat multiple deletions jointly rather than independently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same protocol could transfer beyond public figures to any entity-level deletion—brands, organizations, or fictional characters—provided a probeable profile and reference image exist.
  • Editorial inference: a direct test of the benchmark's assumption would be to compute each target model's own pre-unlearning accuracy on the Forget Set; low 'Before' accuracy would mean the Forget Set partly measures suppression of knowledge the model never had, and per-model NKS filtering would be the natural fix.
  • Editorial inference: the similarity-gate mechanism suggests a general recipe for corpus-free editing: use the model itself to generate displacement data, protect a calibration-defined subspace, and condition the edit on prototype similarity—this could apply to other knowledge types such as copyrighted text or harmful capabilities.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces PRMU, a corpus-free benchmark for person-centric knowledge unlearning in multimodal large language models (MLLMs). PRMU is constructed from public-figure profiles derived from Wikidata and Wikipedia, with 1,080 targets, 50,649 textual probes, and 41,303 visual probes. A Native Knowledge Score (NKS) filter using Qwen2.5-VL-32B retains probes that are likely acquired during pretraining. The benchmark provides a corpus-free protocol in which unlearning methods receive only the original model and a target specification, plus a model-generated proxy corpus. It evaluates forgetting, neighbor-based locality, utility, and adversarial robustness under single-target and batch-target settings. The paper also proposes SGPE, a lightweight corpus-free unlearning baseline using knowledge displacement, protected projection editing, and a similarity gate. Experiments on LLaVA-1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B compare SGPE with GA, RT, NPO, and DPO, reporting that existing methods suffer from unfavorable forgetting-locality trade-offs and multimodal reactivation, while SGPE offers a more competitive trade-off.

Significance. If the reported findings hold, PRMU addresses a genuine gap: existing MLLM unlearning benchmarks either assume access to original forget/retain corpora or rely on injected knowledge, whereas PRMU targets naturally acquired knowledge in a corpus-free setting. The benchmark has substantial scope, including external-knowledge-based probes, neighbor sets over multiple relationship types, adversarial attack styles, and a proxy-corpus mechanism for adapting existing unlearning methods. A notable strength is that the evaluation is not circular: probes are built from Wikidata/Wikipedia rather than from the unlearning methods or the proxy corpus. The main risk is the transferability of the NKS filter across models of different sizes and knowledge distributions; unless that risk is resolved, the numerical findings are conditional on knowledge being native to the reference model rather than to the evaluated models. The contribution is potentially valuable, but the central validity concern requires revision.

major comments (3)
  1. [Native Knowledge Filtering / Table 2] The NKS filter defines 'known knowledge' through Qwen2.5-VL-32B with Token-F1 > 0.5, but the evaluated models are smaller and have different knowledge distributions. Table 2's Before rows show that this assumption fails for the actual test models: in the batch-target setting, LLaVA-1.5-7B has Forget QA(text) = 0.630 and QA(vis) = 0.479, and Qwen2.5-VL-7B has Forget QA(text) = 0.606, before any unlearning. This means a substantial fraction of Forget probes are not natively answerable by these models, so the reported 'forgetting' improvements and the resulting trade-offs in Findings 1-4 partly reflect suppression of knowledge the model never possessed rather than removal of acquired person knowledge. The manuscript says the Appendix contains knowledge-coverage statistics, but the main-text conclusions are not conditioned on per-model native subsets. Please re-run or re-analyze the evaluation on probes each model can answer natively, or state explicitly that PRMU measures suppression relative to the reference model rather than unlearning for each target model.
  2. [Table 4 / Impact of Deletion Scale] Finding 4, that scaling to multiple targets increases forgetting difficulty and collateral interference, is not established by the current design. Table 4's nested batches are said to control target composition, but the Before rows differ substantially across batch sizes: Forget QA(text) is 0.666 for N=50, 0.680 for N=100, and 0.777 for N=150 on Qwen3-VL-8B. Because average baseline knowledge differs across the subsets, the absolute and relative post-unlearning scores cannot separate a scale effect from a target-difficulty effect. A proper analysis would vary batch size over matched or repeated random target samples and report variance across the samples.
  3. [Experimental Setup / Equations (3)-(6)] SGPE is a central contribution, but the main text does not report the hyperparameters needed to reproduce it. Equations (3)-(6) involve lambda_u, lambda_z, lambda_c, gamma, T, and tau, and the text also requires selecting the editable layer ell, the prototype banks, and the optimization procedure. Without these values, SGPE cannot be reimplemented and the paper's comparative claims about SGPE cannot be independently verified. Please provide a complete hyperparameter table and describe the optimization budget, calibration procedure for thresholds, and any early-stopping or regularization choices.
minor comments (5)
  1. [Figure 1] The NKS threshold 'Token-F1 > 0.5' appears only in the figure; the main text should formally define Token-F1 and explain how model answers are parsed and compared with ground-truth answers.
  2. [Equation (2)] The signs of N3 and N4 in Eq. (2) are not explained in the notation: state explicitly that N3 and N4 are normalized distances and therefore enter the composite score negatively, and clarify the ranges of all five components.
  3. [Tables 2-6] No error bars, standard deviations, or repeated runs are reported for any of the main comparisons. At minimum, report variance over target subsets or random seeds for the headline single-target and batch-target results so that the trade-offs in Fig. 3 are not driven by a single run.
  4. [Figure 4] The attack-style abbreviations (MC, Para, CH, RP, IR, RC, PEC, CN, CF, TA, FS, MP, BCC, CL) are only defined in the Appendix; please define them in the caption or in the main text for readability.
  5. [Table 2] Table 2 is extremely dense, with 24 metric columns per setting. Consider splitting the results by probe type or using grouped bar plots to make the bidirectional trade-offs easier to inspect.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: PRMU's probes are built from external knowledge sources and filtered by a fixed reference model, not fitted to the evaluated models; the main weakness is a construct-validity concern about knowledge coverage, not circularity.

full rationale

The benchmark construction is self-contained against external sources. Probes are generated from Wikidata/Wikipedia profiles via GPT-5.4 and filtered by the Native Knowledge Score using Qwen2.5-VL-32B as a fixed reference ('we employ Qwen2.5-VL-32B without fine-tuning as a reference model and retain probes that are correctly answered'). This is a preprocessing step, not a parameter fitted to the evaluated models; no evaluation score is the filter output by construction. The proxy corpus generated from the original MLLM is an input to unlearning methods, not the evaluation target, so GA/NPO/RT/DPO/SGPE results are measured on independent held-out probes. SGPE's displacement anchors are self-referential as a method design, but the paper does not derive a benchmark result from that design; it is ablated and evaluated externally. The one shared-author citation (Huo et al. 2025, MMUnlearner, in related work) is not load-bearing. The real concern is the NKS transfer assumption: the paper itself notes 'detailed statistics' of MLLM knowledge coverage are deferred to the Appendix, and Table 2's Before rows show LLaVA-1.5-7B Forget Set QA(text)=0.630 and Qwen2.5-VL-7B QA(text)=0.606, so some probes may not be natively known by the evaluated models. That weakens the interpretation of forgetting scores as unlearning of acquired knowledge, but it is a validity limitation, not a circular derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark relies on several domain assumptions and design choices: external knowledge sources as ground truth, a fixed reference model for native knowledge filtering, model-generated proxy corpora as surrogates for missing training data, and chosen evaluation metrics. The SGPE method adds hand-tuned hyperparameters. No new physical or conceptual entities are introduced beyond the algorithmic method itself.

free parameters (4)
  • NKS Token-F1 threshold = 0.5
    Threshold for retaining probes in Native Knowledge Filtering; referenced to Qwen2.5-VL-32B answers.
  • Neighbor set size and composite weights = top-5, equal weights
    Design choice for neighbor selection; equal weighting stated to avoid preference toward specific relationship types.
  • Batch size settings = 150 primary, 50 and 100 nested
    Unlearning scenario configurations; nested subsets are used to control target composition across deletion scales.
  • SGPE hyperparameters = not reported in main text
    Lambda_u, lambda_z, lambda_c, gamma, eta, rho, tau_T, tau_V, temperature T, and editable layer l; tuning details are deferred to the appendix.
assumptions (5)
  • domain assumption Public figure profiles from Wikidata and Wikipedia are accurate ground truth for the target knowledge.
    Probe construction and evaluation assume these sources define the facts to be forgotten and preserved.
  • ad hoc to paper Native knowledge filtered by Qwen2.5-VL-32B transfers as 'native' to other MLLMs.
    A single reference model determines which probes are considered native knowledge for all evaluated models, despite differing architectures and scales.
  • domain assumption Model-generated proxy corpus is a valid surrogate for the inaccessible original forget corpus.
    The corpus-free protocol provides proxy data generated from the original MLLM, and methods use it in place of original training data.
  • domain assumption ROUGE-L recall and MCQ accuracy on the defined probe sets measure forgetting and preservation.
    The paper uses these as primary metrics and acknowledges that generation-based metrics mainly measure target knowledge suppression.
  • ad hoc to paper Composite neighbor score with equal weights captures semantic relatedness for locality evaluation.
    Neighbor Set construction is based on an unvalidated equal-weight combination of occupation, nationality, birth year, popularity, and visual similarity signals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/MKSR5AYI

@misc{pith2026260811149,
  author       = {Pith},
  title        = {Pith review of: PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MKSR5AYI}},
  note         = {Machine review of arXiv:2608.11149}
}
read the original abstract

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in storing and recalling rich person-related knowledge, raising increasing concerns about reliable knowledge removal. However, existing machine unlearning approaches for MLLMs typically assume access to original forget and retain corpora, which are often unavailable in realistic deletion scenarios. To address this limitation, we introduce PRMU, a benchmark for evaluating corpus-free multimodal unlearning under realistic person-centric deletion requests. PRMU focuses on naturally acquired person-related knowledge and evaluates whether models can remove target knowledge while preserving related knowledge through diverse textual and visual probes, including adversarial evaluation and fine-grained locality analysis. To facilitate research in this setting, we further introduce Similarity-Gated Projection Editing (SGPE), a lightweight corpus-free unlearning baseline with knowledge displacement, protected parameter-space editing, and locality-aware multimodal control. Extensive experiments on representative MLLMs reveal that existing unlearning methods often suffer from unfavorable forgetting-locality trade-offs, with significant locality degradation under aggressive forgetting settings, and remain vulnerable to multimodal knowledge reactivation. Meanwhile, SGPE provides a competitive trade-off between target forgetting, locality preservation, and general multimodal utility. We hope PRMU can facilitate future research toward realistic and scalable multimodal machine unlearning. Code and dataset will be released at https://github.com/2231122/PRMU.

Figures

Figures reproduced from arXiv: 2608.11149 by the authors.

Figure 1
Figure 1. Overview of the PRMU benchmark pipeline. PRMU constructs person-centric targets from public figures, generates [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Trade-off between forgetting and knowledge preser [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. The heatmap reports ROUGE-L scores under 14 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages

  1. [5]

    InFindings of the Association for Computational Linguistics: ACL 2025, 20582–20603

    Clear: Character unlearning in textual and visual modalities. InFindings of the Association for Computational Linguistics: ACL 2025, 20582–20603. Eldan, R.; and Russinovich, M

    work page 2025

  2. [6]

    org/abs/2310.02238, 1(2):

    Who’s harry potter? approximate unlearning in llms, 2023.URL https://arxiv. org/abs/2310.02238, 1(2):

    arXiv 2023

  3. [8]

    Golatkar, A.; Achille, A.; and Soatto, S. 2020a. Eternal sunshine of the spotless net: Selective forgetting in deep networks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9304–9312. Golatkar, A.; Achille, A.; and Soatto, S. 2020b. Forgetting outsidethebox:Scrubbingdeepnetworksofinformationac- cessible from input-out...

    arXiv 2026

  4. [9]

    Jang, J.; Yoon, D.; Yang, S.; Cha, S.; Lee, M.; Logeswaran, L.; and Seo, M

    Knowledge sanitization of large language models.arXiv preprint arXiv:2309.11852. Jang, J.; Yoon, D.; Yang, S.; Cha, S.; Lee, M.; Logeswaran, L.; and Seo, M

    arXiv

  5. [10]

    InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14389–14408

    Knowledge unlearning for mitigating privacy risks in language models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14389–14408. Kurmanji, M.; Triantafillou, P.; Hayes, J.; and Triantafillou, E.2023. Towardsunboundedmachineunlearning.Advances in neural information processing systems, 36:...

    work page 2023

  6. [11]

    Qi,X.;Panda,A.;Lyu,K.;Ma,X.;Roy,S.;Beirami,A.;Mit- tal, P.; and Henderson, P

    Tofu: A task of fictitious unlearning for llms.arXiv preprint arXiv:2401.06121. Qi,X.;Panda,A.;Lyu,K.;Ma,X.;Roy,S.;Beirami,A.;Mit- tal, P.; and Henderson, P

    arXiv

  7. [12]

    InInternationalCon- ference on Learning Representations, volume 2025, 54911– 54941

    Safety alignment should be mademorethanjustafewtokensdeep. InInternationalCon- ference on Learning Representations, volume 2025, 54911– 54941. Touvron,H.;Martin,L.;Stone,K.;Albert,P.;Almahairi,A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al

    work page 2025

  8. [13]

    Xu, Z.; Zhou, P.; Tang, W.; Ai, J.; Zhao, W.; Wang, K.; Peng, X.; Shao, W.; Yao, H.; and Zhang, K

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288. Xu, Z.; Zhou, P.; Tang, W.; Ai, J.; Zhao, W.; Wang, K.; Peng, X.; Shao, W.; Yao, H.; and Zhang, K

    arXiv

Show all 16 references
  1. [14]

    Yao, Y.; and Xu, X

    Pebench:Afictitiousdatasettobenchmarkmachineunlearn- ing for multimodal large language models.arXiv preprint arXiv:2503.12545. Yao, Y.; and Xu, X

    arXiv

  2. [15]

    Zhang,R.;Lin,L.;Bai,Y.;andMei,S.2024

    Large language model unlearn- ing.Advances in Neural Information Processing Systems, 37: 105425–105475. Zhang,R.;Lin,L.;Bai,Y.;andMei,S.2024. Negativepref- erence optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868. Zheng, H.; Pang,...

    2024 arXiv

  3. [16]

    In Findings of the Association for Computational Linguistics: ACL 2026, 12602–12620

    Offside: Benchmarking unlearning misinformation in multimodal large language models. In Findings of the Association for Computational Linguistics: ACL 2026, 12602–12620

    2026

  4. [2021]

    In2021 IEEE symposium on security and privacy, 141–159

    Machine unlearning. In2021 IEEE symposium on security and privacy, 141–159. IEEE. Cao,P.;Wang,C.;He,Z.;Yuan,H.;Li,J.;Chen,Y.;Liu,K.; Zhao,J.;etal.2024.Rwku:Benchmarkingreal-worldknowl- edge unlearning for large language models.Advances in Neural Information Processing Systems,...

    2024

  5. [2023]

    Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W.; Gao, C.; Ge, C.; et al

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774. Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W.; Gao, C.; Ge, C.; et al

    arXiv

  6. [2024]

    Bourtoule, L.; Chandrasekaran, V.; Choquette-Choo, C

    To each (textual sequence) its own: Improving memorized-data unlearning in large language models.arXiv preprint arXiv:2405.03097. Bourtoule, L.; Chandrasekaran, V.; Choquette-Choo, C. A.; Jia, H.; Travers, A.; Zhang, B.; Lie, D.; and Papernot, N

    arXiv

  7. [2025]

    Barbulescu, G.-O.; and Triantafillou, P

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631. Barbulescu, G.-O.; and Triantafillou, P

    arXiv

  8. [2026]

    Li,N.;Pan,A.;Gopal,A.;Yue,S.;Berrios,D.;Gatti,A.;Li, J

    Cross-modal unlearning via influential neuron path editinginmultimodallargelanguagemodels.InProceedings oftheAAAIConferenceonArtificialIntelligence,volume40, 35589–35597. Li,N.;Pan,A.;Gopal,A.;Yue,S.;Berrios,D.;Gatti,A.;Li, J. D.; Dombrowski, A.-K.; Goel, S.; Phan, L.; et al. ...

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.