{"id":"d67b5b0a-776b-4386-9be6-ce54e860f18c","arxiv_id":"2608.11149","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PRMU provides a corpus-free, person-centric multimodal unlearning benchmark with proxy corpora and neighbor-based locality evaluation, plus SGPE, a projection-editing baseline.","lead":"This paper introduces PRMU, a benchmark for testing whether multimodal AI models can forget facts about specific people without access to their original training data. It also proposes a lightweight unlearning method, SGPE, and shows that common unlearning methods harm related knowledge and can be reactivated through images.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The NKS filter keeps probes answerable by Qwen2.5-VL-32B, not by the evaluated models; LLaVA-1.5-7B's pre-unlearning Forget Set QA text is only 0.630, so current 'forgetting' scores partly measure suppression of knowledge the model never possessed.","rationale":"I agree with the Reader that the fixed-reference NKS filtering is the weakest load-bearing assumption. The Table 2 'Before' rows supply direct evidence: for LLaVA-1.5-7B, about a third of Forget Set QA(text) probes are not answerable before unlearning, and even Qwen2.5-VL-7B, same model family as the reference, drops to 0.606. Because the same reference-filtered probe sets are used for Neighbor and Other evaluation, every locality and utility comparison in the paper is entangled with whether the target model knows the tested facts. This is not a disagreement with current consensus; it is an internal validity condition. A model-specific known-subset re-analysis is straightforward with the released data and would settle whether the reported trade-offs survive. I do not see a reason to change the Reader's CONDITIONAL verdict: the concern is addressable, the benchmark has substantial construction effort, and the SGPE contribution is secondary. I would keep the paper conditional on this re-analysis and on code/data release. A separate detail worth checking during revision is Finding 2's claim that vision-language probes have consistently higher residual scores than text-only probes; several rows of Table 2 show the opposite, so that sentence needs clarification or correction, but it is not the load-bearing issue.","tokens_in":15964,"tokens_out":9342,"duration_ms":89955,"concrete_test":"Using PRMU's released probe set (or the Table 2 'Before' rows), partition each model's Forget/Neighbor/Other sets into known and unknown probes by pre-unlearning correctness: for generation probes use a ROUGE-L threshold (e.g., >= 0.5 or exact-answer match after manual normalization) and for MCQ use correct choice. Recompute the entire main table on each model's known subset only, for all methods and both single-target and batch-target settings, and report the known proportion per model/set. If the qualitative findings (baseline trade-offs, SGPE's trade-off, Findings 1-4) are unchanged, the fixed-reference filter is not invalidating; if they change, or if LLaVA/Qwen-7B known subsets are too small to support the conclusions, PRMU needs per-model NKS filtering or conditional reporting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that PRMU's Forget, Neighbor, and Other sets contain knowledge the evaluated MLLM actually possesses; otherwise 'forgetting' is not unlearning and 'locality degradation' is not collateral damage to retained knowledge. The Native Knowledge Filtering section makes this conditional on Qwen2.5-VL-32B alone: probes with Token-F1 > 0.5 on that reference are kept, explicitly for a unified benchmark. Table 2's Before rows show this fails for the actual test models. In the batch setting LLaVA-1.5-7B has pre-unlearning Forget Set QA(text) = 0.630, QA(vis) = 0.479, AA(text) = 0.630; Qwen2.5-VL-7B has QA(text) = 0.606. A substantial fraction of supposedly known probes are not answerable by these models. Consequently, methods that lower ROUGE-L on those probes may be inducing refusals or generic outputs on questions the model never knew, not removing acquired person knowledge. The same fixed filter applies to Neighbor and Other sets, so the reported forgetting-locality trade-offs and SGPE's advantage are not cleanly interpretable as preserving related knowledge. The Appendix is said to contain MLLM knowledge coverage statistics, but the main text does not provide a per-model known-subset analysis or re-run the evaluation on probes each model can answer natively. This is load-bearing because all four findings and the benchmark's validity depend on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PRMU, a corpus-free benchmark for person-centric knowledge unlearning in multimodal large language models (MLLMs). PRMU is constructed from public-figure profiles derived from Wikidata and Wikipedia, with 1,080 targets, 50,649 textual probes, and 41,303 visual probes. A Native Knowledge Score (NKS) filter using Qwen2.5-VL-32B retains probes that are likely acquired during pretraining. The benchmark provides a corpus-free protocol in which unlearning methods receive only the original model and a target specification, plus a model-generated proxy corpus. It evaluates forgetting, neighbor-based locality, utility, and adversarial robustness under single-target and batch-target settings. The paper also proposes SGPE, a lightweight corpus-free unlearning baseline using knowledge displacement, protected projection editing, and a similarity gate. Experiments on LLaVA-1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B compare SGPE with GA, RT, NPO, and DPO, reporting that existing methods suffer from unfavorable forgetting-locality trade-offs and multimodal reactivation, while SGPE offers a more competitive trade-off.","tokens_in":16354,"tokens_out":6820,"duration_ms":69759,"significance":"If the reported findings hold, PRMU addresses a genuine gap: existing MLLM unlearning benchmarks either assume access to original forget/retain corpora or rely on injected knowledge, whereas PRMU targets naturally acquired knowledge in a corpus-free setting. The benchmark has substantial scope, including external-knowledge-based probes, neighbor sets over multiple relationship types, adversarial attack styles, and a proxy-corpus mechanism for adapting existing unlearning methods. A notable strength is that the evaluation is not circular: probes are built from Wikidata/Wikipedia rather than from the unlearning methods or the proxy corpus. The main risk is the transferability of the NKS filter across models of different sizes and knowledge distributions; unless that risk is resolved, the numerical findings are conditional on knowledge being native to the reference model rather than to the evaluated models. The contribution is potentially valuable, but the central validity concern requires revision.","major_comments":[{"comment":"The NKS filter defines 'known knowledge' through Qwen2.5-VL-32B with Token-F1 > 0.5, but the evaluated models are smaller and have different knowledge distributions. Table 2's Before rows show that this assumption fails for the actual test models: in the batch-target setting, LLaVA-1.5-7B has Forget QA(text) = 0.630 and QA(vis) = 0.479, and Qwen2.5-VL-7B has Forget QA(text) = 0.606, before any unlearning. This means a substantial fraction of Forget probes are not natively answerable by these models, so the reported 'forgetting' improvements and the resulting trade-offs in Findings 1-4 partly reflect suppression of knowledge the model never possessed rather than removal of acquired person knowledge. The manuscript says the Appendix contains knowledge-coverage statistics, but the main-text conclusions are not conditioned on per-model native subsets. Please re-run or re-analyze the evaluation on probes each model can answer natively, or state explicitly that PRMU measures suppression relative to the reference model rather than unlearning for each target model.","section":"Native Knowledge Filtering / Table 2"},{"comment":"Finding 4, that scaling to multiple targets increases forgetting difficulty and collateral interference, is not established by the current design. Table 4's nested batches are said to control target composition, but the Before rows differ substantially across batch sizes: Forget QA(text) is 0.666 for N=50, 0.680 for N=100, and 0.777 for N=150 on Qwen3-VL-8B. Because average baseline knowledge differs across the subsets, the absolute and relative post-unlearning scores cannot separate a scale effect from a target-difficulty effect. A proper analysis would vary batch size over matched or repeated random target samples and report variance across the samples.","section":"Table 4 / Impact of Deletion Scale"},{"comment":"SGPE is a central contribution, but the main text does not report the hyperparameters needed to reproduce it. Equations (3)-(6) involve lambda_u, lambda_z, lambda_c, gamma, T, and tau, and the text also requires selecting the editable layer ell, the prototype banks, and the optimization procedure. Without these values, SGPE cannot be reimplemented and the paper's comparative claims about SGPE cannot be independently verified. Please provide a complete hyperparameter table and describe the optimization budget, calibration procedure for thresholds, and any early-stopping or regularization choices.","section":"Experimental Setup / Equations (3)-(6)"}],"minor_comments":[{"comment":"The NKS threshold 'Token-F1 > 0.5' appears only in the figure; the main text should formally define Token-F1 and explain how model answers are parsed and compared with ground-truth answers.","section":"Figure 1"},{"comment":"The signs of N3 and N4 in Eq. (2) are not explained in the notation: state explicitly that N3 and N4 are normalized distances and therefore enter the composite score negatively, and clarify the ranges of all five components.","section":"Equation (2)"},{"comment":"No error bars, standard deviations, or repeated runs are reported for any of the main comparisons. At minimum, report variance over target subsets or random seeds for the headline single-target and batch-target results so that the trade-offs in Fig. 3 are not driven by a single run.","section":"Tables 2-6"},{"comment":"The attack-style abbreviations (MC, Para, CH, RP, IR, RC, PEC, CN, CF, TA, FS, MP, BCC, CL) are only defined in the Appendix; please define them in the caption or in the main text for readability.","section":"Figure 4"},{"comment":"Table 2 is extremely dense, with 24 metric columns per setting. Consider splitting the results by probe type or using grouped bar plots to make the bidirectional trade-offs easier to inspect.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and the SGPE method are potentially useful, but the NKS transferability issue is load-bearing for the central claims and should be resolved before publication. The paper would be substantially stronger if the authors provided per-model native-knowledge coverage, re-ran the evaluation on per-model known subsets, and gave complete hyperparameters for all methods. I do not see a circularity problem, since the evaluation probes come from external knowledge bases rather than from the unlearning methods or the proxy corpus."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. PRMU is a real contribution: the first person-centric MLLM unlearning benchmark that does not need the original forget or retain corpora, with neighbor-based locality evaluation and adversarial probes. The construction is careful and the scale is impressive: 1,080 targets, roughly 92K probes, three MLLMs, four baselines plus SGPE, single- and batch-target settings. This could become a standard testbed for multimodal knowledge removal.\n\nThe soft spot is load-bearing. The NKS filter keeps probes that Qwen2.5-VL-32B answers, not probes that the evaluated model necessarily knows. Table 2 shows it: LLaVA-1.5-7B pre-unlearning QA(text) on the Forget Set is 0.630, Qwen2.5-VL-7B is 0.606. A large fraction of the 'forget' probes are not natively answerable. When GA, NPO, or RT push ROUGE-L down on those probes, they may be inducing refusals or generic outputs on questions the model never knew, not removing acquired knowledge. The same fixed filter applies to Neighbor and Other sets, so the forgetting-locality trade-off and SGPE's reported advantage are not cleanly interpretable as preserving related knowledge. The authors say coverage statistics are in the appendix, but the main text never re-runs evaluation on probes each model answers natively. That re-analysis is needed.\n\nNothing else is as serious. No error bars is sloppy for a benchmark paper, and SGPE's hyperparameters are under-specified, but those are minor. The citation pattern is fine; PPU-Bench is properly acknowledged as the closest prior work.\n\nThis paper is for people working on multimodal unlearning or evaluating knowledge removal in MLLMs. If the NKS issue is addressed, it deserves to be a widely used benchmark.\n\nMy recommendation: accept for peer review, expect major revision. The core idea is sound and the validity concern is addressable with a per-model known-subset analysis or a re-scored evaluation.","headline":"A genuinely useful corpus-free multimodal unlearning benchmark, but the NKS filter is calibrated to Qwen2.5-VL-32B, so the forgetting scores for smaller models partly measure suppression of knowledge they never had.","tokens_in":16829,"tokens_out":3406,"would_cite":true,"duration_ms":31902,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PRMU claims that realistic person-centric unlearning in multimodal models must be evaluated without original training corpora, with neighbor-based locality, and that current methods fail this test.","keywords":["machine unlearning","multimodal large language models","person-centric knowledge","corpus-free benchmark","locality preservation","knowledge reactivation","PRMU","SGPE"],"falsifier":"Measure each target model's own pre-unlearning accuracy on the PRMU Forget Set. If a substantial fraction of Forget Set probes are not answerable by that model—for instance, LLaVA-1.5-7B's 'Before' QA(text) score is 0.630 in the batch setting—then the benchmark's forgetting scores conflate unlearning with refusal of questions the model never knew, and the NKS filter should be per-model rather than reference-model-based.","tokens_in":15801,"feed_emoji":"🧹","tokens_out":6581,"duration_ms":94741,"temperature":0.7,"pith_summary":"The paper tries to establish that machine unlearning of a specific person from a multimodal large language model should be testable in the realistic situation where the requester supplies only the person's name and a reference image, not the original training data. To make that test possible it builds PRMU, a benchmark of 1,080 public figures with 50,649 textual and 41,303 visual probes, all filtered to knowledge that a strong reference model already holds. The benchmark adds a Neighbor Set of related individuals so that collateral damage to nearby knowledge is measured separately from general utility loss. Running four existing unlearning methods and a new lightweight baseline, SGPE, the paper reports that current methods sacrifice related knowledge for forgetting, that vision-language probes expose residual knowledge text probes miss, and that larger batches of deletion requests worsen both problems.","feed_headline":"Benchmark: erasing one person from an AI hurts related people","feed_subtitle":"A corpus-free benchmark with 91,000 probes shows existing unlearning methods leave multimodal traces and damage neighbor knowledge.","key_machinery":"The load-bearing objects are the PRMU pipeline and SGPE. PRMU uses a Native Knowledge Score (NKS) filter: a reference multimodal model must answer a probe with Token-F1 above 0.5, so only knowledge the model family already possesses enters the Forget, Neighbor, and Other Sets. The Neighbor Set is built from a composite score over five person-level signals—occupation match, nationality match, birth-year distance, popularity distance, and CLIP visual similarity—so locality is measured against people who resemble the target, not just random facts. SGPE carries the argument that unlearning can be localized: it derives displacement anchors from model-generated biographies, facts, and QA pairs, projects edits into the orthogonal complement of a target-blind activation basis, and activates the edit through a similarity gate comparing target prototypes to background prototypes. That gate is what converts a global parameter update into an input-conditioned unlearning operator.","core_discovery":"On the paper's own terms, PRMU is the first person-centric multimodal unlearning benchmark that is corpus-free: unlearning methods receive only the original model and a target specification, while a target-specific proxy corpus generated by the model itself stands in for inaccessible training data. The benchmark's central discovery is a systematic trade-off: gradient ascent, rejection tuning, preference optimization, and direct preference optimization all reduce target answers, but at the cost of larger drops on the Neighbor Set than on unrelated knowledge, and their residual scores are consistently higher on vision-language probes than on text-only probes. PRMU attributes this to multimodal associations that reactivate target knowledge, and to the absence of locality-aware evaluation in prior benchmarks. Its SGPE baseline, which writes edits into a protected parameter subspace and gates them by target-relevant similarity, is offered as evidence that a corpus-free method can suppress target knowledge while keeping most neighbor and unrelated knowledge intact.","pith_inferences":["Editorial inference: the same protocol could transfer beyond public figures to any entity-level deletion—brands, organizations, or fictional characters—provided a probeable profile and reference image exist.","Editorial inference: a direct test of the benchmark's assumption would be to compute each target model's own pre-unlearning accuracy on the Forget Set; low 'Before' accuracy would mean the Forget Set partly measures suppression of knowledge the model never had, and per-model NKS filtering would be the natural fix.","Editorial inference: the similarity-gate mechanism suggests a general recipe for corpus-free editing: use the model itself to generate displacement data, protect a calibration-defined subspace, and condition the edit on prototype similarity—this could apply to other knowledge types such as copyrighted text or harmful capabilities."],"forward_implications":["If PRMU's findings hold, text-only evaluations of MLLM unlearning understate residual knowledge; vision-language probes should be part of any deletion test.","Neighbor-based locality scores should accompany forgetting metrics, because collateral damage concentrates on related individuals rather than unrelated knowledge.","Corpus-free deletion protocols are feasible: model-generated proxy corpora can substitute for inaccessible forget and retain data, and SGPE shows a parameter-editing approach can exploit them.","Batch deletion requests introduce cross-target interference, so scalable unlearning needs methods that treat multiple deletions jointly rather than independently."],"supporting_citations":[{"why":"Defines machine unlearning and the retraining-free removal problem that PRMU operationalizes for multimodal models.","marker":"Bourtoule et al. 2021"},{"why":"MLLMU-Bench, the synthetic/injected-knowledge benchmark PRMU contrasts with; supplies the baseline paradigm and the need for corpus-free evaluation.","marker":"Liu et al. 2025b"},{"why":"PPU-Bench, the closest person-centric benchmark; PRMU extends it with corpus-free protocol, proxy corpus, and neighbor-based locality.","marker":"Guang et al. 2026"},{"why":"Introduces gradient ascent, one of the four unlearning baselines adapted to the proxy corpus.","marker":"Jang et al. 2023"},{"why":"Introduces Negative Preference Optimization (NPO), one of the baselines whose forgetting-locality trade-off is measured.","marker":"Zhang et al. 2024"},{"why":"Introduces rejection tuning, another baseline evaluated under the corpus-free protocol.","marker":"Ishibashi and Shimodaira 2023"},{"why":"Defines LLaVA-1.5, one of the three MLLMs on which the benchmark and baselines are evaluated.","marker":"Liu et al. 2024"},{"why":"Qwen3-VL, one of the evaluated MLLMs; also anchors the Qwen family architecture used in the experiments.","marker":"Bai et al. 2025"},{"why":"Cross-modal editing method that assumes explicit corpora; cited as evidence that current multimodal unlearning lacks corpus-free locality evaluation.","marker":"Li et al. 2026"}],"fun_headline_variants":["Corpus-free unlearning benchmark exposes multimodal memory leaks","Erasing one person from AI hurts others: new benchmark measures it","Vision-language probes revive erased people in multimodal LLMs","PRMU: first benchmark for unlearning without original training data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that a probe judged 'native knowledge' by the reference model Qwen2.5-VL-32B is also knowledge possessed by each evaluated smaller model; if a model cannot answer a Forget Set probe before unlearning, then measuring a drop in that probe counts suppression of absent knowledge, not unlearning.","fun_headline_variants_meta":{"raw":{"variants":["Corpus-free unlearning benchmark exposes multimodal memory leaks","Erasing one person from AI hurts others: new benchmark measures it","Vision-language probes revive erased people in multimodal LLMs","PRMU: first benchmark for unlearning without original training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1680,"prompt_tokens":975,"completion_tokens":705,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":637}},"tokens_in":591,"tokens_out":705,"duration_ms":82529,"temperature":1.0,"reasoning_tokens":637,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:11:46.608766+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure each target model's own pre-unlearning accuracy on the PRMU Forget Set. If a substantial fraction of Forget Set probes are not answerable by that model—for instance, LLaVA-1.5-7B's 'Before' QA(text) score is 0.630 in the batch setting—then the benchmark's forgetting scores conflate unlearning with refusal of questions the model never knew, and the NKS filter should be per-model rather than reference-model-based.","supporting_citations":[],"review_version":1}