{"id":"79d813c4-d199-4c11-93d0-ae8a425b68e5","arxiv_id":"2608.13478","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A concept-erasure method built on MapRoute raises the official ERR score from 0.600 to 0.721 on the Genµ2.0 benchmark, but it does not beat MapRoute on object, animal, or action categories.","lead":"MapRoute++ inserts small learned networks into Stable Diffusion so the model no longer draws requested concepts such as artworks, animals, or actions, while still drawing other things. On the Genµ2.0 benchmark it scores 0.721 on the ERR metric, 12.1% above the strongest earlier method, with the gain concentrated in scene and style concepts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLaVA-based ERR is the sole evidence for the 12.1% SOTA claim; without human calibration or proof the local LLaVA matches the official judge, the headline gap could be an evaluation artifact.","rationale":"The paper's central claim is an empirical benchmark result, so the measurement is the load-bearing link. The reader's weakest assumption (LLaVA judge faithfulness) is real and foundational. I sharpen it: the paper never shows its local LLaVA-1.6-Mistral-7B reproduces the official Genµ2.0 evaluator, and never reports human agreement or error bars. The 0.121 average gain over FADE is unevenly distributed (Object +0.265, Action +0.250, Style -0.035), so the average depends on categories where a permissive 'no' from LLaVA could inflate scores. The qualitative checks in Figures 2-3 cover only a few concepts and actually confirm LLaVA detects reported failures, so this is not an indictment of the method, but a validation gap. The correct disposition remains CONDITIONAL: the paper should release the evaluation code, report human-judged agreement, and provide confidence intervals before the SOTA claim is fully established. This does not change the reader's verdict, hence UNCHANGED.","tokens_in":6996,"tokens_out":17529,"duration_ms":157391,"concrete_test":"Have three annotators label 100 randomly sampled images per method (MapRoute++ and FADE) with the same yes/no questions as LLaVA, then compute (a) per-axis Cohen's kappa between LLaVA and the majority human label and (b) the human-judged average ERR gap between MapRoute++ and FADE. If kappa < 0.7 on Afgt/Aind/Aadv or the human-judged gap is < 0.05, the claimed 12.1% improvement is an artifact of the unvalidated judge. As a secondary check, rerun FADE with the authors' exact LLaVA pipeline; if the reproduced FADE average differs from the Table 1 value of 0.600 by more than 0.02, the baseline comparison is not apples-to-apples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; Section 4.2, Table 1) is a 0.121 absolute ERR advantage over FADE. Every ERR number is produced by the authors' local LLaVA-1.6-Mistral-7B yes/no evaluation (Section 4.1), but the paper (i) never reports agreement of this judge with human labels, (ii) never shows that this locally instantiated judge reproduces the official Genµ2.0 evaluator used for baseline scores, and (iii) reports no variance or error bars. The gap is also uneven: MapRoute++ beats FADE by 0.265 on Object and 0.250 on Action but is 0.035 worse on Style (Table 1), so the '12.1% average' hinges on categories where LLaVA may be most permissive in answering 'no' after any embedding perturbation. Since Figure 3 confirms LLaVA does detect the failures it reports (Van Gogh, Blue Jay), the judge is not obviously broken; the issue is that the benchmark's lynchpin measurement is unvalidated and possibly inconsistent across methods, so the SOTA claim is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MapRoute++, a method for visual concept unlearning in Stable Diffusion v1.4, as a submission to the Genμ2.0 Challenge Task 3. The method builds on MapRoute by adding task-specific training objectives, richer concept representations, and a semantic routing mechanism that selects per-concept mapper modules at inference. The central claim is that MapRoute++ achieves a state-of-the-art average ERR score of 0.721, a 12.1% absolute improvement over the strongest baseline FADE (0.600), while preserving unrelated and semantically adjacent concepts. The evaluation uses a local LLaVA-based yes/no judge on 20 generated images per concept, aggregated into five axes (target forgetting, retention, adjacent retention, indirect robustness, adversarial robustness) and combined via harmonic mean into ERR. The paper reports per-category and per-concept scores, qualitative successful and failure cases, and discusses surrogate selection and error analysis.","tokens_in":7286,"tokens_out":3398,"duration_ms":31584,"significance":"If the claimed results hold, the contribution is practically significant: a lightweight, frozen-model mapper that improves concept erasure and retention over prior modular and optimization-based methods would be a useful addition to the visual-unlearning toolbox. The paper is also honest about failure cases in the Style category and provides qualitative evidence that the local LLaVA judge detects obvious failures (e.g., Van Gogh brushwork, Blue Jay bird-like outputs). The method is compute-efficient and the code is promised on GitHub. However, the headline significance rests on a single unvalidated evaluation pipeline and on a set of comparisons whose internal consistency is not established; those issues are the focus of the major comments.","major_comments":[{"comment":"The headline claim of a 12.1% ERR improvement over FADE rests entirely on a locally instantiated LLaVA-1.6-Mistral-7B yes/no judge. The paper never reports agreement between this judge and human labels, nor does it show that this judge reproduces the official Genμ2.0 evaluator used for the baseline scores in Table 1. If the local judge is more permissive in answering 'no' after embedding perturbation, the entire 0.721-vs-0.600 gap could be an evaluation artifact rather than genuine erasure. Please provide human calibration on a sample of generated images or a side-by-side comparison with the official judge on a common set of prompts and images.","section":"Section 4.1, Table 1"},{"comment":"The claim that advantages are 'most pronounced in the Object and Scene categories' is contradicted by Table 1. MapRoute++ is below MapRoute on Object (0.863 vs 0.919), Animal (0.604 vs 0.796), and Action (0.857 vs 0.871); only Scene shows a substantial gain (0.707 vs 0.190). The category-specific strengths discussion should be corrected to reflect that the main advantage is in Scene, while Object performance actually degrades relative to MapRoute.","section":"Section 4.2, Table 1"},{"comment":"Table 2 states that 'Animals are combined in the objects category,' yet Table 1 reports a separate Animal category. In Table 2, Blue Jay and Labrador Retriever are listed under 'Objects,' which is inconsistent with the headline table's taxonomy. This also explains why the Table 2 average (0.7245) does not exactly match the Table 1 average (0.721), and it obscures the method's weakness on Animal concepts. Please reconcile the category labels and recompute the per-category averages accordingly.","section":"Table 2"},{"comment":"No variance or confidence intervals are reported for any ERR score, and each concept is evaluated on only 20 generated images. Given that the overall 0.121 gap is averaged over five categories and that on two categories (Animal, Style) MapRoute++ is essentially tied with or worse than FADE, the reported differences may be within noise for several individual categories. Please report error bars, seeds, or at least the per-image distribution for the main comparison.","section":"Section 4.1"}],"minor_comments":[{"comment":"The abstract claims evaluation on 'the official benchmark,' but Section 4.1 says the LLaVA-based ERR is used locally. The distinction should be stated clearly in the abstract or introduction to avoid implying that the official challenge evaluator was used for MapRoute++ scores.","section":"Abstract and Section 4.1"},{"comment":"The table header has formatting artifacts ('F ADE', 'A vg') that should be fixed; the method name 'FMN' should be expanded at first use.","section":"Table 1"},{"comment":"The hyperparameters α and β are said to default to 1, but there is no ablation or sensitivity analysis for these weights or for the top-k in semantic routing. A brief analysis would strengthen the claim that these choices are not responsible for the observed performance.","section":"Section 3.2"},{"comment":"The statement that surrogate selection 'without using information from the challenge dataset's indirect prompts, adversarial prompts, or adjacent-concept prompts' is a claim about process, not an observed property of the method. The paper should either provide a reproducible procedure for selecting surrogates or acknowledge that such selection is heuristic.","section":"Section 5.1"},{"comment":"There are several typos, e.g., 'Aurora Borialis' (Section 2), 'T able' (Table 1 caption), and 'Scenerie' (Table 2). The GitHub link for code is mentioned but not actually provided; please include a complete URL.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main reason for major revision rather than reject is that the core methodological idea (lightweight mappers with semantic routing) is plausible and the failures are honestly analyzed. However, the paper's central quantitative claim is currently supported only by an unvalidated local evaluator and by a table that contradicts its own category-level narrative. These can be fixed with additional experiments and reporting, but they are not merely cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a modest, honest extension of MapRoute, and the headline 12.1% ERR gain over FADE is real in the paper's tables but rests on a LLaVA judge that is never checked against human labels or the official Genμ evaluator. The stress-test note is right. Figure 3 shows the judge catches the failures, so it isn't obviously broken, but that doesn't make the SOTA claim solid.\n\nWhat's new: multi-synonym concept sets, per-category choice of the Stage 2 loss (dropping Lkeep2 for objects), and top-k routing. These are sensible engineering tweaks on MapRoute. The paper is transparent about its failure cases and limitations (Van Gogh, Blue Jay), which I respect.\n\nSoft spots, in order. First, evaluation. No error bars, no human agreement, no evidence that the local LLaVA reproduces the official judge. Since every number in Table 1 comes from this judge, the 0.121 gap could be an artifact of the judge being permissive after embedding shifts. Second, the paper overclaims the category story: Table 1 shows MapRoute++ is worse than MapRoute on Object (0.863 vs 0.919), Animal (0.604 vs 0.796), and Action (0.857 vs 0.871), yet the text says advantages are 'most pronounced in Object.' That's unsupported. Scene is the real win (0.707 vs 0.190). Third, the code link in the abstract is not actually provided in the camera-ready text I read. Minor: per-concept Table 2 groups Animals into Objects, and the average differs slightly from Table 1 (0.7245 vs 0.721), probably rounding.\n\nBottom line: the method is plausible, the artifacts are honest, and the benchmark is the right one for the claim, but the evaluation gap is load-bearing and needs fixing. I'd send it to review with a request for human-calibrated LLaVA or agreement stats, error bars, and a clarified per-category discussion. The incremental novelty doesn't bother me; a clean, reproducible erasure module for frozen SD models would be a useful result.","headline":"Incremental MapRoute extension with a plausible method but an unvalidated LLaVA judge; the 12.1% SOTA claim needs human calibration before it holds.","tokens_in":7797,"tokens_out":2587,"would_cite":false,"duration_ms":21428,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Surrogate-guided mapper beats concept-erasure baselines by 12.1%","keywords":["concept unlearning","visual concept erasure","text-to-image diffusion models","semantic routing","embedding mapper","surrogate concepts","ERR metric","frozen text encoder"],"falsifier":"Show a random sample of the generated images from the high-scoring concepts to independent human raters and compare their 'is the target present?' answers with LLaVA's decisions; if human 'target still visible' rates are materially higher than the reported forgetting scores, the ERR numbers overstate erasure. A cheaper check is to run an off-the-shelf detector or CLIP retrieval on the same images and compare aggregate scores.","tokens_in":6829,"feed_emoji":"🖼️","tokens_out":7757,"duration_ms":69318,"temperature":0.7,"pith_summary":"The paper claims that visual concept unlearning in text-to-image diffusion models can be done without tuning the generative model itself: a small per-concept mapper inserted between the frozen CLIP text encoder and the U-Net redirects the target concept's embedding toward a surrogate while identity training keeps every other concept's embedding unchanged. On the Genmu2.0 benchmark, the method reports an average ERR of 0.721, beating the strongest baseline by 12.1 percentage points while preserving unrelated and semantically adjacent concepts. The authors argue that this makes erasure cheap, modular, and reversible, because each concept has its own lightweight module and a semantic router applies only the relevant modules at inference. A sympathetic reader would care because, if correct, the approach offers a compute-efficient alternative to fine-tuning-based unlearning in frozen diffusion models.","feed_headline":"Small mapper beats concept-erasure baselines by 12.1%","feed_subtitle":"MapRoute++ edits only text embeddings, scoring 0.721 ERR on the official benchmark while preserving unrelated concepts.","key_machinery":"The central object is a per-concept residual mapper $M_{c_{\\text{tar}}}$, a three-layer GELU MLP with residual norm, trained to act as the conditional identity $M_{c_{\\text{tar}}}(E(c)) = E(c_{\\text{sur}})$ when $c = c_{\\text{tar}}$ and $M_{c_{\\text{tar}}}(E(c)) = E(c)$ otherwise. Training runs in two stages: a stage-1 identity loss $L_{\\text{stage1}} = \\mathbb{E}_{c \\in C} \\| M_{c_{\\text{tar}}}(E(c)) - E(c) \\|_2^2$ fixes the default behavior, and a stage-2 objective $L_{\\text{stage2}} = L_{\\text{learn}} + \\alpha L_{\\text{keep1}} + \\beta L_{\\text{keep2}}$ adds surrogate redirection and retention regularizers over a general vocabulary and a set of proper names. At inference, semantic routing computes the similarity between the prompt's mean token embedding and each stored target embedding, then applies the top-$k$ mappers token-wise to the full embedding sequence.","core_discovery":"MapRoute++ is a two-stage residual MLP per target concept. Stage 1 trains the mapper to reproduce any input embedding, so the default behavior is identity; Stage 2 adds a loss pulling the target concept's embedding toward a surrogate concept, plus two retention regularizers that keep the identity mapping on a general vocabulary and on a curated set of proper names. At inference, a router compares the prompt's mean token embedding with stored target embeddings and applies the top-k mappers token-wise to the full embedding sequence. The paper's key claim is that this sequence-level editing of text embeddings achieves an average ERR of 0.721 on Stable Diffusion v1.4 across five concept categories, with Scene rising from MapRoute's 0.190 to 0.707 and Action reaching 0.857, while Style remains the weak category with Van Gogh forgetting at 0.300 and Doodle at 0.100.","pith_inferences":["I would not treat the 0.721 ERR as a settled measure of real-world erasure until an independent judge, such as human raters or a detector, reproduces LLaVA's yes/no decisions; the paper reports no human calibration of the evaluator.","The Style-category numbers suggest a division of labor: embedding-space mappers handle discrete, mentionable concepts, whereas global style attributes require interventions in cross-attention weights or latent feature statistics, so future methods could combine both.","The routing-by-prompt-similarity idea could be tested as an open-set eraser by training mappers on a few seed concepts and checking whether nearest-neighbor routing suppresses unseen concepts that share embedding neighborhoods.","The surrogate-selection heuristic is under-specified; a concrete test would be to vary surrogate choice systematically and see whether ERR variance across surrogates exceeds the reported 12.1% gap."],"forward_implications":["If the result transfers beyond the 20 benchmark concepts, any text-to-image model with a frozen CLIP text encoder could receive a targeted erasure by training only small MLPs, without retraining or fine-tuning the diffusion model.","Erasure becomes per-concept and reversible: removing a mapper restores the original behavior, and the same base model can host many independent mappers selected at inference.","The failure pattern on styles suggests that token-embedding edits are enough for object- and scene-like concepts but hit a ceiling for global texture and style attributes, pointing to where future work must act on cross-attention or latent features.","The semantic router makes multi-concept erasure compositional, since the top-$k$ selection decides which mappers fire for a given prompt.","Because non-target embeddings are left essentially unchanged, the method can be stacked with other edits to the same frozen base model without retraining the whole pipeline."],"supporting_citations":[{"why":"Supplies the base MapRoute architecture, the per-concept residual mapper, and the semantic routing idea that MapRoute++ extends.","marker":"[10]"},{"why":"FADE is the strongest baseline, and its 0.600 average ERR is the number MapRoute++ claims to beat by 12.1 percentage points.","marker":"[22]"},{"why":"LLaVA provides the yes/no image-judging model that defines the ERR evaluation scores.","marker":"[11]"},{"why":"Stable Diffusion v1.4 is the frozen text-to-image model whose text encoder the mapper is inserted into.","marker":"[18]"},{"why":"ESD is one of the optimization-based baselines compared in the ERR table.","marker":"[6]"},{"why":"CA is one of the baseline erasure methods compared in the ERR table.","marker":"[9]"},{"why":"FMN is one of the baseline erasure methods compared in the ERR table.","marker":"[24]"}],"fun_headline_variants":["MapRoute++ beats erasure baselines by 12.1% on ERR","Semantic routing lifts scene erasure from 0.190 to 0.707","Text-embedding mapper tops SOTA by 12.1% on concept removal","Router-selected mappers erase targets, keep adjacent concepts","Action erasure reaches 0.857 ERR with MapRoute++"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That LLaVA's forced yes/no answer is a faithful proxy for whether a human would say the concept appears in the image, and the paper does not report human agreement on that judgment.","fun_headline_variants_meta":{"raw":{"variants":["MapRoute++ beats erasure baselines by 12.1% on ERR","Semantic routing lifts scene erasure from 0.190 to 0.707","Text-embedding mapper tops SOTA by 12.1% on concept removal","Router-selected mappers erase targets, keep adjacent concepts","Action erasure reaches 0.857 ERR with MapRoute++"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000516,"raw_usage":{"total_tokens":2441,"prompt_tokens":823,"completion_tokens":1618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":1532}},"tokens_in":439,"tokens_out":1618,"duration_ms":13276,"temperature":1.0,"reasoning_tokens":1532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:08:10.026994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show a random sample of the generated images from the high-scoring concepts to independent human raters and compare their 'is the target present?' answers with LLaVA's decisions; if human 'target still visible' rates are materially higher than the reported forgetting scores, the ERR numbers overstate erasure. A cheaper check is to run an off-the-shelf detector or CLIP retrieval on the same images and compare aggregate scores.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","cited_arxiv_id":null,"evidence_quote":"Supplies the base MapRoute architecture, the per-concept residual mapper, and the semantic routing idea that MapRoute++ extends."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"FADE is the strongest baseline, and its 0.600 average ERR is the number MapRoute++ claims to beat by 12.1 percentage points."}],"review_version":1}