{"id":"99f92e83-e564-45f3-92da-5fda51657be2","arxiv_id":"2512.16864","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Coupling a GRPO-trained reasoning planner with training-free attention-region injection lets RePlan execute complex multi-region image edits with higher consistency, at the cost of slightly lower effect scores, on the new IV-Edit benchmark.","lead":"RePlan improves instruction-based photo editing by first planning which parts of the image to change — an AI reads the image and the written request, then outputs a list of regions and per-region instructions — and then steering an existing editor's internal attention to exactly those regions, with no extra training. It also introduces an 800-task benchmark for hard editing requests and a small reinforcement-learning step that makes the planner more reliable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Weighted metric in §5.1 cannot reproduce Table 1; the headline quantitative claim may rest on undefined arithmetic.","rationale":"The reader's weakest assumption concerns the validity of Gemini-2.5-Pro as judge, given the shared model family with IV-Edit's construction. That remains a real concern. But a more immediate, internally verifiable problem is that the Weighted metric—the paper's main defense against consistency inflation—cannot reproduce the reported Table 1 values from the stated formula. The implied mean of Effect×Consistency is negative, which is impossible. This is not a matter of external judgment bias; it is an internal arithmetic inconsistency that affects the headline numbers. I therefore focus the stress test there. I do not claim the authors manipulated results; a typo in the formula or in the table is plausible. But as written, the paper does not support its quantitative claim. The reader's CONDITIONAL verdict is still appropriate—perhaps more strongly—because the missing raw data, corrected formula, and statistical significance are precisely what would resolve the concern. Hence UNCHANGED: the verdict remains conditional, with the condition now including metric reproducibility. Agreement is partial because the reader identified evaluator validity while this attack identifies a distinct but related weakness in the evaluation arithmetic.","tokens_in":13297,"tokens_out":6920,"duration_ms":71193,"concrete_test":"Request the per-sample Gemini scores for all models in Table 1 and recompute Weighted exactly as stated in §5.1, including any omitted normalization. If the recomputed values match Table 1, the formula in the paper is a typo and should be corrected; if they do not match, the table's Weighted column is not derived from the stated metric. Then recompute pairwise comparisons (especially Flux.1 Kontext dev ± RePlan and Qwen-Image-Edit ± RePlan) using the corrected Weighted and bootstrap confidence intervals. If the corrected Weighted gain is not significant or reverses, the central 'consistently outperforms' claim fails on the paper's own metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that RePlan consistently outperforms open-source baselines on IV-Complex editing—depends heavily on the Weighted score in Tables 1–5, since raw Consistency gains are structurally induced by attention rules 4 and 5 and the Effect dimension actually regresses on both backbones (Flux: 2.73→2.59; Qwen: 3.24→3.16). However, the definition in §5.1 is arithmetically inconsistent with the reported values. The text defines Weighted = (1/N) Σ (Target+Quality+Effect+Effect×Consistency)/4. Given the column means in Table 1, the implied mean of Effect×Consistency is negative: for Flux baseline, 4×2.49 − (3.93+3.34+2.73) = −0.04; for Flux+RePlan, 4×2.55 − (3.47+4.16+2.59) = −0.02. Since all scores are in [1,5], this is impossible. Either the formula is mis-specified (missing a normalization, a different weighting, or a different aggregation) or the table numbers do not follow from the stated metric. Because Weighted is precisely the metric introduced to prevent consistency inflation, an unreproducible Weighted means we cannot tell whether RePlan's headline advantage over Flux baseline (Overall +0.24, Weighted +0.06) survives on a bias-adjusted aggregate. The reported raw Weighted gains are also small (+0.06 on Flux; +0.29 on Qwen), and no error bars are given, so statistical significance is unestablished. This internal metric inconsistency is at least as load-bearing as the Gemini-judge concern: even granting the judge's validity, the paper's own quantitative core cannot be verified from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RePlan, a plan-then-execute framework for instruction-based image editing under 'Instruction-Visual Complexity'. A VLM planner (Qwen2.5-VL-7B) is trained via two-stage GRPO with roughly 1k instruction-only examples to output region-aligned guidance (bounding boxes plus per-region hints). The diffusion decoder (Flux.1 Kontext or Qwen-Image-Edit, both MMDiT) executes this guidance through a training-free attention-region injection mechanism governed by five attention rules. The authors introduce IV-Edit, an 800-pair benchmark emphasizing fine-grained grounding and knowledge-intensive edits, and evaluate RePlan against several open and closed baselines using Gemini-2.5-Pro as a judge. They report consistent improvements in Overall and Consistency, along with ablations on planner choice, reasoning, and staged RL, and a bbox-perturbation robustness test.","tokens_in":13490,"tokens_out":5554,"duration_ms":53492,"significance":"If the quantitative claims are valid, the central contribution is notable: a small RL-trained planner plus a training-free attention regime can substantially improve open-source MMDiT editors on hard grounding tasks without requiring paired editing data. The attention-region injection mechanism is clearly specified, and the internal ablations (planner choice in Table 2, reasoning on/off and staged RL in Table 3) support the importance of the planner and the RL training. The bbox perturbation robustness test (Table 5) is a genuine and commendable probe of the method's dependence on region accuracy. However, the evaluation protocol currently prevents acceptance: the Weighted metric used for the headline comparisons is arithmetically inconsistent with the reported numbers, and the benchmark construction/judge share the same model family, with no human evaluation. The claimed edge over baselines is therefore not yet established.","major_comments":[{"comment":"The definition 'Weighted = (1/N) Σ (Target+Quality+Effect+Effect×Consistency)/4' cannot reproduce Table 1. Since all ratings are in [1,5], Effect×Consistency ≥ 1 for every sample. For the Flux baseline, the lower bound of the weighted average is (3.93+3.34+2.73+1)/4 = 2.75, but the table reports 2.49. The implied mean of Effect×Consistency would have to be about −0.04, which is impossible. Because Weighted is the metric introduced to counter consistency inflation, the central comparison (e.g., +0.06 for Flux, +0.29 for Qwen) is unsupported. Please correct the formula or provide per-sample computation and verify all Weighted values in Tables 1, 2, 3, and 5.","section":"§5.1, Eq. (Weighted) and Table 1"},{"comment":"The benchmark construction and evaluation are not independent: Gemini-2.5-Pro was used to generate and filter instructions (Appendix C), Qwen2.5-VL-72B annotated ground-truth bounding boxes, and Gemini-2.5-Pro also serves as the grading judge (Section 4). The judge therefore shares a model family with the benchmark generator. More seriously, RePlan's largest measured gains are in Consistency (+0.76 on Flux, +1.45 on Qwen in Table 1), and consistency is directly hard-enforced by attention rules 4 and 5, which restrict background tokens to attend only to the global text. Without a human evaluation or an independently developed judge, the risk that Gemini's ratings reward what the mechanism trivially enforces cannot be dismissed. Provide a human study on a representative sample or use a second, independently built VLM judge, and report agreement between judges.","section":"§4 Evaluation Protocol and Appendix C"},{"comment":"The Effect dimension regresses on both backbones (Flux 2.73→2.59; Qwen 3.24→3.16), while the largest gains are in Consistency. This contradicts the paper's claim of 'consistently outperforming' and 'improving overall fidelity' in the abstract and introduction. The Weighted metric was designed to handle exactly this trade-off, but it is arithmetically broken (see above). Please provide a per-task breakdown, test whether the Effect regression is statistically significant, and discuss whether the attention rules trade away editing strength for preservation. Without this, the net benefit of RePlan on edit quality remains unclear.","section":"Table 1, Effect column"},{"comment":"The quantitative claims lack statistical validation. The reported comparisons are aggregate 5-point means over roughly 800 samples, with no error bars, confidence intervals, or significance tests. The headline Weighted improvement for Flux is +0.06 (2.49→2.55), a difference likely below inter-annotator noise. Report bootstrap confidence intervals, per-sample distributions, or paired significance tests (e.g., Wilcoxon) for the main comparisons, and make clear whether the reported numbers are means over all samples or medians.","section":"§5.1 / all tables"}],"minor_comments":[{"comment":"Typos: 'Specificly' in Section 4, 'anlysis' in §3.2, 'ensuring' in §3.3 ('This ensuring global stylistic coherence'), and inconsistent naming 'Gemini2.5-pro' in Table 2 vs 'Gemini-2.5-Pro' elsewhere. The formula in §5.1 uses 'Weighted=P samples...' due to typesetting; please use proper summation notation.","section":"Throughout"},{"comment":"The citation to Goodfellow et al. (2016) in the context of VLM reasoning abilities is inappropriate (it is a deep-learning textbook). Please replace it with a relevant reference or remove it.","section":"Related Work / References"},{"comment":"The paper mentions 'manual verification' and 'manual sample-by-sample filtering' for the test set, but provides no details on annotator expertise, number of annotators, or inter-annotator agreement. Add these details for reproducibility.","section":"§4 / Appendix C"},{"comment":"The 'Reasoning quality reward' uses the length of <think> content as a proxy, with a cap. This is an ad-hoc axiom; please justify this proxy or provide evidence that longer reasoning correlates with plan quality. Also clarify whether the '1k instruction-only examples' are used for supervised alignment before RL and what that alignment consists of.","section":"§3.4"},{"comment":"Consider marking the second-best open-source model clearly; the underline is easy to miss in the printed table. Also, the closed-source models are included without discussion of their access/cost differences.","section":"Table 1, row ordering"}],"recommendation":"major_revision","confidential_remarks":"The central mechanism is creative and the internal ablations suggest the approach works, but the evaluation is currently not trustworthy due to the broken Weighted metric and the circularity between benchmark construction and judging. The authors should be asked to fix the metric, provide independent human or second-judge evaluation, and add statistical error bars. If these are addressed, the paper could become acceptable. The strictness of the report is proportionate to the strength of the headline claims, which currently outrun the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. The core idea is good: use a small RL-trained VLM to emit bbox-level edit hints, then enforce them with a training-free attention mask in an MMDiT. The five attention rules in Sec. 3.3 are concrete and plausible, and the design choices — hint isolation, region constraint, background constraint — directly address the multi-region contamination problem. The two-stage GRPO reward with format/reasoning first, then image-level feedback, is also sensible, and the ablations (Tables 2–4) support it. The bbox perturbation test (Table 5) is a nice robustness check. IV-Edit, with its 7 referring and 16 task types, is a useful asset for the community, even if it's not the first reasoning-oriented editing benchmark.\n\nWhat worries me is the evaluation arithmetic. The Weighted score in Sec. 5.1 is defined as the mean of (Target+Quality+Effect+Effect×Consistency)/4. Plug the Table 1 means into that for the Flux baseline: (3.34+3.93+2.73 + mean(E×C))/4 = 2.49, which forces mean(E×C) = -0.04. Since E and C are 1–5 ratings, their product is at least 1. The same contradiction appears for Flux+RePlan (-0.02). So the formula as written cannot reproduce the reported Weighted values. That is a load-bearing flaw, because the paper introduces Weighted precisely to guard against consistency inflation, and uses it to claim a fair advantage over baselines. If Weighted is undefined arithmetic, we cannot tell whether the method's apparent edge survives a bias-adjusted aggregate. The raw Weighted gains are also small (+0.06 on Flux, +0.29 on Qwen), with no error bars.\n\nThe second issue is the evaluation loop. IV-Edit's instructions and filtering were produced by Gemini-2.5-Pro and Qwen2.5-VL-72B (Appendix C), and the final grading is by Gemini-2.5-Pro. The largest gains are in Consistency, which rules 4–5 of the attention mask enforce by construction; Effect actually regresses on both backbones (2.73→2.59; 3.24→3.16). There is no human evaluation to calibrate the judge. This doesn't prove the gains are fake, but it means the headline \"consistently outperforms\" is not established.\n\nNet: the method is worth knowing about, and the internal evidence — planner choice, staged RL, bbox robustness — argues that region-level guidance helps. But the quantitative core as written is unreliable. This deserves a serious referee, but the authors need to fix the metric definition, add human evaluation or at least a validated judge, and report variance before I'd trust the results. If they do, this could be a solid paper.\n\nRecommendation: send to review, prepare for heavy revision.","headline":"Clever training-free attention injection and a new benchmark, but the Weighted metric in Table 1 is arithmetically impossible and the evaluation is too self-referential to support the headline claim.","tokens_in":14250,"tokens_out":4753,"would_cite":false,"duration_ms":42937,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RePlan shows that a reasoning-driven region planner plus training-free attention masking lets open-source diffusion editors handle complex, fine-grained image edits with far fewer artifacts and better grounding.","keywords":["instruction-based image editing","region planning","attention masking","multimodal diffusion transformer","GRPO reinforcement learning","instruction-visual complexity","image editing benchmark","grounding"],"falsifier":"A human-rated study on a random subset of IV-Edit comparing RePlan against Flux.1 Kontext dev on the target and effect dimensions; if humans do not rate RePlan higher on those dimensions, the central claim weakens. A sharper ablation: feed the same region hints to Flux as a global prompt without attention injection — if scores are unchanged, the attention mask is not the cause of the gains.","tokens_in":12954,"feed_emoji":"🎯","tokens_out":3947,"duration_ms":36086,"temperature":0.7,"pith_summary":"RePlan argues that state-of-the-art instruction-based editors fail on cluttered scenes because vision-language models serve only as coarse semantic encoders. The paper's central claim is that making the VLM output explicit region-aligned guidance — bounding boxes plus per-region edit hints — and then constraining the diffusion model's attention so each region only attends to its own hint yields substantially more precise edits. With just about 1,000 instruction-only examples used to reinforce the planner via GRPO, RePlan reports consistent gains over open-source baselines trained on orders of magnitude more data, and closes much of the gap with proprietary editors on the new IV-Edit benchmark. The paper also introduces IV-Complexity as a challenge axis and IV-Edit as a curated testbed for it.","feed_headline":"Region planning sharpens open-source image editing on hard tasks","feed_subtitle":"A 1k-sample RL-trained planner and attention masks reduce edit spillover and ground edits to exact regions.","key_machinery":"The attention region injection mask is the central mechanism: a binary mask over the token sequence of an MMDiT (a diffusion transformer that jointly attends to text, image, and noise-latent tokens). The mask enforces five rules — intra-group connection, hint isolation, full image–latent interaction, region constraint (region tokens see only their own hint and the global hint), and background constraint (background tokens see only the global hint). It is training-free and applied at every attention layer, and it is what lets the model execute the planner's region-aligned guidance precisely.","core_discovery":"On the paper's own terms, the discovery is that a reasoning-driven planner plus attention masking is enough to unlock precise multi-region editing in a pretrained multimodal diffusion transformer. The VLM planner (Qwen2.5-VL 7B, reinforced with GRPO) decomposes an instruction into structured reasoning and a set of bounding-box/hint pairs, including negative hints that tell regions to stay unchanged. The editor then applies a training-free binary attention mask with five rules: text hints are isolated from each other, each image region attends only to its own hint and the global hint, background tokens attend only to the global hint, and image–latent tokens remain fully connected. This keeps","pith_inferences":["If the attention-isolation principle transfers, other diffusion architectures (e.g., cross-attention U-Nets) could gain similar spillover resistance by constraining which text tokens each spatial region attends to.","The structured region-hint output makes interactive editing practical: a user can directly adjust a bounding box or hint and re-run the same one-pass pipeline.","One testable extension: swap the planner for a stronger base VLM without RL to isolate how much of the gain comes from the attention mask versus the reinforced reasoning.","The benchmark's consistency dimension is structurally aligned with the method's design, so future work should report human ratings or an independent judge before generalizing the claimed margin."],"forward_implications":["Open-source MMDiT-based editors can be made substantially more precise on complex grounding tasks without retraining the diffusion model.","Multiple region edits can be executed in a single pass, avoiding the accumulated errors and extra compute of iterative inpainting.","A small reinforcement-learned planner (about 1k instruction-only examples) can produce reliable region guidance, reducing dependence on large paired editing datasets.","Region-level negative prompts are supported directly, preventing unintended changes to semantically similar areas.","The IV-Edit benchmark provides a public testbed for instruction–visual complexity, with fine-grained grounding and knowledge-intensive edits."],"fun_headline_variants":["Reasoning planner guides precise multi-region edits","Attention masks and 1K RL examples tame complex editing","Plan-then-edit: reasoning and region grounding fix edits","Small RL planner beats large models on hard image editing","Grounding edits with reasoning: RePlan nails intricate tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation premise: Gemini-2.5-Pro's five-point ratings on IV-Edit are a valid, bias-free measure, even though the benchmark's instructions and filters were generated with the same model family and the consistency dimension favors edits that leave regions untouched.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning planner guides precise multi-region edits","Attention masks and 1K RL examples tame complex editing","Plan-then-edit: reasoning and region grounding fix edits","Small RL planner beats large models on hard image editing","Grounding edits with reasoning: RePlan nails intricate tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1064,"prompt_tokens":701,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":284}},"tokens_in":445,"tokens_out":363,"duration_ms":4137,"temperature":1.0,"reasoning_tokens":284,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T15:23:23.600017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human-rated study on a random subset of IV-Edit comparing RePlan against Flux.1 Kontext dev on the target and effect dimensions; if humans do not rate RePlan higher on those dimensions, the central claim weakens. A sharper ablation: feed the same region hints to Flux as a global prompt without attention injection — if scores are unchanged, the attention mask is not the cause of the gains.","supporting_citations":[],"review_version":1}