Pith. sign in

REVIEW 6 major objections 7 minor 62 references

Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editing

T0 review · 6 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An MLLM planner decomposes complex image-editing instructions into masks and boxes, improving existing edit models.

desk verdict X-Planner is a solid, useful planning system for complex image editing, with a fixable Table 2 error and an unvalidated pseudo-label pipeline that should be discussed but doesn't sink the paper. read the letter →

arxiv 2507.05259 v1 pith:RPWRNG5J submitted 2025-07-07 cs.CV

Chun-Hsiao Yeh , Yilin Wang , Nanxuan Zhao , Richard Zhang , Yuheng Li , Yi Ma , Krishna Kumar Singh This is my paper
classification cs.CV
keywords instruction-basedimageeditingmultimodallargelanguagemodelchain-of-thoughtreasoninginstructiondecompositionsegmentationmasksinsertionboundingboxcomplexbenchmarkdiffusionmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the bottleneck in instruction-based image editing is not the editing model itself but how complex, indirect user instructions are presented to it. It introduces X-Planner, a multimodal large language model that reads a source image and a complex instruction, reasons step by step, and emits a plan of simpler sub-instructions, each tagged with an edit type and a segmentation mask, plus a bounding box for insertion edits. The claim is that feeding these structured plans to existing editing models produces edits that are better localized and more faithful to the user's intent than giving the raw instruction, and that it can even match or beat human-supplied masks. To support this, the paper builds a large-scale automatically annotated dataset, COMPIE, and a new benchmark for complex editing instructions.

What carries the argument

The load-bearing object is X-Planner itself, an MLLM built on the GLaMM-style grounding architecture (Vicuna-7B plus a segmentation decoder) and fine-tuned on COMPIE. Its chain-of-thought output couples language decomposition with spatial control: each sub-instruction names an editing anchor phrase, and the segmentation decoder converts that phrase into a mask, while insertion instructions carry an additional predicted bounding box. The mask is refined according to edit type—used directly for texture/color/background, dilated by 20% for shape and replace edits, and set to the whole image for style changes—so the control signal matches the spatial footprint of each kind of edit. The mechanism works because it turns an ambiguous global request into a sequence of localized, executable operations that a conventional editing model can follow without over-editing.

What would settle it

Sample images from COMPIE and compare each training mask/box against human annotations: if IoU for local texture/color masks or box coverage for insertion edits is far below what the downstream editing models need, the planner's control signals are unreliable. A cleaner test is to train a version of X-Planner on a small human-corrected subset of COMPIE and compare downstream edit scores; if the corrected version does not improve, localization error is not the limiting factor.

Watch

Extended reading notes

Core claim

X-Planner's central discovery is that complex image editing can be recast as planning: an MLLM decomposes a single complex instruction into one to five simpler sub-instructions, assigns each an edit type (local texture, local color change, shape change, replace, background, style, remove, insertion), and produces a segmentation mask for the edited region, with sizes tailored to the edit type—tight for color/texture, dilated for shape changes and replacements, full-image for style. For insertion edits, the model additionally predicts a bounding box that locates where the new object should appear, using world knowledge rather than external detectors. Trained on the COMPIE dataset of over 260K complex-simple instruction pairs, this planner acts as a model-agnostic front end: the sub-instructions and control regions are fed iteratively into whichever editing model best fits each edit type. On MagicBrush, X-Planner plus UltraEdit with predicted masks and boxes reports lower L1 distance than UltraEdit with human-labeled masks, and on the proposed COMPIE benchmark it improves both UltraEdit and InstructPix2Pix* on most metrics, especially on MLLM-based alignment scores that better capture complex instruction following.

Load-bearing premise

The training pseudo-labels—Grounded SAM masks for anchors and MLLM-predicted bounding boxes for insertions—are assumed accurate enough to teach the planner correct localization, and no human verification guarantees this in the automated pipeline.

Editorial extensions

If this is right

  • Existing editing models such as UltraEdit and InstructPix2Pix* improve on complex-instruction benchmarks when given X-Planner's decomposed sub-instructions and masks instead of the raw instruction.
  • On the MagicBrush single-turn benchmark, X-Planner-generated masks and boxes beat human-labeled masks when paired with UltraEdit, lowering the reported L1 distance from 0.0575 to 0.0513.
  • The planner is model-agnostic: it can route each edit type to a different editing model (for example, a dedicated inpainting model for removal and a style model for style changes), so improvements compound in a bag-of-models configuration.
  • At inference, X-Planner requires no external object detector, segmentor, or closed-source LLM; everything needed for planning is inside the trained model.
  • Training data generated by an open-source MLLM (Pixtral-Large) yields comparable planner performance, suggesting the pipeline does not depend on GPT-4o specifically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pseudo-label bottleneck is real, a small amount of human-corrected masks or a learned mask-quality critic on COMPIE's training distribution should further improve downstream edits; this is a direct testable consequence of the paper's reliance on automated annotation.
  • Because the planner is a front end, gains should transfer to newer, stronger editing backbones without retraining: any model that accepts a mask or box conditioning can consume the plan.
  • The decomposition format suggests a general template for other instruction-following generation tasks: emit structured per-step control signals (region + operation) rather than a single free-form instruction, which may reduce ambiguity for the executor.
  • The box-prediction capability for unseen objects, trained only on MULAN-style paired data, may extend to other compositional generation tasks such as layout-conditioned text-to-image, where hallucinated placement is key.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 7 minor

Summary. The paper proposes X-Planner, an MLLM-based planner built on GLaMM that decomposes complex image-editing instructions into sub-instructions with edit types, segmentation masks, and, for insertion edits, bounding boxes. Training data (COMPIE, >260K pairs) is generated automatically: GPT-4o produces complex/simple instruction pairs (Level 1), Grounded SAM with edit-type-dependent refinement produces masks (Level 2), and an MLLM fine-tuned on MULAN produces pseudo-label boxes for insertions (Level 3). At inference, X-Planner is plugged into existing editing models such as UltraEdit or InstructPix2Pix*, which are applied iteratively with the predicted control signals. Evaluation covers MagicBrush single/multi-turn, a new COMPIE-Eval benchmark (550 images with GPT-4o-generated instructions and InternVL2-based metrics plus a user study), mask performance on PIE, box localization on MULAN, and ablations including open-source Pixtral-Large-generated training data and an MLLM-based verification loop. The central claim is that automatic decomposition with edit-type-specific masks and hallucinated insertion boxes improves existing editing models, matching or beating human masks on MagicBrush (e.g., L1 0.0513 with mask+box versus 0.0575 with human masks, Table 1) and improving MLLM alignment scores on COMPIE-Eval.

Significance. If the results hold, the paper is a useful contribution to instruction-based image editing: it reduces manual annotation burden, is editing-model-agnostic, and ships an automated data pipeline plus a new evaluation benchmark. Strengths include a large-scale dataset (260K pairs) with a described quality pipeline, an open-source variant using Pixtral-Large (Section 11-12 of the supplement), a well-defined evaluation protocol with a closed-source evaluator (InternVL2) deliberately chosen to avoid direct overlap with the GPT-4o-based data generator, and the MagicBrush comparison against human-mask control, which is a sensible point of reference. The main significance rests on whether the automatically generated pseudo-labels are reliable enough to support the claim of "precise edit types and segmentation masks," and whether the COMPIE-Eval benchmark can support the cross-model comparisons reported. The claims are plausible and the architecture is reasonable, but several evaluation gaps and at least one data-integrity issue in Table 2 need to be resolved before the results can be taken at face value.

major comments (6)
  1. [Table 2 (COMPIE benchmark)] The rows labeled "GenArtist [43] + UltraEdit" and "X-Planner + UltraEdit" with "X-Planner's Decomposed Instruction" (and likewise for InstructPix2Pix*) report identical numbers across L1, CLIPim, CLIPout, DINO, MLLMti, and MLLMim. Either the table contains duplicated rows (making the GenArtist comparison invalid as presented) or X-Planner's decomposed instructions coincide exactly with GenArtist's; neither reading supports Table 2's claim that X-Planner significantly improves over GenArtist with decomposition and masks. The authors should correct the table and re-run or remove the GenArtist comparison.
  2. [Section 3.2, Level 2; Table 3] The central claim of precise, edit-type-specific masks rests entirely on pseudo-labels: Grounded SAM masks generated from GPT-4o-generated anchors, with no human verification on the COMPIE training distribution. The PIE benchmark (Table 3) evaluates instruction-to-mask grounding on simple referring expressions, not on COMPIE-style anchors such as 'lion's fur' or 'area around the cat,' so it does not validate mask quality on the actual training distribution. The paper should report mask-quality statistics (e.g., IoU against a hand-labeled sample of COMPIE training images, or at least agreement rates with human-judged masks per edit type) before claiming that X-Planner generates 'precise' masks.
  3. [Section 3.2, Level 3; Table 4] Bounding-box quality is measured on MULAN, where the inserted object is absent from the input image and the ground-truth box is known. COMPIE insertion instructions include relational placements (e.g., 'around the cat', 'behind the lion') that are not necessarily represented by MULAN's single-object insertion boxes. The paper should either provide a human-evaluated sample of COMPIE insertion boxes (e.g., percentage of boxes judged plausible or matching an agreement threshold) or clearly state the distribution shift between MULAN and COMPIE and justify why the MULAN numbers transfer.
  4. [Section 4.2, COMPIE-Eval benchmark] COMPIE-Eval instructions are generated by GPT-4o, the same model family used to generate X-Planner's training data (Level 1, Section 3.2). This creates a circularity risk: X-Planner may be learning to mimic GPT-4o's decomposition style rather than learning a general planning capability. The authors mitigate this by using InternVL2 as the evaluator and by providing a Pixtral-Large-trained variant (supplement Sections 11-12), but the main-table results are all for the GPT-4o-trained model. Reporting COMPIE-Eval results for the Pixtral-trained X-Planner in the main text, or adding a separate human-verified subset of COMPIE-Eval with instructions written by humans, would substantially reduce this concern.
  5. [Section 4, Baselines and Tables 1, 2, 5] No error bars, confidence intervals, or significance tests are reported for any metric. Several headline improvements are small (e.g., MagicBrush L1 0.0513 vs. 0.0528, or L2 0.0168 vs. 0.0171 in Table 1; several COMPIE-Eval metric differences in Table 2 are around 0.01-0.02). Given that the pipeline involves stochastic sampling and an MLLM-based evaluator, the paper should report variance across seeds or test-set bootstrap intervals for at least the main comparisons, and show that the key differences are not within noise.
  6. [Section 4, InstructPix2Pix* baseline] InstructPix2Pix* is described as an 'improved version' of InstructPix2Pix trained on an internal dataset with mask conditioning (Section 4). Because this baseline is not public and its training data and architecture are not specified beyond this sentence, the generalizability claim made from Tables 2, 5, and the qualitative figures is difficult to verify. The authors should either release the InstructPix2Pix* model/checkpoint and describe its training data in detail, or restrict claims to the public UltraEdit baseline and clearly label InstructPix2Pix*-based numbers as on an internal model.
minor comments (7)
  1. [Section 1] The phrase "the ability to to perform these edits" in the first paragraph of the introduction contains a duplicated 'to'.
  2. [Section 3.2, Level 3] The text states that "Direct segmentation of the intended placement region (e.g., 'lion') often leads to imprecise masks," but the term 'placement region' is confused with the anchor object; it should read "the intended placement anchor (e.g., 'lion')" for clarity.
  3. [Section 4.1, Table 1] The table caption says "For Bag of Models, we utilize PowerPaint for removal tasks, InstructDiff for style changes, and UltraEdit for other edit types," but the main text in the Settings paragraph names 'InstructD' only in passing; consider standardizing the citation to 'InstructDiffusion' [13] for consistency.
  4. [Section 4.2, Table 2 caption] The caption says X-Planner provides "control guidance inputs (e.g., masks)" but for the rows labeled "X-Planner's Decomposed Instruction" the control input is only the decomposed text; the caption should distinguish these two control configurations.
  5. [Section 4.3 and Figure 8] Figure 8 shows repeated bounding-box predictions for the same instruction but no quantitative variance; the claim of "consistent bounding box with repeated runs" would be more convincing with a small variance table or a sentence reporting the standard deviation of the box coordinates.
  6. [Supplement, Section 8, Table 4] The caption of Table 4 says "Pseudo-labeling improves AP50 by over 2x at K=1," but the table shows AP50=0.34 for Mask Only and AP50=0.70 for +Pseudo-Label, which is 2.06x; the claim is accurate but the text would benefit from stating the exact numbers to avoid overstating the gain.
  7. [Supplement, Section 7] The training data ratio is given as [1, 3, 3, 3, 1, 3, 3, 9, 9, 9] but the components of this vector are not explicitly mapped to the listed datasets (Semantic Segm, RefCoco GCG, PSG GCG, Flickr GCG, GranDf GCG, InstructPix2Pix GCG, UltraEdit GCG, SEEDX GCG, MULAN GCG); a one-line mapping would remove ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the COMPIE-Eval GPT-4o instruction overlap is a benchmark-design caveat, not a construction-level circular reduction.

full rationale

X-Planner is a supervised MLLM planner, not a first-principles derivation. Its training labels come from GPT-4o decomposition, Grounded SAM masks, and MLLM-predicted boxes; these are training signals, and the paper does not claim to derive planning ability from anything other than supervised learning on these pseudo-labels. The central evaluation targets are external: MagicBrush and Emu Edit use human ground-truth edited images, PIE uses instruction-to-mask benchmarks, and MULAN uses ground-truth bounding boxes. For the new COMPIE-Eval benchmark, the paper explicitly avoids using GPT-4o as the evaluation model, instead using InternVL2, and also reports a user study. The only same-distribution element is that COMPIE-Eval complex instructions are generated by GPT-4o, the same model family used to create COMPIE training decompositions. This is a legitimate generalization and benchmark-design concern, but it is not a circular step: the reported improvements come from comparing editing models with and without X-Planner on the same instructions, and the mask/box quality claims are validated on independent benchmarks. No load-bearing self-citation, imported uniqueness theorem, ansatz-by-citation, or fitted-parameter-renamed-as-prediction chain was found.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests primarily on automatically generated synthetic data and the capability of pretrained editing models. No new physical entities are introduced. The hand-chosen hyperparameters (dilation, box enlargement, LoRA alpha, data ratios) are listed as free parameters, and the key domain assumptions concern the quality of pseudo-labels and the representativeness of GPT-4o-generated instructions.

free parameters (4)
  • Mask dilation factor = 20% for shape change and replace edits
    Chosen by hand in Level 2 Stage 2 to enlarge masks for shape changes and replacements; affects training masks and thus the planner's outputs.
  • Small box enlargement threshold = less than 5% of image size
    Boxes smaller than 5% of image size are enlarged during training to improve insertion localization; threshold is hand-chosen, described in supplement Section 8.
  • LoRA scaling factor = alpha=8
    LoRA fine-tuning hyperparameter for the language model in X-Planner, described in supplement Section 7.
  • Data source ratio = [1,3,3,3,1,3,3,9,9,9]
    Ratio of training datasets used to mix GLaMM and COMPIE data, described in supplement Section 7; hand-chosen.
assumptions (3)
  • domain assumption Diffusion-based editing models can execute localized edits accurately when given masks and bounding boxes as control inputs.
    X-Planner's value depends on the editing model pool using masks and boxes effectively; evaluated only with UltraEdit, InstructPix2Pix*, PowerPaint, InstructDiff, and GPT-4o.
  • domain assumption GPT-4o-generated complex instructions are representative of real user instructions.
    Used to generate both training data and the COMPIE benchmark; no independent corpus of human complex editing instructions is used.
  • domain assumption Grounded SAM and MULAN-derived pseudo-labels provide valid supervision for learning masks and boxes.
    Pseudo-labels are not human-verified; Table 4 validates boxes on MULAN but mask quality in the training distribution is not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editing." pith.science (2026). https://pith.science/paper/RPWRNG5J

@misc{pith2026250705259,
  author       = {Pith},
  title        = {Pith review of: Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RPWRNG5J}},
  note         = {Machine review of arXiv:2507.05259}
}
read the original abstract

Recent diffusion-based image editing methods have significantly advanced text-guided tasks but often struggle to interpret complex, indirect instructions. Moreover, current models frequently suffer from poor identity preservation, unintended edits, or rely heavily on manual masks. To address these challenges, we introduce X-Planner, a Multimodal Large Language Model (MLLM)-based planning system that effectively bridges user intent with editing model capabilities. X-Planner employs chain-of-thought reasoning to systematically decompose complex instructions into simpler, clear sub-instructions. For each sub-instruction, X-Planner automatically generates precise edit types and segmentation masks, eliminating manual intervention and ensuring localized, identity-preserving edits. Additionally, we propose a novel automated pipeline for generating large-scale data to train X-Planner which achieves state-of-the-art results on both existing benchmarks and our newly introduced complex editing benchmark.

Figures

Figures reproduced from arXiv: 2507.05259 by the authors.

Figure 1
Figure 1. Left. Given a source image and complex instruction, our MLLM based X-Planner decomposes the complex instruction into simpler sub-instructions (with edit type) along with auto-generated segmentation masks indicating the editing regions (shown in bottom left of each edited image) and hallucinates additional bounding box of object for the insertion case. We iteratively perform localized editing, by providing X-Planner’… view at source ↗
Figure 2
Figure 2. Overview of X-Planner for Complex Instruction-Based Editing. Our X-Planner comprises two main branches: First, MLLM decomposes the complex instruction into multiple simpler sub-instructions along with editing anchors (e.g., cat, dog, and background) which are given to segmentation decoder to get corresponding masks for each sub-instruction. Also, for the insertion edit, MLLM outputs bounding box coordinates along wi… view at source ↗
Figure 3
Figure 3. Level 1: Complex-Simple Instruction Pair Generation. Using our structured template, we prompt GPT-4o to generate com￾plex instructions—including indirect, multi-object, and multi-task instructions (as defined in Section 1)—along with their correspond￾ing simpler instructions, object anchors, and edit types. case, we also want to predict the location of the object to be inserted but we cannot use the segmentation dec… view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Level 3: Insertion Task-Based Mask & Box Local￾ization. For insertion task, Grounded SAM struggles to segment objects not present in the source image. We pre-train an MLLM on a bounding box-annotated dataset [41], enabling it to pseudo￾annotate our data with bounding b…
Figure 6
Figure 6. Figure 6: Qualitative Comparison for Complex Instruction-Based Editing Benchmark. Integrating X-Planner with editing methods, InstructPix2Pix* and UltraEdit, brings drastic boosts in preserving object identities with X-Planner generated masks and boxes (display in bottom-left of…
Figure 8
Figure 8. Figure 8: Visualize Consistent Bounding Box with Repeated Runs. We show X-Planner can generate consistent bounding boxes with repeated runs to yield plausible location variations. 4.3. GLaMM Comparison and BBox Localization In [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 7
Figure 7. Figure 7: User Study on COMPIE Benchmark. We compare against InstructPix2Pix* and UltraEdit. “Better” means the gener￾ated images by using our X-Planner is preferred and vice versa. User Study. We conducted a user study with 100 random samples from 550 images in the COMPIE bench…
Figure 9
Figure 9. Figure 9: X-Planner’s Training Dataset Summary (Generated from GPT-4o). This figure provides a distribution summary of our X-Planner’s training dataset, including (a) the data sources, (b) the distribution of edit types for decomposed instructions, and (c) the number of decompos…
Figure 10
Figure 10. Figure 10: COMPIE Benchmark Summary. This figure summarizes the proposed COMPIE benchmark, which consists of 550 samples spanning various types of complex instructions shown in (a), including (1) general complex instructions, (2) indirect instructions, (3) multi-object instructi…
Figure 11
Figure 11. Figure 11: X-Planner’s Training Dataset Summary (Generated from Pixtral-Large). This figure illustrates key statistics of our automatically constructed dataset used to train X-Planner using an open-sourced model, comprising around 300K instruction-image pairs. (a) Source Composi…
Figure 12
Figure 12. Figure 12: X-Planner’s Decomposition Outputs (Simplified Instructions, Edit Types, Masks, and Bounding Boxes). This figure illustrates examples of X-Planner’s decomposition results, showcasing its ability to handle diverse edit types. The segmentation masks generated are tailore…
Figure 13
Figure 13. Figure 13: Qualitative Comparison on MagicBrush, and on COMPIE Benchmark between X-Planner and UltraEdit (I). This figure presents: (1) a comparison of X-Planner with several baseline methods for multi-turn editing on the MagicBrush dataset and (2) a qualitative comparison with …
Figure 14
Figure 14. Figure 14: Qualitative Comparison between X-Planner and both InstructPix2Pix* (I) and UltraEdit Baseline (II). This figure compares results from both InstructPix2Pix* and UltraEdit Baseline with and without the integration of X-Planner. These examples highlight the advantages of…
Figure 15
Figure 15. Figure 15: Qualitative Comparison between X-Planner and InstructPix2Pix* Baseline (II). This figure compares results from InstructPix2Pix* with and without the integration of X-Planner. For the [replace] edit type in Rows 4, X-Planner generates a dilated segmentation mask for th…
Figure 16
Figure 16. Figure 16: Examples of Non-Rigid and Compositional Edits with Model-Specific Failure Cases. We show diverse complex edits (especially the [shape change] edits) generated by plugging X-Planner into three editing models: InstructPix2Pix*, GPT-4o [1], and IC￾Edit [49]. Despite not …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 37 canonical work pages

  1. [43]

    Genartist: Multimodal llm as an agent for unified image gen- eration and editing

    Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal llm as an agent for unified image gen- eration and editing. arXiv preprint arXiv:2407.05600, 2024. 2, 4, 8

    arXiv 2024

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2, 4, 5, 3, 11

    arXiv 2023

  3. [2]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Bap- tiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073,

    arXiv

  4. [3]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18208–18218, 2022. 2, 3, 6

    work page 2022

  5. [4]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 2, 3, 5, 6, 1

    work page 2023

  6. [5]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 22560–22570, 2023. 2, 3

    work page 2023

  7. [6]

    Training-free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343–5353, 2024. 2, 3

    work page 2024

  8. [7]

    Anydoor: Zero-shot object-level image customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6593–6602, 2024. 2, 3

    work page 2024

Show all 62 references
  1. [8]

    Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024. 6, 1, 3

    2024

  2. [9]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Jour- nal of Machine Learning Research , 24(240):1–113, 2023. 3

    2023

  3. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  4. [11]

    Guiding instruction-based image editing via multimodal large language models

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction-based image editing via multimodal large language models. arXiv preprint arXiv:2309.17102, 2023. 1, 2, 3, 6, 8

    2023 arXiv

  5. [12]

    Seed-x: Multi- modal models with unified multi-granularity comprehension and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multi- modal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024. 3, 5, 1

    2024 arXiv

  6. [13]

    Instructdiffusion: A generalist modeling in- terface for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al. Instructdiffusion: A generalist modeling in- terface for vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  7. [14]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 2, 3

    2022 arXiv

  8. [15]

    Style aligned image generation via shared at- tention

    Amir Hertz, Andrey V oynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared at- tention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4775–4785,

  9. [16]

    Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models. In Proceedings of the IEEE/CVF Conferen...

    2024

  10. [17]

    Pnp inversion: Boosting diffusion-based editing with 3 lines of code

    Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Pnp inversion: Boosting diffusion-based editing with 3 lines of code. In The Twelfth International Conference on Learning Representations, 2024. 8

    2024

  11. [18]

    Noise conditional flow model for learning the super-resolution

    Younggeun Kim and Donghee Son. Noise conditional flow model for learning the super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2021. 7, 1

    2021

  12. [19]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 2

    2023

  13. [20]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9579–9589, 2024. 2

    2024

  14. [21]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023. 2, 3

    2023

  15. [22]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3, 4

    2024

  16. [23]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Con- ference on Computer Vision, pages 38–55. Spring...

    2024

  17. [24]

    Repaint: Inpainting us- ing denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting us- ing denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 2

    2022

  18. [25]

    Sdedit: Guided image synthesis and editing with stochastic differential equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021. 3, 6

    2021 arXiv

  19. [26]

    Dragondiffusion: Enabling drag-style manipula- tion on diffusion models

    Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipula- tion on diffusion models. arXiv preprint arXiv:2307.02421,

    arXiv

  20. [27]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021. 2, 6

    2021 arXiv

  21. [28]

    Compositional text-to-image generation with dense blob representations

    Weili Nie, Sifei Liu, Morteza Mardani, Chao Liu, Benjamin Eckart, and Arash Vahdat. Compositional text-to-image generation with dense blob representations. arXiv preprint arXiv:2405.08246, 2024. 2, 3

    2024 arXiv

  22. [29]

    Zero-shot image-to-image translation

    Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceed- ings, pages 1–11, 2023. 3

    2023

  23. [30]

    Dreambench++: A human-aligned bench- mark for personalized image generation

    Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned bench- mark for personalized image generation. arXiv preprint arXiv:2406.16855, 2024. 1

    2024 arXiv

  24. [31]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2, 3

    2023 arXiv

  25. [32]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrah- man Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2024

  26. [33]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159,

    arXiv

  27. [34]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 3

    2022

  28. [35]

    Laion-5b: An open large-scale dataset for training next gen- eration image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next gen- eration image-text models. Advances in Neural Infor...

    2022

  29. [36]

    Empowering visual creativity: A vision- language assistant to image editing recommendations

    Tiancheng Shen, Jun Hao Liew, Long Mai, Lu Qi, Jiashi Feng, and Jiaya Jia. Empowering visual creativity: A vision- language assistant to image editing recommendations. arXiv preprint arXiv:2406.00121, 2024. 6

    2024 arXiv

  30. [37]

    Emu edit: Precise image editing via recognition and genera- tion tasks

    Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and genera- tion tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

  31. [38]

    Dragdiffusion: Harnessing diffusion models for interactive point-based image editing

    Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Han- shu Yan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2024

  32. [39]

    Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 3

    2024 arXiv

  33. [40]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  34. [41]

    Mulan: A multi layer annotated dataset for controllable text-to-image generation

    Petru-Daniel Tudosiu, Yongxin Yang, Shifeng Zhang, Fei Chen, Steven McDonagh, Gerasimos Lampouras, Ignacio Iacobacci, and Sarah Parisot. Mulan: A multi layer annotated dataset for controllable text-to-image generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vi...

    2024

  35. [42]

    Instancediffusion: Instance- level control for image generation

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Ro- hit Girdhar, and Ishan Misra. Instancediffusion: Instance- level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6232–6242, 2024. 2, 3

    2024

  36. [44]

    Turboedit: Instant text-based image editing

    Zongze Wu, Nicholas Kolkin, Jonathan Brandt, Richard Zhang, and Eli Shechtman. Turboedit: Instant text-based image editing. ECCV, 2024. 3

    2024

  37. [45]

    Omnigen: Unified image generation

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xin- grun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340, 2024. 3

    2024 arXiv

  38. [46]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  39. [47]

    Magicbrush: A manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 2, 3, 6

    2024

  40. [48]

    Hive: Harnessing human feedback for instructional visual editing

    Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, 10 Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  41. [49]

    In-context edit: Enabling instructional image editing with in- context generation in large scale diffusion transformer

    Zechuan Zhang, Ji Xie, Yu Lu, Zongxin Yang, and Yi Yang. In-context edit: Enabling instructional image editing with in- context generation in large scale diffusion transformer. arXiv preprint arXiv:2504.20690, 2025. 4, 11

    2025 arXiv

  42. [50]

    Ultraedit: Instruction-based fine-grained image edit- ing at scale

    Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained image edit- ing at scale. arXiv preprint arXiv:2407.05282, 2024. 2, 3, 5, 6, 1, 4

    2024 arXiv

  43. [51]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 6, 1

    2023

  44. [52]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 3

    2023 arXiv

  45. [53]

    A task is worth one word: Learning with task prompts for high-quality versatile image inpainting

    Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. arXiv preprint arXiv:2312.03594, 2023. 6 11 Overview In this supplementary, we first provide additional details a...

    2023 arXiv

  46. [54]

    COMPIE Dataset & Benchmark Summary X-Planner’s Training Dataset Statistics (COMPIE).We explore the details of our proposed COMPIE, a large-scale and high-quality dataset specifically designed to address complex instruction-based image editing. COMPIE con- tains over 260K compl...

  47. [55]

    Our X-Planner leverages GLaMM [32] as the base model, built on Vicuna-7B [51]

    Implementation Details X-Planner’s Setup. Our X-Planner leverages GLaMM [32] as the base model, built on Vicuna-7B [51]. The approach incorporates key components inspired by GLaMM [32], par- ticularly the design of the region encoder, grounding image encoder, and pixel decoder...

  48. [56]

    X-Planner’s Bounding Box Localization In Table 4, we present results on the MULAN validation benchmark [41], demonstrating the localization effective- ness of our X-Planner. Key observations include: (1) X- Planner’s bounding box localization significantly improves with pseudo...

  49. [57]

    Quantitative Comparison on Emu Edit We also compare the effectiveness of ourX-Planner on Emu Edit test set [37] which is similar to MagicBrush [47] test set and focuses on simpler instructions. We apply ourX-Planner with the UltraEdit [ 50] model and we can see in Table 5 that...

  50. [58]

    Multi-Step Editing Error Propagation Our X-Planner is less prone to errors since it decomposes complex instructions into simpler, model-friendly steps. As stated in the main paper, we can further enhance reliability by introducing a closed-loop verification mechanism using str...

  51. [59]

    Generate Training Data from Open- Sourced Model, Pixtral-Large To ensure reproducibility and accessibility of our pipeline, we also build a secondary version of the training dataset using an open-sourced MLLM, Pixtral-Large [ 2], a 124B- parameter multimodal model built upon M...

  52. [60]

    This allows us to isolate the effect of training signal quality from the underlying data generator

    Train X-Planner with Generated Data from Open-Sourced Model, Pixtral-Large To ensure a fair comparison, we maintain identical training settings, including model architecture, learning rate, batch size, and number of epochs, as described in Section 7 which were used for GPT-4o ...

  53. [61]

    Each ex- ample demonstrates how X-Planner tailors its outputs to 3 Figure 11

    Additional Qualitative Results In Figure 12, we provide a detailed look at X-Planner’s decomposition capabilities, emphasizing its ability to ef- fectively manage a wide range of edit types by generating precise and context-aware segmentation masks. Each ex- ample demonstrates...

  54. [62]

    Non-Rigid Edits on X-Planner In Figure 16, we present a diverse set of non-rigid and compositional edits using three different editing models: InstructPix2Pix*, GPT-4o [1], and IC-Edit [49]. These ex- amples demonstrate the plug-and-play flexibility of our X- Planner, which ca...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.