Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation

Ranran Haoran Zhang; Rui Zhang; Xiaoxin Lu; Yusen Zhang

REVIEW 3 major objections 5 minor 15 references

Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An autoregressive edit-refine loop keeps generated text-image task plans consistent and coherent.

desk verdict The autoregressive loop is a real idea, but the main comparison is confounded by the fine-tuned image editor; the paper is worth a serious referee, not a desk reject. read the letter →

arxiv 2506.11380 v1 pith:DNGF466Q submitted 2025-06-13 cs.CV cs.AI

Xiaoxin Lu , Ranran Haoran Zhang , Yusen Zhang , Rui Zhang This is my paper

classification cs.CVcs.AI

keywords text-imageplangenerationmultimodaltaskplanningvisualcoherenceconsistencyautoregressivepseudo-PDDLInstructPix2Pixinstructionfollowing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that text-image task plans—paired sequences of instructions and pictures for daily procedures—fail because each step's image is generated independently, and because text and image can drift apart. It proposes a step-by-step autoregressive framework in which the next text draft is written from the goal and prior text, the next image is produced by editing the previous image according to that draft, structured visual information is extracted from the new image, and the text draft is then rewritten using that information. The claim is that this loop makes visual steps cohere (objects, containers, and scenes persist) and keeps each image aligned with its instruction. The paper reports gains over single-shot baselines and a prior multimodal planner across three backbone models, with a new 1,100-task benchmark and new evaluation metrics. If true, the core idea is that reciprocal text-image feedback at generation time, rather than stronger single-model generation, is what fixes consistency.

What carries the argument

The load-bearing mechanism is a four-stage autoregressive loop. Stage 1, the text drafter, prompts an LLM with the goal and prior text steps to propose the next step. Stage 2, the image editor, uses InstructPix2Pix—an image-editing diffusion model conditioned on both a reference image and a text instruction—to transform the previous visual step into the next one, which is what enforces visual coherence. Stage 3, the visual-information extractor, converts the generated image into a pseudo-PDDL tuple of objects, tools, actions, and goal; this structured format, rather than free-form captioning, is the key to suppressing noise before refinement. Stage 4, the text refiner, rewrites the draft using that structured visual information, which grounds the text in the actually rendered image and closes the consistency loop. The same backbone LLM is reused as drafter, extractor, and refiner, so the framework adds no new architecture, only staged prompting.

What would settle it

Compute the intersection of task titles or plan text between the 20,000 wikiHow fine-tuning tasks and the 1,000-task evaluation set; if a substantial fraction of evaluation tasks or near-duplicate plans appear in the fine-tuning data, the improved visual coherence on wikiHow could reflect memorization rather than the iterative loop. A second check: run the framework with Stage 4 (text refinement) disabled; if the text-image consistency scores do not drop, the cross-modal loop is not doing the claimed work.

Watch

Extended reading notes

Core claim

The central claim is that multimodal task planning should be treated as an iterative, autoregressive process rather than a one-shot generation problem. At each step the framework: (1) drafts the next instructional text from the goal and previously generated text; (2) edits the most recent visual step with an instruction-following image editor to obtain the next image; (3) converts the new image into a pseudo-PDDL structure listing objects, tools, actions, and the apparent goal; and (4) uses that structure to refine the textual draft. The generated text and image then become the conditioning history for the next step. The paper reports that this design improves text-image consistency (CLIP score and judge-based text-image alignment) and visual coherence (perplexity-based metric and judge-based image-to-image continuation) relative to vanilla LLM-plus-diffusion baselines, a prior multimodal planning framework, and several ablations, across Gemini-1.5-flash, GPT-4o, and Mistral-7B backbones. The framework is presented as a plug-and-play wrapper that improves whatever backbone it is applied to.

Load-bearing premise

The evaluation assumes that the 20,000 wikiHow tasks used to fine-tune the image editor are disjoint from the 1,000-task wikiHow evaluation benchmark, yet no intersection check or explicit split is reported, so the reported visual-coherence gains could partly come from the model having already seen the same plans.

Editorial extensions

If this is right

Text-image plans generated by this framework should show fewer object and scene jumps between consecutive images than plans from single-shot generators, because every new image is an edit of the previous one.
Because the text is refined from the image it helped create, each instruction should describe what is actually depicted, reducing contradictions such as text saying indirect sunlight while the image shows direct sunlight.
The approach is model-agnostic: any sufficiently strong LLM plus an instruction-following image editor should inherit the same consistency gains, so the improvement should transfer to future, stronger backbones.
Ablations imply that the structured pseudo-PDDL representation is doing real work: replacing it with free-form descriptions or with raw-image feeding degrades both visual and textual quality.
The framework is most effective on medium-complexity tasks with concrete actions; very abstract or highly dexterous tasks remain hard for the image editor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same loop could be applied beyond static plans—for example, to video storyboards or embodied instruction generation—wherever consecutive frames must stay visually grounded in the previous state.
The pseudo-PDDL extraction step effectively turns the generated image into a cheap verifier of the text draft: if the image encoder cannot find the objects the text mentions, the text gets rewritten, suggesting a general recipe for cross-modal self-correction that does not need a separate reward model.
A testable extension would be to feed the refined text back into the image editor a second time and measure whether the text-image alignment improves monotonically with loop depth, or whether errors start to accumulate.
Because the method inherits the image editor's failure modes, one can predict that plans with frequent large workspace changes or object insertions will show residual incoherence even under the framework.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the iterative text-image loop is a genuine contribution, and the new benchmark is useful. But the headline numbers do not yet isolate the loop from the effect of fine-tuning the image editor. That is the main thing to know. What is new: they extend TIP from single-shot generation to an autoregressive edit loop, using InstructPix2Pix to edit the previous visual step, then extracting structured pseudo-PDDL info from the new image and feeding it back to refine the text. That is a sensible architecture for improving cross-modal consistency, and it is tested on three backbones with a reasonable set of metrics. The 1,100-task benchmark across 11 categories is a real addition, and the paper is honest about limitations, including the indirectness of the PPL metric and co-author human evaluation. The soft spot is the experimental control. Section 5.3 says they fine-tune InstructPix2Pix on 20,000 wikiHow tasks; the evaluation set is 1,000 wikiHow plans. They never state that the fine-tuning set and evaluation set are disjoint. If they overlap, the visual coherence gains could reflect memorized reference transitions. More importantly, all baselines use an off-the-shelf image generator. A fine-tuned editor is exactly the kind of domain adaptation that would improve wikiHow-like visual coherence, independent of the loop. The ablations in Table 3 hold the fine-tuned generator constant, so they do not isolate the loop either. This is load-bearing: without a matched image editor, the main quantitative evidence for the framework is not clean. Minor issues: no significance tests, and the human evaluation is done by co-authors (disclosed, but not neutral). The PPL metric is acknowledged as indirect. These are secondary once the control issue is addressed. My take: the framework is plausible, the benchmark is worth having, and the paper is clearly written. As submitted, the central quantitative claim is not yet supported. I would send it to peer review and let reviewers ask for (1) an explicit disjoint split or intersection check between fine-tuning and evaluation data, and (2) at least one baseline with the same fine-tuned editor, ideally an ablation that uses the fine-tuned editor without the loop. With those, the paper could make a solid contribution.

Referee Report

3 major / 5 minor

Summary. The paper addresses text-image plan generation, proposing an autoregressive framework that interleaves four stages at each step: drafting a textual step from the goal and history, editing the previous visual step with InstructPix2Pix, extracting pseudo-PDDL visual information from the generated image, and refining the draft with that information. The framework is evaluated on a new dataset of 100 Instructables and 1,000 wikiHow tasks, using automatic metrics (ROUGE, BERTScore, CLIP, PPL), an LLM judge (Claude-3.5-Sonnet), and a human evaluation, across three backbone LLMs (Mistral-7B, Gemini-1.5-Flash, GPT-4o) and against vanilla LLM+Stable Diffusion, Stable Diffusion, and TIP baselines. The reported results show consistent gains for the proposed method on text-image alignment and visual coherence, alongside modest textual-plan improvements. The central claims are plausible but currently undercut by a fine-tuning/evaluation overlap issue and by the absence of a matched image-editor baseline.

Significance. If the reported effects are real, the paper would make a useful contribution: the task formulation is underexplored; the new benchmark (1,100 tasks, 11 categories, text-image pairs) is a concrete asset; evaluating across three backbones is a good generalizability check; and the four-stage loop is simple enough to be adapted. The paper also ships code and data and proposes metrics targeted at multimodal planning, with human agreement reported. The main quantitative evidence, however, is not yet identified: because the image generator is fine-tuned on wikiHow-derived triplets and the evaluation samples wikiHow tasks without a stated disjoint split, the visual coherence and text-image consistency gains could be driven by distributional fitting of the editor rather than by the proposed iterative framework. The PPL metric is also acknowledged by the authors to favor the trivial, repetitive SD baseline, undermining one of the quantitative coherence claims. These issues are load-bearing for the central claim and need to be addressed before the paper can be accepted.

major comments (3)

[§3, §5.3, Table 1] The 20,000-task InstructPix2Pix fine-tuning set (Section 5.3) and the 1,000-task wikiHow evaluation set (Section 3) are both sampled from wikiHow plans within the same selected categories, but the paper does not report any task-level or image-level overlap check. If evaluation tasks or their consecutive images appear in the fine-tuning triplets, the I-I, CLIP, and PPL gains could reflect memorized visual transitions rather than the proposed autoregressive loop. Please report whether any evaluation item occurs in the fine-tuning set, and if so, re-run the evaluation on a disjoint split.
[§5.1, §5.3, Table 1] No baseline uses an equally fine-tuned image editor. Ours fine-tunes InstructPix2Pix on wikiHow triplets, while the vanilla and SD baselines use off-the-shelf Stable Diffusion, and the TIP comparison does not appear to use the fine-tuned editor. The reported gains in CLIP, I-I, and T-I could therefore be caused entirely by the image-editor adaptation rather than by the four-stage loop. The ablations in Table 3 hold the fine-tuned editor fixed and vary only the visual-information format, so they do not control for this confound. Please add an OURS variant with the off-the-shelf InstructPix2Pix and/or a baseline equipped with the same fine-tuned editor.
[§5.2, §6.2, Table 1] The PPL visual-coherence metric is not currently interpretable as a coherence measurement. The authors note in §6.2 that SD obtains the best PPL because it is 'incapable of actual task planning' and outputs topic-fixed, visually repetitive images, which trivially lower the perplexity of consecutive image descriptions. In every row of Table 1, SD beats OURS on PPL, so the sentence in §6.2 claiming 'consistent improvement in terms of PPL score' is contradicted by the table. The metric should be redefined, for example by conditioning consecutive-description coherence on the actual step transition and penalizing repetition, or the PPL-based coherence claims should be removed.

minor comments (5)

[Figure 3 caption] The caption contains the typo 'outputt k andi k' and the fragment 'visual info -#visual step'; these should be corrected.
[§5.2] The phrase 'verify the LLM evaluation’s reliance' should read 'verify the LLM evaluation’s reliability'.
[§6.5] The ablation names WDES, WIMG, and PPDDL-TO-NL are not expanded at first use; please define them explicitly in the text or in a footnote.
[§5.3] The fine-tuning section reports training/validation/test splits only by loss; reporting a small set of image-edit quality numbers (e.g., on the held-out validation split) would help readers gauge whether the adapted editor is actually doing the intended work.
[Limitations] The Limitations section mentions general LLM data leakage but does not discuss the potential overlap between the InstructPix2Pix fine-tuning corpus and the wikiHow evaluation benchmark; this is a distinct leakage path and should be acknowledged.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the iterative text-image generation and refinement loop is implemented from independently defined stages, and the reported gains are empirical comparisons rather than identity reductions or fitted predictions.

full rationale

The paper's contribution is an engineering pipeline, not a mathematical derivation. Stage 1 drafts text from the goal and previous textual steps (Eq. 1); Stage 2 edits the previous visual step using InstructPix2Pix conditioned on the draft and the prior image (Eq. 2); Stage 3 extracts pseudo-PDDL visual information from the generated image (Eq. 3); Stage 4 refines the draft using that extracted information (Eq. 4). Each stage is a distinct model invocation, and the final textual step is the output of a generation model given the draft and extracted visual information, not an identity, a restatement of the input, or a refitting of the evaluation metric. The pseudo-PDDL format is hand-designed and then tested through ablations against natural-language descriptions, raw images, and a reversed format, so its role is empirical rather than definitional. The InstructPix2Pix fine-tuning on wikiHow-derived triplets is a standard domain-adaptation step; the absence of an explicitly reported disjointness check between the 20,000 fine-tuning tasks and the 1,000 evaluation plans is a data-leakage and experimental-control concern, not a demonstrated circular equivalence. No load-bearing claim rests on a self-citation chain: the closest prior framework, TIP, is used as a baseline rather than as a justification, and the PDDL reference is standard background. The limitations section also honestly notes the indirectness of text-based visual coherence metrics. Overall, no step reduces by construction to its own input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a fine-tuned image editing model, a hand-designed pseudo-PDDL representation, and several unverified evaluation assumptions. The most serious dependency is the assumed disjointness between fine-tuning and evaluation data; without it, the reported improvements could be fitted artifacts. No new physical entities or conserved quantities are introduced.

free parameters (3)

Fine-tuned InstructPix2Pix weights = trained on 20,000 wikiHow-derived triplets
The visual coherence results depend on this fitted component, and no control baseline uses an untuned version of the same image editor, so the gains cannot be attributed solely to the iterative loop.
pseudo-PDDL field set = 4 fields: objects, tools, actions, goal
This hand-designed representation is introduced in Section 4.3 and validated only within the same benchmark, with no external justification for why these four fields are the right abstraction.
InstructPix2Pix fine-tuning epochs = 50
The training ran for up to 50 epochs to reach training loss 0.100 and validation loss 0.105; no sensitivity analysis is provided for this hyperparameter.

assumptions (5)

domain assumption Fine-tuning and evaluation tasks are disjoint.
Required for the reported wikiHow gains to be meaningful; the paper does not state an intersection check between the 20,000 fine-tuning triplets and the 1,000 test plans.
domain assumption Consecutive wikiHow and Instructables images form valid training triplets {previous image, step text, next image}.
Section 5.3 uses this assumption to fine-tune InstructPix2Pix; the model's ability to produce coherent transitions depends on these triplets capturing real procedural state changes.
domain assumption Perplexity over converted textual descriptions is a valid proxy for visual coherence.
Section 5.2 defines this metric, and the Limitations section concedes it is indirect and may miss nuanced visual relationships.
domain assumption LLM judge scores reflect human preferences.
Appendix C uses Claude-3.5-Sonnet for evaluation; reliability is only partially checked, and the human verification is conducted by co-authors.
ad hoc to paper A pseudo-PDDL representation with objects, tools, actions, and goal is the right structured visual summary.
The format is designed by the authors in Section 4.3; ablations support it internally, but there is no external or theoretical justification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation." pith.science (2026). https://pith.science/paper/DNGF466Q

@misc{pith2026250611380,
  author       = {Pith},
  title        = {Pith review of: Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DNGF466Q}},
  note         = {Machine review of arXiv:2506.11380}
}

read the original abstract

People get informed of a daily task plan through diverse media involving both texts and images. However, most prior research only focuses on LLM's capability of textual plan generation. The potential of large-scale models in providing text-image plans remains understudied. Generating high-quality text-image plans faces two main challenges: ensuring consistent alignment between two modalities and keeping coherence among visual steps. To address these challenges, we propose a novel framework that generates and refines text-image plans step-by-step. At each iteration, our framework (1) drafts the next textual step based on the prediction history; (2) edits the last visual step to obtain the next one; (3) extracts PDDL-like visual information; and (4) refines the draft with the extracted visual information. The textual and visual step produced in stage (4) and (2) will then serve as inputs for the next iteration. Our approach offers a plug-and-play improvement to various backbone models, such as Mistral-7B, Gemini-1.5, and GPT-4o. To evaluate the effectiveness of our approach, we collect a new benchmark consisting of 1,100 tasks and their text-image pair solutions covering 11 daily topics. We also design and validate a new set of metrics to evaluate the multimodal consistency and coherence in text-image plans. Extensive experiment results show the effectiveness of our approach on a range of backbone models against competitive baselines. Our code and data are available at https://github.com/psunlpgroup/MPlanner.

Figures

Figures reproduced from arXiv: 2506.11380 by the authors.

**Figure 2.** The distribution of our dataset. Wang, 2018; Valmeekam et al., 2023a), task domain diversity (Yagcioglu et al., 2018; Valmeekam et al., 2023b), or are not intended for plan generation (Yang et al., 2021). Therefore, to benchmark text-image plan generation, we collect a dataset of daily task plans covering various topics. Inspired by previous work, we consider two popular websites affording procedural daily task i… view at source ↗

**Figure 3.** Overview of our autoregressive framework at time step [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: A qualitative comparison of plans output by all three baselines and our approach. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Visual step coherence and text-image align [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Two example plans sampled from Instructables data (left) and wikiHow data (right) respectively. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Expert plan from wikiHow vs. plan generated by [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: Expert plan from Instructables vs. plan generated by [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: Expert plan from wikiHow vs. plan generated by [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

**Figure 10.** Figure 10: User interface of our designed annotation tool. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

[1]

Objects: List salient objects in the image

work page
[2]

Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang

Llm+p: Empowering large language mod- els with optimal planning proficiency.Preprint, arXiv:2304.11477. Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. 2024. Multimodal procedural planning via dual text-image prompting. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 10931–10954, Mi- ami, Flo...

arXiv 2024
[3]

Actions: Describe the specific actions being performed

work page
[4]

make an origami pinwheel

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.arXiv preprint arXiv:2403.05530. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023a. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change.Advances in Neural Infor-...

arXiv 2022
[6]

Tools: Identify any tools, equipment, or instruments being used

work page
[8]

Let’s improve a step toGwith visual information

Goal: Based on the visible actions and context, what appears to be the intended goal? Please format your response as: OBJECTS: [object list] TOOLS: [tool list] ACTIONS: [action list] GOAL: [state the apparent goal] Text Refinement You are a helpful planning assistant. Let’s improve a step toGwith visual information. Original Step:d k Visual Information Ex...

work page
[9]

Be more specific about the objects and tools involved

work page
[10]

Provide clearer action descriptions

work page

Show all 15 references

[11]

Tie” for “Textual Quality

Maintain alignment with the overall goal Please format your response as: [improved step descriptions] Table 5: Prompt templates we use in experiments with backbone model Gemini-1.5-flash. for generating images at each planning step. The InstructPix2Pix fine-tuning represents a...

2022
[12]

Correctness: Does the plan contain all necessary steps that align with the reference? This involves checking if the steps are complete
[13]

Executability: How practical and actionable are the steps? This involves checking if they can be implemented in a real-world setting
[14]

Coherence: Are all steps logically connected to each other? This involves checking if there are temporal conflicts or redundancy
[15]

Informativeness: Does the plan provide sufficient detail? This involves checking if it provides enough information to understand the plan. Grading scale: 1-Poor 2-Fair 3-Good 4-Very Good 5-Excellent Reference Plan: [reference planR] Plan to Evaluate: [evaluated planP] Please p...
[2023]

Instructpix2pix: Learning to follow image editing instructions.Preprint, arXiv:2211.09800. M. Fox and D. Long. 2003. Pddl2.1: An extension to pddl for expressing temporal planning domains.Jour- nal of Artificial Intelligence Research, 20:61–124. Chen Gao, Xiaochong Lan, Nian L...

2003 arXiv
[2024]

InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20256–20264

Generalized planning in pddl domains with pretrained large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20256–20264. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner:...

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Objects: List salient objects in the image

work page

[2] [2]

Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang

Llm+p: Empowering large language mod- els with optimal planning proficiency.Preprint, arXiv:2304.11477. Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. 2024. Multimodal procedural planning via dual text-image prompting. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 10931–10954, Mi- ami, Flo...

arXiv 2024

[3] [3]

Actions: Describe the specific actions being performed

work page

[4] [4]

make an origami pinwheel

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.arXiv preprint arXiv:2403.05530. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023a. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change.Advances in Neural Infor-...

arXiv 2022

[5] [6]

Tools: Identify any tools, equipment, or instruments being used

work page

[6] [8]

Let’s improve a step toGwith visual information

Goal: Based on the visible actions and context, what appears to be the intended goal? Please format your response as: OBJECTS: [object list] TOOLS: [tool list] ACTIONS: [action list] GOAL: [state the apparent goal] Text Refinement You are a helpful planning assistant. Let’s improve a step toGwith visual information. Original Step:d k Visual Information Ex...

work page

[7] [9]

Be more specific about the objects and tools involved

work page

[8] [10]

Provide clearer action descriptions

work page