{"id":"b8148f48-3d40-495a-af9a-9c6e7e8f0916","arxiv_id":"2505.16425","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A pairwise-conditioned diffusion model generates instructional illustrations from procedural text and is finetuned with a text-image alignment reward.","lead":"The paper trains a text-to-image diffusion model to generate a sequence of illustrations from step-by-step instructions, using pairwise step conditioning and a reward model that measures text-image alignment. The authors report improved alignment over baseline models, but their evaluation uses the same reward model they trained with, so the reported gains are not independent evidence.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pairwise factorization is internally inconsistent with the adjacency mask, and the evaluation metric is the same signal used to train the model, so the reported gains do not establish the central claim.","rationale":"The reader's weakest assumption is well-identified: the evaluation metric is not independent of the training objective, creating a circularity that makes the quantitative claims unreliable. My stress-test agrees this is the primary load-bearing issue, and it is sufficient on its own to reject the paper as written. I add that there is a second, independent internal inconsistency in the method itself (pairwise product over all i<j versus adjacency mask that only allows adjacent interactions), which the reader mentioned only in passing. This matters because if only adjacent pairs are used, the model's formal description is wrong, and if all pairs are used, the mask is ineffective. The lack of error bars, significance tests, and sample sizes in Table 1 compounds the problem. The paper does contain some useful components—the constituency-parser-based text encoding and the idea of MLLM-based evaluation for instructional alignment are interesting—but the central claim is not supported as written. A REJECT verdict, with the possibility of resubmission after fixing the formalism and adding independent evaluation, is appropriate.","tokens_in":13376,"tokens_out":3304,"duration_ms":24295,"concrete_test":"Two checks: (1) Re-derive the exact training objective from the released code or from a clarification request, writing out which pairs (i,i+1) versus all i<j and how the Sec. 4.2 mask is applied; if the stated objective and the mask do not coincide, the architecture is not well-specified. (2) Run a human evaluation on a random sample of 100 generated instruction sets per method, asking 3+ annotators to rate step-text alignment and temporal coherence; compare human preferences against the LLM2Vec-based scores on the same samples. If human agreement with the automated metric is poor, or if the metric does not prefer Ours over SDXL/StackedDiffusion, Table 1's claims are not evidence of real alignment.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim rests on two pillars: the pairwise coherence model (Sec. 4.1-4.2) and the evaluation protocol (Sec. 5.2-5.3). The first is internally inconsistent: Sec. 4.1 defines a product over all i<j pairs, but Sec. 4.2's mask restricts attention to adjacent slices; the paper never says which is implemented. If all pairs are used, the mask is wrong; if only adjacent pairs are used, the claimed long-range coherence is not captured. The second is circular: Sec. 4.4 uses the BLIP2/LLaVA+LLM2Vec alignment score as the training reward, and Sec. 5.2/5.3 uses the exact same pipeline as the evaluation metric. Optimizing on a metric and then reporting that metric overstates gains, and the authors even note in Sec. 5.2 that the MLLM gives high scores when object names match but actions differ. Table 1 has no error bars, significance tests, or sample sizes, so the improvements could be noise. Both pillars must hold for the abstract's 'significantly outperforms' claim to be true; the text as written does not establish either.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes I2G, a text-conditioned diffusion framework for generating step-by-step instructional illustrations from procedural text. The method decomposes input into goal and step texts, encodes long steps with a constituency parser, introduces a pairwise factorization with cross-image attention for coherence, and fine-tunes an SDXL-based model using an image-text alignment reward. Experiments on HTStep, CaptainCook4D, and WikiAll report lower KL divergence and chi-square values than SD1.5, SD2.1, SDXL, and StackedDiffusion, alongside FID scores, leading to the claim that the method significantly outperforms existing baselines in procedural text-image alignment.","tokens_in":13615,"tokens_out":4513,"duration_ms":38464,"significance":"If the central claim were established, the paper would make a useful contribution to text-to-image generation for instructional content, a relatively underexplored task. The paper addresses a genuine problem, and the idea of using an LLM-based alignment score both as a training signal and as an evaluation tool is not unreasonable in itself. The authors also report FID as an independent image-quality check and candidly discuss failure cases in Appendix A.3. However, as written, the evidence is not convincing: the evaluation metric coincides with the training reward, the pairwise coherence model is specified inconsistently, and the quantitative results lack statistical support. The significance of the claimed improvements therefore cannot be assessed from the manuscript.","major_comments":[{"comment":"The evaluation metric used in Table 1 is the same signal used as the training reward. Section 5.2 explicitly states that the BLIP2/LLaVA captioning plus LLM2Vec text-similarity framework 'serves not only as an evaluator but also as a reward model to assist in fine-tuning, discussed further in Sec 4.4,' and the objective J(θ) in Sec 4.4 uses exactly this alignment score r(·). Reporting improvements on a metric that the model was optimized against does not constitute independent evidence of better text-image alignment. This concern is amplified by the authors' own observation at the end of Sec 5.2 that the MLLM assigns high scores when object names match even when the described actions differ, which suggests the metric can be gamed by superficial object co-occurrence. The central claim of the paper therefore rests on circular evidence as it stands.","section":"§5.2 and §4.4, Table 1"},{"comment":"The pairwise factorization and the adjacency mask are internally inconsistent. Section 4.1 defines p(v1,...,vn | g,S) ≈ ∏_{i<j} p(vi,vj | g,si,sj), a product over all pairs, while Section 4.2 describes an attention mask that 'restricts attention mostly to one's own latent slice and the adjacent slice.' The paper never states which formulation is actually implemented. If all pairs are used, the adjacency mask is not a faithful implementation of the model; if only adjacent pairs are used, the claimed long-range coherence across non-adjacent steps is not captured. Additionally, a product over all i<j pairs has O(n^2) terms, which is hard to reconcile with the claim that the factorization avoids a combinatorial explosion. This ambiguity bears directly on the main methodological contribution.","section":"§4.1–4.2"},{"comment":"The quantitative comparison in Table 1 lacks the statistical grounding needed to support 'significantly outperforms.' No error bars, confidence intervals, significance tests, or sample sizes are reported, and the differences between some entries are small (e.g., KL 12.71 vs. 12.48 for Ours vs. SDXL on HTStep with Humanllava). It is also not defined what distributions the KL divergence and chi-square statistic are computed over, which distributions are being compared, or how the 'GT Captioner' column and the 'Humanllava'/'Humanblip2' rows interact with the protocol. As written, the improvements in Table 1 could be noise or an artifact of the metric definition.","section":"§5.3, Table 1"},{"comment":"The paper claims that the MLLM+LLM2Vec evaluation 'aligns well with human judgment across three datasets,' but the human evaluation that this claim depends on is not reported in any detail. There is no description of the number of annotators, the number of samples judged, the scoring instructions beyond the score scale, or inter-annotator agreement. Without these details, the validity of the automated metric—and hence of the main evaluation—cannot be checked. This is load-bearing because the automated metric is the basis of Table 1.","section":"§5.2"},{"comment":"The training procedure of the proposed model is not reproducible from the manuscript. The paper does not specify the SDXL variant used, the fine-tuning dataset size, the number of gradient and sampling steps, the learning rate, the CFG scale, the conditioning dropout probability, or how the reward model's gradients are backpropagated through the sampling process. Given that the proposed method is a fine-tuning procedure, these details are necessary for verification and for comparison with existing reward-finetuning methods such as DDPO, DPOK, and AlignProp.","section":"§4.4–4.5"}],"minor_comments":[{"comment":"There is a typo in the notation: the sentence says 'g, S, and S represent the goal text, the sequence of step instructions, and the visual information,' but the third symbol should be V, not S.","section":"Equation (1)"},{"comment":"The objective J(θ) is written with parameter θ, but the text says only the diffusion parameters ϕ are updated; this notational inconsistency should be fixed.","section":"§4.4"},{"comment":"The table header 'GT Captioner' is unclear, and the meaning of the 'Humanllava' and 'Humanblip2' entries is not explained in the table or its caption; the caption should state that the values are multiplied by 100 and clarify which captioner is used for which row.","section":"Table 1"},{"comment":"The paragraph discussing 'Goal Faithfulness' and 'Step Faithfulness' uses CLIP-based similarity, but the actual evaluation in Table 1 uses KL and chi-square statistics; the relationship between these two evaluation schemes should be made explicit.","section":"§5.3"},{"comment":"The failure case in A.3, where the word 'steak' triggers an image of an angel despite a stove being in the prompt, undermines the general claim of accurate step-wise generation; this limitation is acknowledged but should be discussed in the main text as a caveat on the headline results.","section":"Appendix A.3, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from a clearer positioning against StackedDiffusion (Menon et al., 2023), which is closely related; the current text mentions tiling but does not explain the technical difference in formulation or results. The paper also states that code and network weights will be released, but no link or repository identifier is provided; this should be supplied in a revision. The writing shows signs of haste (equation typos, unclear table headers, undefined metrics), and a careful revision should also address the reproducibility gaps in the experimental section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey,\n\nShort version: I2G is about generating step-by-step instructional images from text. The core idea—decompose instructions into goals and steps, then generate a sequence of images with cross-step consistency—is a real problem, and the authors bring some useful tools: a constituency parser to chunk long step texts, and a pairwise conditioning mechanism that extends StackedDiffusion. The writing is honest, and they acknowledge several limitations.\n\nBut the central claim of 'significantly outperforms' doesn't hold up. The big one: the reward function used for RL fine-tuning (Sec 4.4) is exactly the same BLIP2/LLaVA + LLM2Vec alignment pipeline that produces the numbers in Table 1. The authors even say the evaluation framework 'serves... as a reward model.' You can't train on a metric and then report it as independent evidence. That is textbook reward hacking, and it's load-bearing.\n\nSecond, the model description is internally inconsistent. Sec 4.1 factorizes the joint over all i<j pairs, but Sec 4.2's mask restricts attention to adjacent slices. The paper never says which is actually implemented. If all pairs, the mask is wrong; if adjacent, you don't have the long-range coherence you claim. The stress-test note is accurate.\n\nThird, the empirical evidence is thin: no error bars, no significance tests, no sample sizes. And the KL/Chi-square metric is underspecified—lower KL is said to be better, but it's not clear what distributions are being compared. The authors themselves note their MLLM evaluator gives high scores when object names match even when actions differ, which undermines the whole evaluation.\n\nWhat's good: the task is underexplored, the constituency-parser chunking is a sensible fix for long instructions, and the authors did run human evaluations, which is more than many papers in this area. They also promise code and weights.\n\nThis is not a paper to accept as-is. The core evidence is circular and the model description needs repair. But it's not worthless—the problem is real and the pieces are reasonable. I'd lean to reject as written, but if the venue allows, a major revision with independent human evaluation and a cleaned-up model description could make this a decent contribution. I'd send it to reviewers who care about evaluation rigor.\n\nBest.","headline":"The paper proposes a sensible approach to instructional illustration generation, but the evaluation metric is circular and the pairwise model description is internally inconsistent, so the main claim is unsupported.","tokens_in":14132,"tokens_out":4814,"would_cite":false,"duration_ms":36720,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pairwise diffusion design makes generated instruction images match the text better than standard baselines.","keywords":["procedural text-to-image generation","instructional illustrations","diffusion models","pairwise coherence","text-image alignment","evaluation protocol","goal-step decomposition","preference optimization"],"falsifier":"A human preference study on held-out instructions from the same three datasets: show raters paired images from I2G and from each baseline for the same goal and step text, blind to method, and ask which image better matches the step. If human preference does not track the lower KL and Chi-square values, or if swapping BLIP2 for a different captioner reverses the ranking of methods, the central claim is falsified.","tokens_in":13200,"feed_emoji":"🖼️","tokens_out":5401,"duration_ms":43395,"temperature":0.7,"pith_summary":"This paper tries to establish that procedural text, instructions made of a goal and ordered steps, can be translated into a matching sequence of images by treating the task as a language-conditioned diffusion problem. The proposed I2G framework splits each instruction into goal and step texts, generates images with Stable Diffusion XL, and couples neighboring steps through a masked attention layer that lets each image borrow visual context from the adjacent step. To keep long step texts intact, a constituency parser breaks each step into clauses that are encoded separately. The authors argue that this combination produces images that align with the linguistic content and the sequential order of instructions better than four strong baselines, and they support the claim with a new evaluation protocol based on captioning generated images and comparing the captions to the original instructions.","feed_headline":"Pairwise diffusion beats baselines at illustrating how-to steps","feed_subtitle":"By pairing adjacent steps and scoring text-image alignment, the method lowers mismatch scores on three how-to datasets.","key_machinery":"The load-bearing mechanism is the pairwise factorization of the image-generation distribution, implemented as masked cross-step attention between latent pairs: two step images are encoded to latents $z_i,z_j$, stacked, and passed through a masked self-attention layer where a mask $\\hat{M}$ restricts attention mostly to the same slice and the adjacent slice, so step $j$ inherits object identity and setting from step $i$. Two supporting mechanisms carry the text side: a constituency parser that splits long step descriptions into clauses before encoding, with CLIP-ViT/L for goals and OpenCLIP-ViT/G for steps, and a preference-optimization loop that treats a caption-based text-image alignment score as a reward and backpropagates it through the diffusion sampling steps.","core_discovery":"On the paper's own terms, the central discovery is that a pairwise factorization of the joint distribution over instruction images, $p(v_1,\\dots,v_n \\mid g,S) \\approx \\prod_{i<j} p(v_i,v_j \\mid g,s_i,s_j)$, lets a diffusion model capture cross-step coherence without the cost of a fully joint model with a fixed step count. This factorization is realized by concatenating the latent representations of two images and applying a masked self-attention that restricts attention to one's own latent slice and its adjacent neighbor. On HTStep, CaptainCook4D, and WikiAll, the resulting model reports lower KL divergence and Chi-square statistics than SD1.5, SD2.1, SDXL, and StackedDiffusion, under an evaluation that first generates captions with BLIP2 or LLaVA and then measures text similarity with LLM2Vec against the original goal and step texts. The authors also show that CLIPScore and multimodal binding models track human judgments poorly on instructional content, motivating the new protocol.","pith_inferences":["Editorial inference: the same pairwise masked-attention factorization should transfer to video generation, where adjacent-frame coherence plays the role of adjacent-step coherence, making it a plausible inductive bias for text-conditioned video diffusion.","Editorial inference: a testable extension the paper does not run is swapping BLIP2 or LLaVA for a stronger vision-language captioner; if the reported ranking of methods reverses, part of the gain is an artifact of the specific captioner rather than of the generated images.","Editorial inference: because the paper observes that multimodal LLMs reward matching object names even when the described actions differ, a reward that also scores verb and action agreement would likely push generated images further toward correct procedural states."],"forward_implications":["On all three tested datasets, the method's KL divergence and Chi-square values are lower than SD1.5, SD2.1, SDXL, and StackedDiffusion, meaning fewer generated images whose captions mismatch the instruction text.","The pairwise factorization gives multi-step coherence without a fixed number of steps, unlike the StackedDiffusion baseline, which cannot generate more than six steps.","The caption-then-compare protocol, BLIP2 or LLaVA captions plus LLM2Vec text similarity, can serve both as an evaluator and as a reward signal for finetuning diffusion models on procedural text.","Because goals are contextual rather than visual, goal faithfulness is harder to improve than step faithfulness, and the paper's human evaluations show evaluators struggle to connect goal text to images.","The framework is not limited to cooking: the three datasets span many how-to tasks, so the same goal-and-step decomposition applies to education and assembly instructions."],"supporting_citations":[{"why":"Supplies the HT-Step dataset of instructional articles aligned with how-to videos, one of the three evaluation datasets.","marker":"Afouras et al., 2023"},{"why":"Supplies the CaptainCook4D dataset of cooking procedures in real kitchens, which the paper filters by removing error steps.","marker":"Peddi et al., 2023"},{"why":"Provides the VGSI WikiHow-derived source merged into the WikiAll evaluation dataset.","marker":"Yang et al., 2021b"},{"why":"Defines the Stable Diffusion 1.5 and 2.1 baselines that the method is compared against.","marker":"Rombach et al., 2022"},{"why":"Provides Stable Diffusion XL, the diffusion backbone on which the pairwise masked-attention model is built and also a baseline.","marker":"Podell et al., 2023"},{"why":"Provides StackedDiffusion, the joint-model baseline with a fixed step-count limitation that motivates the pairwise factorization.","marker":"Menon et al., 2023"},{"why":"Supplies LLM2Vec, the text encoder used both in the evaluation protocol and as part of the image-text alignment reward.","marker":"BehnamGhader et al., 2024"},{"why":"Provides CLIP, used for goal-text encoding and as the CLIPScore alignment baseline that the paper argues is inadequate for instructions.","marker":"Radford et al., 2021"},{"why":"Provides OpenCLIP-ViT/G, the text encoder chosen for long step descriptions.","marker":"Cherti et al., 2023"},{"why":"Defines CLIPScore, the standard text-image alignment metric whose failure on procedural text motivates the new evaluation protocol.","marker":"Hessel et al., 2021"}],"fun_headline_variants":["Pairwise diffusion makes how-to images stay coherent","Text-conditioned diffusion pairs steps for better instructions","How-to illustrations get a coherence boost from pairwise attention","Diffusion model links adjacent steps to draw clearer instructions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison stands on the assumption that the automated score, caption the generated image and measure text similarity between that caption and the original instruction, faithfully matches what human raters would call text-image alignment, and that KL and Chi-square differences on that score translate into visibly better instruction illustrations.","fun_headline_variants_meta":{"raw":{"variants":["Pairwise diffusion makes how-to images stay coherent","Text-conditioned diffusion pairs steps for better instructions","How-to illustrations get a coherence boost from pairwise attention","Diffusion model links adjacent steps to draw clearer instructions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1292,"prompt_tokens":943,"completion_tokens":349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":559,"tokens_out":349,"duration_ms":3539,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:00:21.646498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human preference study on held-out instructions from the same three datasets: show raters paired images from I2G and from each baseline for the same goal and step text, blind to method, and ask which image better matches the step. If human preference does not track the lower KL and Chi-square values, or if swapping BLIP2 for a different captioner reverses the ranking of methods, the central claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HT-Step dataset of instructional articles aligned with how-to videos, one of the three evaluation datasets."},{"cited_title":"Generating Illustrated Instructions","cited_arxiv_id":"2312.04552","evidence_quote":"Provides StackedDiffusion, the joint-model baseline with a fixed step-count limitation that motivates the pairwise factorization."}],"review_version":1}