{"id":"a76f346e-d812-4aaa-9097-aceec06564b1","arxiv_id":"2505.02236","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning on synthetic images of empty or absent objects improves text-to-image models' ability to depict such states, with gains of 8+ points on GenAI-Bench and 24+ points on a new 200-prompt benchmark.","lead":"This paper describes a fully automatic pipeline that creates synthetic images of objects in empty or absent states, then fine-tunes text-to-image models on this data so they better follow prompts like 'a table without a bottle'. The authors report average correctness gains of 8 to 25 percentage points across five open-source models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains rely on the same GPT-4o-mini judge that filtered training data; independent VQA scores show much smaller gains, so the claimed effect size may be inflated.","rationale":"The paper is methodologically careful in many places: it reports VQA scores from an independent model, a user study on visual quality, FID and CLIP checks, generalization to unseen objects, and ablations of recaptioning and data source. These give real support to the qualitative claim that fine-tuning on filtered synthetic data helps. The single most load-bearing weakness is not the method but the headline metric. Because GPT-4o-mini both selects the training data and scores the outputs, the reported 8.2 and 24.6 point gains are exactly the quantity most exposed to a train/eval loop with the same judge. The VQA numbers, while independent, are reported without uncertainty and are much smaller, which is consistent with an inflated GPT-based effect. A clean human or independent-VLM correctness evaluation on the released prompts and code would determine whether the effect size is real or judge-specific. Until that test is run, conditional acceptance is the right verdict.","tokens_in":15201,"tokens_out":4991,"duration_ms":59128,"concrete_test":"Recruit three annotators to judge a random 100-prompt subset of Object State Bench: for each prompt, show baseline and fine-tuned SDXL images generated with identical seeds, side by side and in randomized order, with model identity hidden, and ask: 'Which image, if either, correctly shows the empty/absent state described by the prompt?' Use majority vote. Compare the fraction of prompts where the fine-tuned image is judged correct with the GPT-4o-mini improvement on the same subset. A large discrepancy (e.g., human-measured improvement below 10 points versus 23 points GPT-measured) would confirm that judge circularity inflates the headline; a match within 5 points would refute it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 (Sec. 4.3) reports average GPT-score improvements of 8.2 points on GenAI-Object-State and 24.6 points on Object State Bench. These scores come from GPT-4o-mini answering the binary question in Fig. 11, which asks whether the main object appears empty or unoccupied and whether the absent object is not present. The same model, with nearly the same prompt (Fig. 4), was used to filter the synthetic training images and to recaption the prompts (Fig. 10). This creates a closed loop: fine-tuned models are optimized to satisfy GPT-4o-mini's specific notion of emptiness/absence, and the same system then measures how well they satisfy it. The independent FLAN-T5 VQA metric, which was not part of the filtering loop, shows the same direction but appreciably smaller gains (Table 1: +5.2 vs +8.2 on GenAI-Object-State; +17.2 vs +24.6 on Object State Bench). This gap is exactly what one would expect if part of the GPT-measured gain reflects judge alignment rather than a general improvement in physical state rendering. No confidence intervals, significance tests, or inter-annotator statistics are provided, so at the reported sample sizes (214 and 200 prompts) the claimed effect sizes are not statistically anchored.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an automatic pipeline for generating synthetic training data that depicts objects in empty or absent states (e.g., 'an empty bookshelf', 'a table without any bottle'). The pipeline uses GPT-4o-mini to generate and rephrase prompts, and Stable Diffusion 1.5 to render images, with GPT-4o-mini as a filter to keep only images that correctly show the target state. The resulting 7,600 synthetic image-text pairs are used to fine-tune five open-source text-to-image models (SD1.5, SD2.1, SDXL, Flux Dev, OmniGen) via LoRA. The authors evaluate on a manually filtered subset of GenAI-Bench negation prompts (214 prompts, called GenAI-Object-State) and on a new 200-prompt benchmark (Object State Bench). They report average improvements of 8.2 percentage points (GPT-4o-mini score) and 5.2 points (FLAN-T5 VQA score) on GenAI-Object-State, and 24.6 and 17.2 points on Object State Bench. Additional experiments cover generalization to unseen objects, performance on full-state prompts, the effect of the data generator, comparison with real-world datasets (COCO, VidOSC), recaptioning, tuning steps, visual quality (FID, CLIP, user study), and CommonsenseT2I.","tokens_in":15482,"tokens_out":5874,"duration_ms":56307,"significance":"If the reported gains are robust, the paper would provide a simple, data-centric recipe for improving object-state understanding in text-to-image models, with potentially broad applicability. The strengths are the reproducibility efforts (released code and prompts), the breadth of models tested, and the inclusion of an independent VQA metric that partially corroborates the results. The cross-model consistency (five models show improvement on both benchmarks) and the generalization to unseen objects are encouraging. However, the significance is tempered by the circular use of GPT-4o-mini in data filtering and evaluation, the lack of statistical validation, and the manual curation of the evaluation set, which jointly leave the magnitude of the true effect uncertain.","major_comments":[{"comment":"The evaluation and training-data filtering use the same GPT-4o-mini model with near-identical prompts (Fig. 4 vs Fig. 11). This creates a closed loop: the model is fine-tuned to satisfy GPT-4o-mini's notion of emptiness/absence, and the same system then measures that notion. The independent FLAN-T5 VQA score (Table 1) shows smaller gains (e.g., +5.2 vs +8.2 on GenAI-Object-State), consistent with a partial judge-alignment effect. I request an evaluation on a held-out human-annotated subset or with a different VLM (e.g., GPT-4o or Claude) to quantify the effect, and a discussion of the discrepancy.","section":"Sec. 4.1, Fig. 4 and Fig. 11, Table 1"},{"comment":"No error bars, confidence intervals, or significance tests are provided for any of the reported accuracy numbers. With only 214 and 200 prompts respectively, a difference of 5 percentage points (e.g., SD1.5 GPT: 16% vs 21%) may not be statistically robust. The word 'significant' is used loosely throughout (e.g., abstract, Sec. 4.3). I request bootstrap confidence intervals or a paired significance test for the main comparisons, and a clarification of which differences actually reach significance.","section":"Tables 1-7"},{"comment":"The manual filtering of GenAI-Bench's 347 negation prompts to 214 (Sec. 4.1) is subjective and undocumented. The criteria for 'more aligned to our task' are not stated, nor are examples of excluded prompts or inter-annotator agreement. A manually curated benchmark with a subjective filtering step can bias the seemingly improvement if the retained prompts disproportionately match the training distribution. I ask that the full list of 214 prompts and the exclusion list with reasons be released, or that an objective filtering procedure (e.g., keyword-based or using an independent model) be used and documented.","section":"Sec. 4.1, GenAI-Object-State subset description"},{"comment":"Section 4.8 claims that fine-tuning does not deteriorate performance on unrelated prompts based on a random sample of 50 GenAI-Bench prompts. However, Table 7 shows consistent degradation on the CommonsenseT2I benchmark for four of five models (e.g., SD2.1 GPT: 41% to 36%; Flux: 45% to 40%). These results contradict the claim. The authors should either reconcile these findings, provide a more nuanced statement about the trade-off, or limit the claim in Sec. 4.8 to the specific sample used.","section":"Sec. 4.8 and Table 7"}],"minor_comments":[{"comment":"The abstract states 'across four models' but Table 1 reports five models; the Introduction also says 'four open-source models' (line 2). Please correct the inconsistency.","section":"Abstract"},{"comment":"The sentence 'We again use GPT 4o-mini [10] to filter out images which incorrectly capture object states as mentioned in 3 and to rephrase the template-like prompts 10' contains unclear references; presumably it should refer to Sections 3 and Fig. 10. Please fix the wording.","section":"Sec. 4.1"},{"comment":"The caption says 'passive voive prompts' and the described transformation is not passive voice in the linguistic sense; the example 'An empty table' → 'A table without any bottles on it' is not passive. Please correct the terminology and the typo.","section":"Fig. 10 caption"},{"comment":"Table 9 lists resolution 512 for Flux.1 DEV and OmniGen, while Sec. 4.2 states these models generate 768×768. Please clarify which resolution was actually used for fine-tuning and inference.","section":"Table 9 vs Sec. 4.2"},{"comment":"The SDXL reference [17] is actually SDEdit; the SDXL paper should be cited as [21] only. Also, [24] is used for both Stable Diffusion 1.5 and 2.1, which is acceptable as the original paper, but the naming in the table rows should be consistent.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's heavy reliance on GPT-4o-mini in both data generation and evaluation is a concern for independent verification; the manuscript would benefit from a more thorough disclosure of the closed-source dependencies and versioning. The manual filtering of the GenAI-Bench subset also deserves scrutiny. Given the severity of the circularity and significance issues, I recommend major revision rather than acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main take: this is a solid data-centric paper with a genuine, reproducible contribution, but the headline effect sizes are probably inflated because the same GPT-4o-mini system that filtered the synthetic training data also scores the outputs. The independent FLAN-T5 VQA metric shows the same direction with noticeably smaller gains, which is exactly the pattern you'd expect from judge alignment.\n\nWhat's new: a fully automatic pipeline that generates template prompts for objects in empty/absent states, generates synthetic images with SD1.5, filters with GPT-4o-mini, recaptions for diversity, and then LoRA-finetunes several open models. The cross-model generalization to 100 unseen objects (GPT +7, VQA +6) is the strongest evidence that the model learns a concept rather than memorizing training objects. The ablations are thoughtful: they test different data sources (COCO, VidOSC), recaptioning, number of steps, and show no degradation on non-empty states or CommonsenseT2I. They release prompts and code, which is real evidence.\n\nSoft spots, in proportion: the main evaluation uses GPT-4o-mini with nearly the same prompt used for filtering. That's a closed loop. The VQA metric, which wasn't part of the filtering, gives smaller gains (5.2 vs 8.2 on GenAI-Object-State; 17.2 vs 24.6 on Object State Bench). I don't think the gains vanish, but the true effect is closer to the VQA numbers. There are no confidence intervals or significance tests, and the benchmark prompts are partly machine-generated with the same pipeline, so there's a risk of overfitting to the pipeline's distribution. Also, the absolute scores remain low even after fine-tuning (best GPT score 23% on GenAI-Object-State), so this is an improvement on a hard task, not a solution. Minor: the abstract says four models but the table shows five.\n\nThe citation pattern looks fine; related work is covered, and the synthetic-data comparison against COCO/VidOSC is a good check. I'd send this to peer review: the method is useful, the experiments are mostly thorough, and the central claim of improvement is credible. The main fixes are to report uncertainty, add a human-judged subset, and ideally evaluate with a judge that wasn't used to filter the data.","headline":"Solid data-centric fix for object-state failures; headline gains are likely inflated by same-model judge, but the independent VQA metric and generalization results make it worth a serious look.","tokens_in":15949,"tokens_out":2275,"would_cite":true,"duration_ms":24223,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Synthetic empty-scene data fixes image AI's 'full state' bias","keywords":["text-to-image generation","object states","empty and absent states","negation prompting","synthetic data pipeline","LoRA fine-tuning","vision-language evaluation","Object State Bench"],"falsifier":"Score the fine-tuned and baseline models on a human-annotated subset of Object State Bench, or with a vision-language judge that played no role in generating or filtering the training data; if the 24.6-point average lead shrinks to near zero or reverses, the apparent improvement is an artifact of the overlapping judge rather than a real gain in object-state generation. A second check is to compute paired per-prompt differences with confidence intervals, since the paper reports no significance testing and gains within noise would also weaken the claim.","tokens_in":15027,"feed_emoji":"🖼️","tokens_out":7868,"duration_ms":94312,"temperature":0.7,"pith_summary":"Text-to-image models trained on web-scale data almost always render a kitchen shelf or table in its 'full' state, because that is what the data shows and because captions rarely mention absent objects. This paper claims that a fully automatic pipeline—LLM-generated prompts about empty or absent objects, synthetic images generated from those prompts, automatic filtering by a vision-language judge, and recaptioning into natural phrasings—produces training data that fixes much of this failure. LoRA fine-tuning of five open models on the resulting 7,600 image-text pairs lifts average prompt-image alignment by 8.2 points on a curated subset of GenAI-Bench and 24.6 points on the authors' new Object State Bench, and the gain transfers to objects never seen in training. The point of the paper is that object-state understanding can be taught cheaply with targeted synthetic data rather than requiring architectural changes or inference-time correction.","feed_headline":"Synthetic empty-scene data fixes image AI's 'full state' bias","feed_subtitle":"Fine-tuning five open models on 7,600 synthetic pairs lifts prompt-to-image state accuracy by 24.6 points","key_machinery":"The central mechanism is a filter-then-finetune synthetic data loop. An LLM writes thousands of short prompts describing everyday objects in empty or absent states; a diffusion model turns them into candidate images; a vision-language model keeps only candidates that truly show the requested absence; and an LLM rewrites the surviving prompts into more natural, non-template wording. Fine-tuning with low-rank adapters (LoRA) then lets each base model absorb the visual pattern of 'absence' without retraining it wholesale. What carries the argument is the filtering step, which converts noisy synthetic generations into a clean training signal, and the recaptioning step, worth several points by the paper's ablation, which keeps the model from latching onto the single word 'empty.'","core_discovery":"Current text-to-image systems fail on prompts like 'a table without any bottle' or 'an empty tumbler,' often regenerating the very object that should be absent; the paper attributes this to contextual bias in training data and to text encoders' weak grasp of negation. The paper's discovery is that such state concepts can be instilled by fine-tuning with synthetic examples that make absence visible. The pipeline curates roughly 3,000 everyday objects, writes template prompts for their empty states, generates candidate images with Stable Diffusion 1.5, filters out any image that a vision-language model judges not to show the requested absence, and paraphrases the prompts to avoid template-like wording. Fine-tuning with low-rank adapters on the filtered set improves average scores across Stable Diffusion 1.5, 2.1, SDXL, Flux Dev, and OmniGen on both evaluation sets; holds up on 100 unseen objects; leaves full-state generation and unrelated prompts largely intact; and outperforms fine-tuning on matched real data from COCO and VidOSC. The authors take this as evidence that the model learns a generalizable notion of an object's empty or absent state rather than memorizing training examples.","pith_inferences":["Editorial inference: because the same vision-language model filters the training data and scores the results, part of the reported gain may reflect the judge rewarding images that resemble its own filtering behavior; an independent judge or human labels would give a truer effect size.","Editorial inference: the same synthetic-data loop could plausibly attack sibling failure modes the authors name as future work—counting, spatial relations, and attribute negation—since the mechanism is just 'generate candidate, filter on the target relation, fine-tune.'","Editorial inference: the 400-step, 7,600-pair recipe is probably not the optimum; scaling the object list, using a stronger generator for the synthetic candidates, or adding negative examples of objects that should be present could push the same idea further."],"forward_implications":["Across Stable Diffusion 1.5, 2.1, SDXL, Flux Dev, and OmniGen, average alignment with object-state prompts rises 8.2 points on GenAI-Object-State and 24.6 points on Object State Bench, so the fix transfers across architectures.","Fine-tuned models also improve on 100 objects that never appeared in the synthetic training set, meaning the learned state concept is not just memorization.","Training on synthetic 'empty state' pairs beats fine-tuning on the same volume of real images from COCO and VidOSC, which suggests data composition, not dataset size, is the bottleneck for this failure mode.","Recaptioning template prompts into conversational negatives adds 3-8 points depending on benchmark and metric, so how absence is phrased is itself a significant part of the cure.","Full-state prompts, unrelated GenAI-Bench prompts, and visual quality (FID, CLIP score, user preference) stay roughly level after fine-tuning, indicating the targeted data does not degrade general generation."],"supporting_citations":[{"why":"Supplies GenAI-Bench, the public compositional benchmark whose negation subset the authors manually filter into GenAI-Object-State.","marker":"[12]"},{"why":"Supplies the vision-language model used for prompt generation, image filtering, recaptioning, and final evaluation.","marker":"[10]"},{"why":"Introduces the VQA-score evaluation that provides the second, independent accuracy number for every model.","marker":"[14]"},{"why":"Provides Stable Diffusion, both the base models fine-tuned and the generator used to create the synthetic training images.","marker":"[24]"},{"why":"Provide SDXL, one of the five fine-tuned models and a synthetic-data generator tested in the generator ablation.","marker":"[17, 21]"},{"why":"Provides Flux Dev, another fine-tuned model and candidate synthetic-data generator.","marker":"[11]"},{"why":"Provides OmniGen, the fifth fine-tuned model.","marker":"[31]"},{"why":"Supplies COCO captions and images used as the real-data comparison condition in the data-source ablation.","marker":"[13]"},{"why":"Supplies VidOSC video-derived images used as the second real-data comparison condition.","marker":"[32]"}],"fun_headline_variants":["Synthetic absences teach text-to-image models to respect 'empty' prompts","Data pipeline makes image models show absent objects on command","Fine-tuning on synthetic 'empty' examples boosts object absence accuracy","Making absence visible: synthetic data fixes AI's object state errors","Synthetic data pipeline instills empty-state awareness in image models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a vision-language model's yes/no answer reliably measures whether an image shows the requested empty or absent state; since the same judge filters the training data and later scores the outputs, the reported gains could partly reflect the judge favoring images that match its own filtering choices.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic absences teach text-to-image models to respect 'empty' prompts","Data pipeline makes image models show absent objects on command","Fine-tuning on synthetic 'empty' examples boosts object absence accuracy","Making absence visible: synthetic data fixes AI's object state errors","Synthetic data pipeline instills empty-state awareness in image models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001364,"raw_usage":{"total_tokens":5527,"prompt_tokens":935,"completion_tokens":4592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":4505}},"tokens_in":551,"tokens_out":4592,"duration_ms":37428,"temperature":1.0,"reasoning_tokens":4505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:56:42.677944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the fine-tuned and baseline models on a human-annotated subset of Object State Bench, or with a vision-language judge that played no role in generating or filtering the training data; if the 24.6-point average lead shrinks to near zero or reverses, the apparent improvement is an artifact of the overlapping judge rather than a real gain in object-state generation. A second check is to compute paired per-prompt differences with confidence intervals, since the paper reports no significance testing and gains within noise would also weaken the claim.","supporting_citations":[],"review_version":1}