{"id":"7ba58fa5-1753-420a-a530-b26a8e6b988c","arxiv_id":"2412.18224","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors expand the VSR benchmark with 50 question templates and diffusion-augmented images, fuse four vision encoders, and report a VSR-specialist VLLM (VSRE) that substantially outperforms LLaVA1.5 on spatial reasoning questions.","lead":"This paper expands the VSR spatial reasoning benchmark with 50 question templates and diffusion-generated images, and reports a vision language model, VSRE, that improves accuracy on spatial relation questions. The result is a practical recipe for making multimodal models more sensitive to positional details and less biased by wording.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 27% claim is inflated by test-template leakage: VSRE's 79% Test-S is trained and selected on the same fixed template, while the baseline is from a different 200-sample subset; a fair held-out comparison is needed.","rationale":"The reader's conditional verdict already captures the main risks, but the weakest assumption is better located in the evaluation protocol than in the diffusion-generated image quality. The reported 79% is obtained on Test-S, whose fixed template is also present as the final `turn-s 11k` training stage, and the model is selected by best performance on the test sets themselves. The 52% baseline appears to come from a different 200-sample subset rather than from the zero-shot Test-S accuracy of 68.4 reported in Table 1. These two issues directly undermine the headline 27% improvement and the generalization claim. A fair held-out evaluation with new templates and a checkpoint trained without turn-s 11k would settle the matter. Even if such an evaluation shows a smaller gain, the paper still demonstrates a real improvement from data scaling and multi-encoder fusion; the conditional verdict remains appropriate pending corrected reporting and the promised release of code and data.","tokens_in":12048,"tokens_out":4014,"duration_ms":36088,"concrete_test":"Construct 20 new answer templates not in the 50-template pool, apply them to the VSR test triplets to form Test-H; also fine-tune the 4-encoder model identically but without the final `turn-s 11k` stage and evaluate on Test-S. Compare VSRE to LLaVA1.5 13B zero-shot on Test-H and on no-turn-s Test-S; if the improvements are less than 10 absolute points, revise the 27% claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that VSRE surpasses LLaVA1.5 13B by 27% (from 52% to 79%) on the VSR test set (Conclusion). The 79% figure is the Test-S accuracy of the four-encoder VSRE in Figure 4. Test-S freezes the template `[caption], True or false.` (Datasets section). The paper's own Table 1 shows that adding `turn-s 11k`, a training set using exactly that template, raises Test-S from 75.7 to 76.6 for the 13B model, and the 'best-performing model' is then selected by Test-G/Test-S performance ('we designated the best-performing model with 4 backbones ... as VSRE'). The 52% baseline is not the zero-shot Test-S accuracy of LLaVA1.5 13B, which is 68.4 in Table 1; 52% appears only in Figures 5-6 captions on a 200-sample, 7-relation subset. Thus the headline mixes a test-set-contaminated evaluation with a non-comparable baseline. The same template pool used to build Test-G and turn-g 500k, so 'generalizes better to different instructions' is not tested on held-out instruction formats. The diffusion-image consistency concern raised in the reader's weakest assumption is secondary: even if every repainted image preserves its label, the evaluation protocol does not support the claimed 27% generalization gain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses visual spatial reasoning (VSR) for vision large language models (VLLMs). The authors introduce unified instruction test sets (Test-G and Test-S) derived from the VSR dataset, expand the training data through template-based text rewriting and diffusion-based image repainting, and merge four vision encoders (CLIP, SigLIP, DINOv2, SAM) into a single VLLM architecture. The resulting model, VSRE, is reported to achieve large gains on VSR and related subsets of MME, MMBench, and SEED, with a headline claim of a 27% accuracy increase over LLaVA1.5 13B on the VSR test set.","tokens_in":12356,"tokens_out":5817,"duration_ms":47741,"significance":"The paper's strengths are its concrete contributions: a reproducible protocol for expanding both text and image data for spatial reasoning, a systematic comparison of vision encoder combinations, and an open-sourced model and data release. If the reported gains survive a clean evaluation, the work would be a useful step toward making VLLMs more robust to instruction phrasing and more sensitive to spatial layout. The multi-encoder study is informative and the scaling experiments are extensive. However, the central quantitative claim is compromised by an incomparable baseline and by potential overlap between training and test templates, so the significance is not yet established.","major_comments":[{"comment":"The headline claim that 'VSRE surpasses the performance of LLaVA1.5 13B by 27% (from 52% to 79%) on VSR test set' is not supported by the reported numbers. The 52% figure appears in the captions of Figures 5 and 6 and refers to a 200-sample subset of 7 spatial relations, while the 79% figure is the Test-S accuracy of the four-encoder VSRE from Figure 4. Table 1 reports the zero-shot LLaVA1.5 13B on the full Test-S as 68.4 and on Test-G as 57.7; the improvement from the best data-only model to the four-encoder model is from 75.7 to 79 on Test-S and from 70.3 to 74.4 on Test-G. Please replace the 52%-to-79% comparison with an apples-to-apples comparison on the same test set, and adjust the abstract and conclusion accordingly.","section":"Conclusion; Table 1; Figures 5-6"},{"comment":"The evaluation protocol has a circularity problem for the instruction-generalization claim. Test-G randomly samples a template from the same 50-template pool that is used to construct the turn-g 500k training data, and Test-S freezes the template '[caption], True or false,' which is also used in the turn-s 11k training set. In addition, the 'best-performing model' (VSRE) is selected by its Test-G/Test-S accuracy in Figure 4. Therefore, the claim that VSRE 'generalizes better to different instructions' has not been tested on held-out instruction formats. A clean test would reserve a disjoint set of templates (or template families) for evaluation, or the claim should be explicitly qualified as performance on instruction formats seen during training.","section":"Datasets; Training and Inference; Figure 4"},{"comment":"The diffusion-based image expansion lacks a quality check on the key property it is meant to enforce. The paper states that 'we freeze the text captions as prompts' and uses SDXL with image-to-image, text-to-image, and inpainting with bounding-box masks, but it does not report any human or automatic verification that the generated images still contain the labeled spatial relation. If a nontrivial fraction of generated images do not display the relation (e.g., 'in' rendered as 'next to'), then training on them may inject label noise rather than a stronger spatial prior, which would undermine the interpretation of the scaling gains. Please provide a verification protocol, such as human evaluation of a random sample or agreement with an existing VSR classifier, and report the consistency rate.","section":"Expansion on Image Data"}],"minor_comments":[{"comment":"The last row uses numeric subscripts (1,2,3) to denote training order, but the order is only explained in the text; please add an explicit notation explanation in the table caption.","section":"Table 1"},{"comment":"The legend refers to 'rainbows' with dashed and solid lines, but without color it is difficult to distinguish the individual encoder combinations; adding line markers or a table of final accuracies would improve readability.","section":"Figure 4"},{"comment":"Please report the number of samples in the 'relevant subsets' of MME, MMBench, and SEEDv2, and define which categories were selected, since the subset sizes are essential for interpreting the score differences.","section":"Table 3"},{"comment":"There are typos in the experimental section: 'tunning' should be 'tuning' and 'pertaining' should be 'pre-training' (e.g., in the 'Experiment' paragraph and the 'Expansion on Vision encoder' section).","section":"Experiment"},{"comment":"The claim of expanding spatially positioned image data with diffusion models 'for the first time' should either be supported by a literature search or softened, since the related-work section does not fully justify this novelty statement.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a large amount of useful empirical work, but the headline claim and the evaluation protocol need to be corrected before it can be accepted. The data and model releases are valuable, and I believe the core issues are addressable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The 27% claim in the abstract and conclusion is not supported by the paper's own numbers: \"from 52% to 79%\" compares a 200-sample, 7-relation subset against the full Test-S, while the actual zero-shot Test-S accuracy of LLaVA1.5 13B in Table 1 is 68.4. And the Test-S score of 79 is suspect because the turn-s 11k training set uses the exact same fixed template as Test-S, and the model was selected on Test-G/Test-S performance. The template leakage is real.\n\nThat said, the underlying result is still a genuine improvement, just smaller and messier than advertised. Data expansion alone moves LLaVA1.5 13B from 57.7 to 70.3 on Test-G (Table 1), and the four-encoder VSRE reaches 74.4 on Test-G (Figure 4). Those are apples-to-apples comparisons within the same test set. The paper also does several things well: it diagnoses the over-sensitivity to instruction phrasing and under-sensitivity to visual position, releases a unified 50-template test set, and runs a serious scaling study across multiple LLMs (vicuna, Llama2, Llama3) and VLLMs (Qwen-VL, BLIP2, InstructBLIP) with consistent gains. The generalization checks on MME, MMBench and SEED are a good addition.\n\nSoft spots beyond the headline. The template leakage means the \"generalizes better to different instructions\" claim is weaker than stated: Test-G uses the same 50-template pool as the turn-g 500k training set, so held-out instruction formats are not really tested. The diffusion-generated images have no verification that the repainted image preserves the labeled spatial relation; a human or automatic check would be needed. And there are no error bars—everything is single-run. These are fixable with mostly-presentational changes plus a bit of extra validation.\n\nWho this is for: anyone working on spatial reasoning benchmarks for VLLMs, or on instruction-following sensitivity. It deserves a serious referee; the core recipe is useful and the scaling evidence is real. But the authors need to correct the headline, run a true held-out template evaluation, and at least spot-check the generated images. I'd send it to review with major revision.","headline":"Real and useful VSR recipe underneath a headline 27% claim that doesn't survive contact with the paper's own tables.","tokens_in":12915,"tokens_out":2814,"would_cite":true,"duration_ms":23259,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A diffusion-and-encoder expansion of the VSR benchmark lifts VLLM spatial reasoning from 52% to 79% accuracy.","keywords":["visual spatial reasoning","vision large language models","diffusion-based data augmentation","multi-encoder fusion","instruction tuning","answer bias","VSR benchmark","SDXL repainting"],"falsifier":"Have annotators verify a random sample of the generated pre-training images for whether the stated relation holds; if a large share are relation-inconsistent, retrain VSRE on only the verified subset and compare VSR accuracy. A drop would confirm the generated images are load-bearing; an unchanged score would show the gain comes from the templates or encoder fusion instead.","tokens_in":11842,"feed_emoji":"🧠","tokens_out":5997,"duration_ms":43144,"temperature":0.7,"pith_summary":"The paper argues that current vision-language models fail at visual spatial reasoning because they over-rely on language instructions and under-use visual positions, and that this can be fixed by expanding both sides of the training pipeline. It expands the VSR benchmark's text with 50 question templates and its images by repainting with the SDXL diffusion model in three modes, then merges four visual encoders (CLIP, SigLIP, DINOv2, SAM) into one vision stack. The resulting model, VSRE, scores 79% on the VSR test set versus 52% for LLaVA1.5 13B, a gain of over 27 percentage points, while also improving on spatial subsets of MME, MMBench, and SEEDv2. If right, the result shows that controlled diffusion-based data expansion and multi-encoder fusion are effective levers for relational reasoning in VLLMs.","feed_headline":"Diffusion-expanded data lift visual-spatial accuracy from 52% to 79%","feed_subtitle":"Training on diffusion-repainted images and four fused visual encoders yields a 27-point gain over LLaVA1.5.","key_machinery":"The load-bearing machinery is a two-arm expansion pipeline plus a merged vision encoder. The text arm rewrites VSR triplets into 50 template-based QA formats (30 hand-written, 20 GPT-4o generated) to reduce prompt sensitivity; the image arm uses SDXL to repaint seed images in three modes—image-to-image, text-to-image, and inpainting with bounding-box masks—to multiply visual examples while keeping captions fixed. The vision arm concatenates features from CLIP, SigLIP, DINOv2, and SAM after per-encoder projectors/adapters, so the LLM receives both language-aligned and self-supervised/segmentation-aligned visual tokens. These three arms together produce the VSRE model and its two test protocols, Test-G (random template) and Test-S (fixed template).","core_discovery":"On the paper's own terms, the central discovery is that a VLLM can be turned into a spatial-reasoning expert by expanding the VSR benchmark along two axes: rewriting every [subject, relation, object] triplet into dozens of instruction-answer pairs, and repainting the seed images with a diffusion model (image-to-image, text-to-image, and bounding-box inpainting) so that the model sees up to 100x more visual examples of the same relation. The second half of the recipe is architectural: freezing CLIP and adding SigLIP, DINOv2, and SAM features, each aligned by a projector/adapter and concatenated along the feature dimension, gives the model finer visual detail. The paper reports that VSRE reaches 79% accuracy on the VSR test set versus 52% for the LLaVA1.5 13B baseline, that t-SNE plots of its visual tokens separate spatial-relation categories more cleanly, and that the yes/no answer bias shrinks while scores on MME, MMBench, and SEEDv2 spatial subsets also rise.","pith_inferences":["If the diffusion repainting does not reliably preserve the labeled relation, part of the 27-point gain could come from learning the template distribution rather than spatial geometry; a human-verified subset would isolate the two.","The same template-expansion trick could be applied to other VSR-style benchmarks (e.g., relationship questions in VQA) to diagnose prompt sensitivity before investing in image generation.","Because the four encoders are frozen, the architecture is cheap to reproduce; swapping in a depth or optical-flow encoder might give further spatial detail at the cost of inference speed.","The t-SNE clustering result is suggestive but not a metric of comprehension; a stricter test would ask VSRE to judge adversarial triplets where the same object pair appears in conflicting relations."],"forward_implications":["VSRE's gains transfer to spatial subsets of MME, MMBench, and SEEDv2, suggesting the method is not overfit to VSR's formatting.","The open-sourced data and model let other teams apply the same diffusion-template-encoder recipe to other relation types or benchmarks.","The reduced yes/no bias indicates that visual attention shifts from entity co-occurrence to actual positional geometry, a claim that can be probed with attention maps.","Scaling pre-training data yields diminishing returns beyond roughly 400k images, so the practical recipe is pre-400k plus 500k instruction samples."],"supporting_citations":[{"why":"Supplies the original VSR dataset with [subject, relation, object] triplets that seed all expansions and the test set.","marker":"(Liu, Emerson, and Collier 2023)"},{"why":"SDXL is the diffusion model used for all repainting (image-to-image, text-to-image, inpainting).","marker":"(Podell et al. 2023)"},{"why":"CLIP is the baseline vision encoder whose features are merged with three others.","marker":"(Radford et al. 2021)"},{"why":"SigLIP provides a language-supervised encoder that the paper finds strongest as a single backbone.","marker":"(Zhai et al. 2023)"},{"why":"DINOv2 contributes self-supervised features that help the model focus on fine visual detail.","marker":"(Oquab et al. 2024)"},{"why":"SAM adds segmentation-oriented features to the merged encoder.","marker":"(Kirillov et al. 2023)"},{"why":"LLaVA1.5 is the base VLLM architecture and the baseline against which the 27% gain is measured.","marker":"(Liu et al. 2024a)"}],"fun_headline_variants":["Fusing four visual encoders and diffusion data makes a spatial VLLM expert","From 52% to 79%: spatial reasoning VLLM via diffusion and multi-encoder fusion","Spatial VLLM expert: 27-point gain from diffusion data and four encoders","Diffusion repaints images to teach VLLM spatial rules; hits 79%","Four encoders plus diffusion data boost VLLM spatial accuracy by 27 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on the assumption that diffusion-repainted images still contain the spatial relation named in the frozen caption, yet the paper applies no check—human or automatic—that the relation actually survives generation.","fun_headline_variants_meta":{"raw":{"variants":["Fusing four visual encoders and diffusion data makes a spatial VLLM expert","From 52% to 79%: spatial reasoning VLLM via diffusion and multi-encoder fusion","Spatial VLLM expert: 27-point gain from diffusion data and four encoders","Diffusion repaints images to teach VLLM spatial rules; hits 79%","Four encoders plus diffusion data boost VLLM spatial accuracy by 27 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001057,"raw_usage":{"total_tokens":4500,"prompt_tokens":1072,"completion_tokens":3428,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":3313}},"tokens_in":688,"tokens_out":3428,"duration_ms":20246,"temperature":1.0,"reasoning_tokens":3313,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:54:37.334913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have annotators verify a random sample of the generated pre-training images for whether the stated relation holds; if a large share are relation-inconsistent, retrain VSRE on only the verified subset and compare VSR accuracy. A drop would confirm the generated images are load-bearing; an unchanged score would show the gain comes from the templates or encoder fusion instead.","supporting_citations":[],"review_version":1}