{"id":"87306ca2-1008-4366-babc-1a9da22da58f","arxiv_id":"2506.18322","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SpuriVerse, a benchmark of 124 real-world spurious correlation types, shows LVLMs fail on such cases and that diverse synthetic training improves robustness to unseen spurious correlations.","lead":"This paper creates a test set of 124 real-world situations where image-question-answering models are misled by irrelevant but common visual cues, such as an umbrella suggesting rain. It then shows that training models on synthetic versions of these misleading situations helps them cope with new, unseen situations, at some cost to accuracy on normal questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 78.40% meta-skill claim lacks a matched control: fine-tuning on the synthetic without-spurious-feature core groups would show whether the gain comes from spurious emphasis or from generic synthetic-image fine-tuning.","rationale":"I read the paper's strongest claim as the generalization result in Section 3.3: fine-tuning on spurious-group images raises held-out anchor accuracy to 78.40% (Qwen) and 80.00% (Llama), which is interpreted as learning a general shortcut-avoidance skill. What must be true for this claim is that the improvement is caused by the spurious-feature emphasis in training, not by the synthetic data format, the shared question-option text, or other properties of the spurious group. The existing non-spurious control is drawn from real source-benchmark images and therefore differs in more than the spurious attribute; it cannot rule out these alternatives. The core group already generated in Step 4 is the natural matched control and is not used in any fine-tuning condition in Table 4. If core-group fine-tuning also recovers most of the gain, the benchmark's curation pipeline could still be valid as a diagnostic, but the paper's central causal claim about synthetic spurious patterns would not be established. This concern is concrete, testable with existing assets, and directly addresses the abstract's 'training on diverse spurious patterns generalizes' assertion. It is close to but more specific than the reader's weakest assumption: the reader focused on confounds between core and spurious image groups in the benchmark's construction; my concern is that the fine-tuning experiment lacks a matched control that would disentangle spurious emphasis from synthetic-image distribution. On the positive side, the paper reports 5-seed finetuning, reasonable held-out-by-type splits, and a consistent pattern across two models, and the proposed control is the main missing piece. I therefore keep the conditional verdict: the paper should be accepted only after the core-group fine-tuning control is run and reported, or the claim should be softened accordingly.","tokens_in":16486,"tokens_out":7807,"duration_ms":85151,"concrete_test":"Run the matched core-group control. Using the splits and hyperparameters of Section 3.3, fine-tune Llama-3.2-11B-vision-instruct and Qwen2.5-VL-7B-instruct on the core group images (the 10 synthetic images per training spurious-correlation type that omit the spurious feature) instead of the spurious group images, then evaluate on the same held-out anchors and held-out spurious groups. If held-out anchor accuracy is close to 78.40/80.00, the spurious-feature emphasis is not the active ingredient, and the meta-skill interpretation should be withdrawn or weakened; if it remains near the anchor-fine-tuning row (45.60/59.20), the result supports the paper's claim. For extra rigor, repeat with text-only (question+options without image) fine-tuning to rule out language-side memorization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3's headline result compares spurious-group fine-tuning against no fine-tuning and against fine-tuning on the 'non-spurious set,' which is sampled randomly from source benchmarks. That control confounds two variables: presence of the spurious feature and the synthetic/regularized image distribution of the spurious group. Each spurious-group sample is one of 10 Stable Diffusion images generated from a curated scene description, all sharing the same question, options, and answer; the non-spurious controls are real images with heterogeneous composition. A model could therefore reach 78.40% on held-out anchors by learning the general mapping from these synthetic VQA instances to correct options—including the repeated instruction-like phrasing of the text—rather than by learning to suppress spurious visual cues. The paper already created the exact matched control but does not run it: the core group images are the same synthetic pipeline, same scene-description prompts, same question/answer, with only the spurious attribute removed (Section 2.2, Steps 3–4). Fine-tuning on core groups, with identical splits and hyperparameters, would isolate whether the improvement depends on exposing the model to the spurious pattern. The reader's confound concern about core/spurious image differences is related, but the decisive missing experiment is the core-group fine-tuning control, because it directly tests the causal role of the spurious feature in the central claim. A secondary note: the held-out anchor set is only 25 items, so the 78.40% vs. 59.20% gap is about five correct answers; this makes the matched control especially important.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SpuriVerse, a benchmark of 124 spurious-correlation types extracted from real-world VQA benchmarks by starting from GPT-4o errors, filtering through LVLM/human verification, and generating paired Stable Diffusion counterfactual images with and without the suspected spurious attribute. Each type contributes 1 real anchor and 10 synthetic spurious-group images, yielding 1,364 multiple-choice questions. The authors evaluate 15 open- and closed-source LVLMs, report that even the best model scores below their stated random baseline, test prompting strategies, and fine-tune Llama-3.2-11B and Qwen2.5-VL-7B on anchors, spurious groups, non-spurious samples, and a mixed set. The headline claim is that fine-tuning on synthetic examples that emphasize spurious correlations raises held-out anchor accuracy to 78.40% on previously unseen spurious-correlation types, suggesting a learnable meta-skill to ignore shortcuts, at the cost of a trade-off on non-spurious samples.","tokens_in":16829,"tokens_out":7052,"duration_ms":71519,"significance":"If the central claims hold, SpuriVerse would be a useful complement to existing spurious-correlation benchmarks because it targets diverse, naturally occurring correlations in general VQA settings rather than a single contrived task, and the cross-type fine-tuning result would be an interesting and falsifiable finding. The paper also has clear strengths: the curation pipeline combines model-error mining, human verification, and synthetic counterfactual validation; the evaluation spans 15 models including closed-weight frontier systems; and the appendix gives substantial reproducibility detail on prompts, hyperparameters, and compute. However, the current evidence does not yet support the headline generalizations. The reported 'random' baseline is inconsistent with the 4-choice format, the benchmark selection criterion partly builds in the failure it later reports, the headline fine-tuning comparison lacks a matched synthetic control, and the main fine-tuning experiment does not document a type-level train/test split. These issues are addressable, so the manuscript merits major revision rather than rejection.","major_comments":[{"comment":"The 'Random' row reports 40.70% in all three columns, but SpuriVerse uses four-option multiple-choice questions, for which uniform random guessing gives 25%. The paper never defines how the 40.70% baseline is computed. If it is a majority-class or answer-frequency baseline, calling it 'random guess' is misleading, and the repeated claim that all LVLMs are worse than random (Abstract and Section 3.1) is unsupported; if it is intended as uniform random, the number is wrong. This matters because several model accuracies (e.g., 37.90%) are above 25% but below 40.70%, so the baseline choice changes the qualitative conclusion. Please specify the baseline construction and revise the claims accordingly.","section":"Section 3.1, Table 2"},{"comment":"The benchmark is constructed by retaining samples only when max over {GPT-4o, Gemini 2.0 Flash, Qwen-VL-Max} of [Acc(core) - Acc(spurious)] is at least 30%. The later observation that models perform poorly on the spurious group is therefore partly a restatement of the selection rule: every retained sample is guaranteed to trip at least one of the three selector models. Evaluating other models on these samples is still informative, but the manuscript should state this explicitly and should report sensitivity of the main conclusions to the 30% threshold and to the choice of the error-source model. In addition, because the core and spurious groups are generated from different scene descriptions, the 30% gap may reflect uncontrolled differences in image composition or visual complexity rather than the targeted spurious attribute alone.","section":"Section 2.2, Step 5"},{"comment":"The headline fine-tuning result lacks a matched control. The 'non-spurious set' is sampled randomly from the source benchmarks and differs from the spurious group in both the presence of the spurious attribute and the synthetic/stylized image distribution, including the repeated question and instruction template shared by all synthetic group samples. The core groups described in Section 2.2, Steps 3-4 are the natural matched control: they use the same Stable Diffusion pipeline, the same scene-description prompts, and the same questions, with only the spurious attribute removed. The paper should fine-tune on the core groups with identical splits and hyperparameters. Without this control, the 78.40% result does not establish that the model learned to suppress spurious cues; it could reflect generic adaptation to the synthetic VQA distribution.","section":"Section 3.3, Table 4"},{"comment":"The claim that fine-tuning generalizes to previously unseen spurious correlations requires that the train/test split be made by spurious-correlation type, so that no test anchor has its corresponding 10 spurious-group images in the training set. Section 3.4 explicitly states that splits are performed by type 'to ensure we are measuring generalization to unseen cases,' but the main fine-tuning setup in Section 3.3 and Appendix B.4 only says the anchor set and spurious groups are divided 70/10/20, without specifying the splitting unit. If the split is sample-level, a test anchor can share its spurious-correlation type with spurious-group images used for training, leaking the exact pattern the paper claims is unseen. Please specify the splitting unit and, if sample-level splitting was used, rerun the experiment with type-level splits.","section":"Section 3.3 and Appendix B.4"}],"minor_comments":[{"comment":"The Abstract says the best model achieves only 37.1% accuracy, but Table 2 reports the best anchor accuracy as 37.90% (llama-3.2-11b and qwen-2.5-32b); these numbers should be reconciled.","section":"Abstract and Table 2"},{"comment":"There are typos and naming inconsistencies: 'Contextural cues' should be 'Contextual cues,' and model names such as 'qwen-2.5-7b' are inconsistent with the 'Qwen2.5-vl-7b-instruct' used elsewhere.","section":"Figure 3b and Table 2"},{"comment":"Minor wording and typographical issues include 'No prompt-based methods such as as Chain-of-Thought' (duplicated 'as') and 'sytem prompt' in Appendix B.2.","section":"Section 1 and Appendix B.2"},{"comment":"The annotation interface description contains 'clickong' instead of 'clicking,' and the appendix does not report inter-annotator agreement despite asserting that human annotations were done by two contributors.","section":"Appendix A.2"},{"comment":"The sentence 'Its accuracy drops 66.48% when finetuned on anchors' should read 'drops to 66.48%,' since the table shows a drop from 73.44% to 66.48%.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The two decisive conditions for publishability are (1) adding the core-group fine-tuning control, and (2) documenting and, if necessary, correcting the type-level split for the headline generalization result. The 40.70% 'random' baseline should also be corrected in revision; as written, it reverses the interpretation of the main evaluation table."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about 2506.18322: the benchmark is a real contribution, and the headline fine-tuning result is not yet supported by the controls in the paper. The 40.70% 'random' baseline is also mislabeled, though that is fixable.\n\nWhat is new: the authors build SpuriVerse bottom-up from GPT-4o's errors on real VQA benchmarks, ask an LLM to attribute errors to spurious features, have humans refine those, then use Stable Diffusion counterfactuals to keep only samples where a model's accuracy drops by at least 30% when the spurious feature is present. That gives 124 distinct spurious correlation types with real anchors plus synthetic counterparts. It is a more realistic and diverse instrument than Waterbirds or CelebA. Evaluating 15 LVLMs, they find all score low; prompting (CoT, spurious-aware) does not fix it; and fine-tuning on a diverse set of spurious examples improves held-out performance, with a clear trade-off on non-spurious data. They report 5 seeds and full hyperparameters.\n\nThe soft spots, in proportion:\n\n- The 'random guess' claim is wrong as stated. With 4-choice questions, uniform random is 25%, not 40.70%. The 40.70% looks like a majority-class baseline. Several models exceed 25%, so 'all models are worse than random' is false under uniform random; it only holds against majority class. Easy to fix, but the current wording misleads.\n\n- The 78.40% meta-skill claim is missing the right control. Fine-tuning on the spurious group is compared to fine-tuning on the non-spurious set, which is real images from the source benchmarks. That confounds two variables: the spurious feature and the synthetic-image distribution. The authors already created the matched control—the core group images from Step 4—but did not fine-tune on them. If core-group fine-tuning also gets near 78%, the 'spurious emphasis' story collapses into a generic synthetic-data effect. This is the experiment I would want before believing the abstract's 'learn to avoid shortcuts' framing.\n\n- The held-out anchor test set is 25 items, so 78.40% vs 59.20% is about five correct answers. The spurious-group test set (250 items) is more stable, but the headline anchor number is noisy.\n\n- The benchmark is partly self-selecting: samples are kept only when some model shows a 30% gap, so 'models fail on SpuriVerse' is partly by construction. Acceptable for a benchmark, but the paper should say so plainly rather than presenting it as an independent finding.\n\nThe authors do not release the dataset in the preprint, which hurts reproducibility. Fixing that and adding the core-group control would make the fine-tuning claim credible.\n\nWho it is for: people building or testing LVLM robustness, and anyone working on shortcut learning in multimodal settings. I would send it to review with a request for the control and baseline correction. It is not ready as-is, but it is worth one serious round.","headline":"A genuinely useful new benchmark for LVLM spurious correlations, but the headline fine-tuning claim needs a missing control before it is believable.","tokens_in":17362,"tokens_out":6082,"would_cite":true,"duration_ms":62925,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper builds a benchmark of 124 naturally occurring visual shortcuts and shows that fine-tuning on synthetic spurious examples transfers to unseen spurious correlations, while revealing a robustness-accuracy trade-off.","keywords":["spurious correlation","large vision-language models","visual question answering","shortcut learning","synthetic counterfactual images","fine-tuning generalization","robustness-accuracy trade-off","benchmark"],"falsifier":"Re-measure core-versus-spurious accuracy using image pairs matched on style, seed, layout, and rendering quality, then see whether the 30-percentage-point gaps that define SpuriVerse's anchors persist; if the gaps collapse, the benchmark is capturing generator artifacts instead of spurious reliance. Alternatively, have human annotators label which images contain the spurious feature and check whether models' errors track human judgments better than the core-group contrast does.","tokens_in":16341,"feed_emoji":"🖼️","tokens_out":6313,"duration_ms":56037,"temperature":0.7,"pith_summary":"SpuriVerse is a benchmark of 124 spurious correlations mined from real visual-question-answering data, each with one real image and ten synthetic images engineered to contain or omit the spurious cue. On these naturally occurring shortcuts, fifteen large vision-language models all score below random guessing on the real anchors, with the best at 37.9% against a 40.7% random baseline. The paper's central experimental claim is that fine-tuning on synthetic images that emphasize diverse spurious patterns transfers to spurious correlations never seen during training: accuracy on held-out anchors rises to 80.0% for Llama-3.2-11B and 78.4% for Qwen2.5-VL-7B. This transfer comes with a cost, because the same fine-tuning lowers accuracy on ordinary non-spurious samples, suggesting models rely on shortcuts for part of their normal performance. The broader point is that shortcut reliance in pretrained multimodal models can be studied, measured, and partly trained away with a single diverse benchmark.","feed_headline":"Synthetic spurious images lift LVLM scores from 35% to 78%","feed_subtitle":"124 real-world visual shortcuts show even top vision-language models beat random only after diverse fine-tuning.","key_machinery":"The load-bearing object is the paired counterfactual image group. For each error sample, the pipeline generates two scene descriptions from the question and correct answer—one containing the candidate spurious attribute and one without it—and renders ten Stable Diffusion images per description. Selection into SpuriVerse requires that at least one of three reference LVLMs (GPT-4o, Gemini 2.0 Flash, or Qwen-VL-Max) shows an accuracy gap of at least 30 percentage points between the core group and the spurious group. That core-versus-spurious gap is the operational definition of spurious reliance; the same synthetic spurious groups are then used as the fine-tuning data that produces the transfer result.","core_discovery":"On its own terms, the paper establishes that state-of-the-art LVLMs systematically fail on real-world VQA questions whenever a dominant but unreliable visual feature points to the wrong answer, and that this failure is not fixed by prompting. The operational discovery is the transfer result: fine-tuning Llama-3.2-11B-vision-instruct on synthetic spurious-group images raises held-out anchor accuracy from 41.60% to 80.00%, and Qwen2.5-VL-7B-instruct from 35.20% to 78.40%, where the held-out set contains spurious correlation types absent from fine-tuning. The same training raises accuracy on held-out synthetic spurious groups to 79.12% and 75.60%. Prompting strategies help only modestly, with the best spurious-aware prompt reaching 51.61% anchors for Qwen2.5-VL-7B, barely above random. The paper interprets these results as evidence that models can learn a general meta-skill of ignoring dominant-but-unreliable cues and attending to overall scene context, while noting a systematic trade-off: accuracy on non-spurious samples falls by roughly 14 to 17 percentage points when models are fine-tuned on spurious groups alone.","pith_inferences":["If the transfer result generalizes beyond these two open-weight models, spurious reliance can be treated as a general attentional failure mode rather than 124 independent memorized correlations, and one diverse counterfactual dataset might serve as a general robustness intervention for other VLMs.","Because the benchmark is seeded from GPT-4o's errors, SpuriVerse is likely biased toward shortcuts that trip GPT-4o; models with different failure modes may need their own error-driven construction, and a fair cross-model comparison would require errors sourced from each model.","A direct test of the meta-skill interpretation would be to fine-tune on spurious groups from only half the categories and check whether held-out gains concentrate in those categories or spread across all six; the paper does not report per-category held-out results.","One testable extension is to compare fine-tuning on synthetic spurious groups with fine-tuning on real anchors matched for size; if synthetic images are the key driver, the transfer should persist even when the real anchors are withheld from training."],"forward_implications":["Any LVLM can be scored on SpuriVerse's 124 anchors to expose shortcut reliance, because the gap between anchor and non-spurious accuracy isolates how much performance comes from spurious cues.","Fine-tuning on the synthetic spurious groups, not the original anchors, is what produces large gains on unseen spurious correlations, so the synthetic counterfactual images are the active ingredient.","Prompt-level fixes such as chain-of-thought or explicit spurious-awareness are insufficient; only training updates materially change shortcut behavior.","A model that avoids spurious cues pays a measurable accuracy penalty on ordinary samples, implying shortcut use contributes to normal-case performance.","The robustness-accuracy trade-off grows with the proportion of spurious training samples, so the diversity of spurious correlations in the training mix controls how much generalization transfers."],"supporting_citations":[{"why":"NaturalBench supplies the adversarial real-world VQA images that seed the error set.","marker":"[Li et al., 2024a]"},{"why":"A-OKVQA is another source of GPT-4o errors and world-knowledge multiple-choice questions.","marker":"[Schwenk et al., 2022]"},{"why":"SEEDBench provides the first batch of multiple-choice VQA samples mined for errors.","marker":"[Li et al., 2023a]"},{"why":"SEEDBench2 is the second SEED source of GPT-4o errors used in Step 1 of curation.","marker":"[Li et al., 2024b]"},{"why":"Stable Diffusion renders the paired core and spurious synthetic images that make counterfactual evaluation possible.","marker":"[Podell et al., 2023]"},{"why":"Waterbirds defines the task-oriented spurious-correlation setup that SpuriVerse generalizes beyond.","marker":"[Sagawa et al., 2019]"},{"why":"Supplies the shortcut-learning taxonomy used to label the 124 correlation categories.","marker":"[Geirhos et al., 2020]"},{"why":"Chain-of-thought prompting is the baseline prompting method shown to be insufficient.","marker":"[Wei et al., 2022]"}],"fun_headline_variants":["Synthetic spurious images train LVLMs to ignore real-world shortcuts","From 35% to 78%: synthetic spurious data beats LVLM shortcut trap","Real-world visual traps: synthetic fine-tuning busts LVLM shortcuts to 78%","Synthetic spurious patterns: the unexpected fix for LVLM real-world failures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that the synthetic core and spurious images differ only in the targeted spurious attribute; if other visual differences (style, quality, composition) leak through, the accuracy gap between groups measures those confounds rather than the model's reliance on the spurious cue.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic spurious images train LVLMs to ignore real-world shortcuts","From 35% to 78%: synthetic spurious data beats LVLM shortcut trap","Real-world visual traps: synthetic fine-tuning busts LVLM shortcuts to 78%","Synthetic spurious patterns: the unexpected fix for LVLM real-world failures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001107,"raw_usage":{"total_tokens":4659,"prompt_tokens":1038,"completion_tokens":3621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":3534}},"tokens_in":654,"tokens_out":3621,"duration_ms":27669,"temperature":1.0,"reasoning_tokens":3534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:51:41.943516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-measure core-versus-spurious accuracy using image pairs matched on style, seed, layout, and rendering quality, then see whether the 30-percentage-point gaps that define SpuriVerse's anchors persist; if the gaps collapse, the benchmark is capturing generator artifacts instead of spurious reliance. Alternatively, have human annotators label which images contain the spurious feature and check whether models' errors track human judgments better than the core-group contrast does.","supporting_citations":[],"review_version":2}