{"id":"eccb920b-44cf-47fb-86da-bc732a87336b","arxiv_id":"2504.16061","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Three vision-language models give inconsistent left/right judgments on simple synthetic tabletop images when the same relation is probed with logically equivalent prompt variations.","lead":"This paper tests whether vision-language models can consistently answer simple questions about which object is left or right of another in clean, synthetic images. Across three state-of-the-art models and eight prompt formats, accuracy swings dramatically with minor rewording, suggesting these models do not form stable spatial representations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central inference rests on an unmeasured human ceiling; without a human baseline the 'unreliable' interpretation is not established.","rationale":"The reader's weakest assumption identifies exactly the vulnerability I would stress. The within-prompt empirical contrasts are internally credible: the evaluations are large, the stimuli are well controlled, and the statistical approach is appropriate for repeated measures. But the step from 'accuracy varies' to 'unreliable' requires a benchmark of expected human behavior. The authors assert this benchmark in Section 1.1 and in the figure captions without measuring it. This is not a missing control in the usual sense; it is the normative yardstick against which every main effect is interpreted. A human experiment is the direct, inexpensive check that would settle the interpretation. I also flag a minor internal inconsistency in Section 3.1, where the text says a model should respond false to two statements that are actually true given Figure 1b; this reinforces the need for released data and code, and for a measured human baseline. Other issues noted by the reader (no confidence intervals, only three models, no shipped data/code) are real but secondary: they affect robustness and generality, not the logical dependency of the central claim as directly as the missing human ceiling. Therefore the conditional verdict remains appropriate, and no further adjustment is needed.","tokens_in":14821,"tokens_out":14012,"duration_ms":138884,"concrete_test":"Run the exact same 8-prompt battery with the same TableTest stimuli on at least 30 neurotypical adults, self-paced, with no feedback, and score with the same accuracy rules. Test whether human accuracy is at or near ceiling (above 0.98) on every prompt and every variation, and whether repeated-measures tests detect any significant main effects or interactions for left/right wording, object order, truth value, or presence of the irrelevant relation. If humans show any significant effects that mirror the VLM effects, the paper's assumption of logical equivalence and triviality is falsified; if humans are at ceiling with no effects, the VLM unreliability claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference from 'accuracy varies across prompt formulations' to 'models lack stable spatial representations' depends on the normative standard that every manipulated variable (left vs. right wording, object order, truth value, presence of a distractor relation) is logically or conceptually irrelevant, and that neurotypical adults are at ceiling on all eight prompts. This standard is asserted, not measured. Section 1.1 defines 'trivial' by reference to expected adult behavior, and the Figure 2/3 captions state 'Humanlike performance anticipated to be at ceiling (accuracy = 1.0)', but no human data are reported anywhere. If adults show any reliable asymmetry on these exact stimuli (e.g., true statements easier than false ones, or a left/right confusion effect), then the observed VLM differences are not ipso facto evidence of unreliability; they may reflect task difficulty or response biases shared by humans. The conclusion would then need to be recast as 'models are less accurate than humans' or 'models are overly sensitive to wording,' rather than 'models lack stable spatial representations.' A related internal slip in Section 3.1 calls two true statements ('the cupcake is to the left of the blender' and 'the blender is to the right of the cupcake') false, which further underscores that the authors' truth-value mapping and the normative assumptions behind it need explicit validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TableTest, a synthetic benchmark of 3D-rendered table scenes with known ground-truth left/right relations, and uses it to probe three VLMs (BLIP, Llama 3.2, Idefics2) across eight prompt formats, each with surface variations that the authors treat as logically equivalent. In multimodal evaluations, accuracies range from .12 to 1.00 across prompts; the authors report stable main effects of left vs. right wording, object order, true vs. false statements, and the presence of an irrelevant relation, and argue that because these variations should not affect a system with a coherent spatial representation, the models are unreliable at trivial spatial cognition. Text-only control conditions generally show higher accuracy, and open-form prompts (6-8) are scored with a permissive custom metric described in Appendix A.","tokens_in":15024,"tokens_out":6309,"duration_ms":59593,"significance":"If the empirical claims hold, the paper makes a useful contribution: a synthetic benchmark with exact ground truth, systematic and reproducible prompt variants, and nonparametric object-level analyses showing that VLM performance is sensitive to surface factors. The cross-model consistency of the main effects is a strength, as is the deliberate focus on trivial tasks with simple stimuli. However, the central interpretation depends on an unmeasured human ceiling and on a scoring procedure partially derived from the tested models' own outputs, so the significance is conditional on those issues being resolved. The work would be more valuable if the benchmark and scoring code were released and if human norming data were collected.","major_comments":[{"comment":"The paper's central inference, from 'accuracy varies across prompt formulations' to 'models lack stable spatial representations,' depends on the assertion that neurotypical adults are at ceiling on all eight prompts and that every manipulated variable is logically irrelevant. These claims are asserted in Section 1.1 and in the figure captions ('Humanlike performance anticipated to be at ceiling'), but no human data on the exact stimuli or prompts are reported. If adults show even a small true-vs-false asymmetry or a left/right advantage on these materials, the observed VLM differences would not demonstrate the absence of stable spatial representations; they would show that the models track difficulty or response biases that humans also share. I recommend adding a human norming study on the same images and prompts, or a clearly applicable published baseline, and tempering the concluding claim accordingly.","section":"§1.1, Figs 2-3, §3.3.9"},{"comment":"There is a factual slip in the Prompt 1 example. The text says that for Figure 1b (blender to the right of a cupcake) a VLM should respond false to the statements 'the cupcake is to the left of the blender' and 'the blender is to the right of the cupcake'; both statements are true for that image. Please correct the example and, more importantly, verify the ground-truth relation labels in the Prompt 1 evaluation code, because the reported true-vs-false accuracy asymmetry is central to the paper's conclusions.","section":"§3.1"},{"comment":"The open-form accuracy metric is partly defined by the evaluated models' own responses. Acceptable object labels are entered into the lookup table whenever a VLM 'routinely generated' them five or more times, and the metric explicitly counts a response as correct unless it fails to recognize the relation. This creates a circularity that can inflate accuracy on Prompts 6-8 and makes those prompts not directly comparable with the closed-form prompts. The acceptable-alternative list should be constructed independently of the tested models (e.g., from human annotation, WordNet synonyms, or the Objaverse labels), and the full-sentence exception in Prompt 6 scoring should be reported as a separate analysis.","section":"Appendix A"}],"minor_comments":[{"comment":"The paper reports aggregate proportions without confidence intervals or effect sizes, and the unit of analysis for the Wilcoxon tests is implicit. Please report bootstrapped confidence intervals or variance across objects, and state explicitly whether the tests are run on per-object accuracy scores.","section":"Table 2, Figs 2-3"},{"comment":"In §3.3.6, 'see 2f' should be 'see Figure 2f'; in Section 1, 'no affect on performance' should be 'no effect on performance.'","section":"§3.3.6 and §1"},{"comment":"The paper refers to Llama 3.2 but cites the Llama 2 paper [25]. Please cite the correct model release or technical report for Llama 3.2.","section":"References"},{"comment":"The statement that 'adequate performance should be ≥ 90%' introduces an arbitrary threshold. Please justify it or explicitly label it as a descriptive criterion rather than a statistical standard.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's main empirical pattern—that logically equivalent prompt variants produce large accuracy swings—is likely to interest the community, but the missing human baseline is a serious risk to the interpretive claim in the title. I would encourage the editor to require either human norming on the same prompts and images or a substantially weakened version of the central claim. The benchmark itself could be a useful resource if the stimulus set and scoring scripts are released."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Karim,\n\nQuick take: this is a decent benchmark paper with a real, if modest, contribution. TableTest and the systematic prompt-variation protocol are genuinely new—Kamath et al. measured spatial accuracy, not invariance under logically equivalent rephrasings. The finding that three VLMs swing from ceiling to near-chance depending on wording, object order, and an irrelevant distractor relation is credible and consistent across architectures. The open-form prompts (6–8) and the text-only controls are a nice addition; the text-only ceiling on Prompt 7 correctly identifies text-matching, not spatial competence.\n\nWhat I'd want fixed before I cite it as established:\n\nFirst, the human baseline. The whole 'trivial' framing rests on the assumption that neurotypical adults are at ceiling on all eight prompts. They never measure a human. I don't think this kills the paper—the invariance argument stands on its own: a model that drops from 98% to 26% on logically equivalent questions is unstable regardless of human performance. But the title says 'unreliable at trivial spatial cognition,' and 'trivial' is a claim about human performance. At minimum they should hedge, or better, run a small human sample on the same images and prompts. That would also let them distinguish 'worse than humans' from 'lacks stable representations.'\n\nSecond, no data or code. They say ancillary analyses are 'available online' but give no link, and the dataset isn't released. For a benchmark paper, that's a real problem. The circularity in Appendix A—semantic alternatives collected from the models' own outputs—is mild, but releasing the lookup tables would make it auditable.\n\nThird, minor but embarrassing: Section 3.1 calls two true statements false (cupcake left of blender; blender right of cupcake, given the image). That's a writing slip, not a result-wrecking error, but it doesn't inspire confidence. Also, the Llama 3.2 citation is actually the Llama 2 paper.\n\nOverall: the core observation holds up. I'd send this to a serious referee, not desk-reject it. The fixes are straightforward: human baseline, data/code release, confidence intervals on the headline proportions. If those land, it's a citeable benchmark for evaluating spatial reliability in VLMs.\n\nRecommendation: engage—peer review with moderate expectation of revision.","headline":"A useful benchmark with a solid core finding—VLMs are unstable across logically equivalent spatial prompts—but the 'unreliable' claim is stronger than the evidence, and the paper needs a human baseline and code/data release.","tokens_in":15590,"tokens_out":2700,"would_cite":true,"duration_ms":26263,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tested vision-language models answer logically equivalent left/right questions inconsistently, with accuracy ranging from 12% to 100% depending on wording, object order, and whether the relation holds.","keywords":["vision-language models","spatial cognition","left/right relations","prompt reliability","synthetic benchmark","relational reasoning","image-text evaluation","TableTest"],"falsifier":"Give neurotypical adults the exact TableTest images with all eight prompt families and their variations; if their accuracy moves with wording, object order, or truth value by more than a small margin, the assumed human ceiling of 1.0 fails and the models' variation cannot be called unreliable by the paper's own standard.","tokens_in":14612,"feed_emoji":"🖼️","tokens_out":9292,"duration_ms":86025,"temperature":0.7,"pith_summary":"The paper sets out to establish that state-of-the-art vision-language models are unreliable at trivial spatial cognition: recognizing which of two objects is left or right of the other in an uncluttered image. Using a new synthetic dataset called TableTest and eight families of simple prompts, the authors show that accuracy swings with variations that should be meaningless, including left-versus-right wording, the order in which objects are named, whether the asked relation is true or false, and the presence of an irrelevant relation. Each tested model falls below a 90% adequacy threshold on at least two prompt families, and no model is consistent across them. The point matters because a model that answers one phrasing correctly but fails an equivalent one cannot be trusted in applications that depend on stable spatial judgments.","feed_headline":"Reworded prompts swing VLM left/right accuracy from 12% to 100%","feed_subtitle":"Vision-language models should give the same answer when a left/right question is reworded; the tested models do not.","key_machinery":"TableTest is the central instrument: a synthetic benchmark of 4,032 two-object images, plus a larger set of three-object scenes, built from 64 Objaverse objects rendered in Blender on a uniform table with object centers placed 200 pixels from the table's center. The paper pairs each image with eight prompt families whose variations, including left/right wording, object order, truth and falsity, and the insertion of an irrelevant relation, are meant to be logically equivalent. The benchmark's work is to hold the visual scene constant while varying only the surface form of the query, so any accuracy shift isolates a model's instability rather than a confound in the image.","core_discovery":"Across three readily available vision-language models, the paper finds that accuracy on trivial left/right recognition depends on nearly every surface variable in the prompt. Overall accuracy spans 0.12 to 1.00 across eight prompt families; every model falls below 90% on at least two multimodal prompts, and factors the paper treats as logically irrelevant, namely whether the query says left or right, whether objects are named in image order, whether the queried relation is true or false, and whether an unrelated relation like \"inside of\" is added, reliably change accuracy. The models also err in different directions from one another, so the variation is not a single stable bias. The paper's conclusion is that these models do not construct coherent spatial representations even for relations a human resolves instantly.","pith_inferences":["We infer that reliability across equivalent prompt variants is a more informative metric than peak accuracy for choosing a model in practice: a model that is consistently adequate on every phrasing may be preferable to one that is very accurate on a single phrasing and near chance on others.","We infer that the pattern of left/right and object-order biases could be formalized as a response-bias model, letting engineers predict on which unseen prompt variations a given model will fail before running a full benchmark.","We infer that systems chaining multiple VLM calls, such as an image caption followed by a relational query, may compound or cancel these prompt-format errors, so reliability should also be measured at the pipeline level.","We infer that a human baseline on the same TableTest prompts is the natural way to set the \"trivial\" threshold; until such data exist, the paper's ceiling of 1.0 should be treated as a stipulated standard rather than a measured one."],"forward_implications":["A high score on a single spatial prompt does not establish spatial competence: the same model can be near ceiling on one logically equivalent phrasing and near chance on another.","Evaluation protocols for VLMs should report reliability across equivalent prompt variations, not only accuracy on a fixed template, because surface format measurably changes results.","Fine-tuning on a single prompt is unlikely to generalize to other phrasings unless the model forms a coherent spatial representation; errors reappear as soon as the wording changes.","Other spatial relations, such as above, below, inside, and between, are likely to exhibit the same instability, since training captions rarely enumerate all relations in an image."],"supporting_citations":[{"why":"Supplies the prior benchmark showing VLMs struggle with relational descriptions; its four-option prompt is the template that motivates the paper's equivalent-prompt variations.","marker":"[14]"},{"why":"Provides the standardized 3D objects from which all TableTest scenes are built.","marker":"[9]"},{"why":"Identifies the Idefics2 architecture that the paper evaluates.","marker":"[17]"},{"why":"Identifies the BLIP architecture underlying the InstructBlip-Vicuna model evaluated.","marker":"[18]"},{"why":"Identifies the Llama architecture used for one of the models evaluated.","marker":"[25]"},{"why":"Grounds the \"trivial\" standard by defining primitive relation recognition in uncluttered scenes as a basic spatial behavior.","marker":"[12]"}],"fun_headline_variants":["VLM spatial skills flip with prompt tweaks: 12% to 100%","Left/right VLM accuracy swings with rephrasing","VLMs fail trivial spatial reasoning: prompts shift accuracy","Prompt wording can swing VLM spatial accuracy to 12-100%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central conclusion rests on the premise that logically equivalent phrasings of a left/right question should produce identical accuracy, together with the paper's assumption, which is not backed by human data, that neurotypical adults answer all these probes at ceiling.","fun_headline_variants_meta":{"raw":{"variants":["VLM spatial skills flip with prompt tweaks: 12% to 100%","Left/right VLM accuracy swings with rephrasing","VLMs fail trivial spatial reasoning: prompts shift accuracy","Prompt wording can swing VLM spatial accuracy to 12-100%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"total_tokens":1132,"prompt_tokens":863,"completion_tokens":269,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":193}},"tokens_in":479,"tokens_out":269,"duration_ms":2693,"temperature":1.0,"reasoning_tokens":193,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:11:14.728968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give neurotypical adults the exact TableTest images with all eight prompt families and their variations; if their accuracy moves with wording, object order, or truth value by more than a small margin, the assumed human ceiling of 1.0 fails and the models' variation cannot be called unreliable by the paper's own standard.","supporting_citations":[{"cited_title":"Objaverse: A universe of annotated 3d ob- jects","cited_arxiv_id":null,"evidence_quote":"Provides the standardized 3D objects from which all TableTest scenes are built."},{"cited_title":"BLIP: Bootstrapping language-image pre- training for unified vision-language understanding and generation","cited_arxiv_id":null,"evidence_quote":"Identifies the BLIP architecture underlying the InstructBlip-Vicuna model evaluated."},{"cited_title":"Spatial lan- guage and spatial representation.Cognition, 55(1):39– 84, 1995","cited_arxiv_id":null,"evidence_quote":"Grounds the \"trivial\" standard by defining primitive relation recognition in uncluttered scenes as a basic spatial behavior."}],"review_version":1}