{"id":"dafa1e4f-72e5-4983-9e2a-b70f46e10447","arxiv_id":"2607.07251","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Open vision-language models fail to select spatial demonstratives based on object distance in a human-like manner across four languages.","lead":"This paper builds a benchmark to test whether vision-language models (VLMs) can correctly use spatial words like 'this' or 'that' across four languages based on object distance. It finds that current VLMs fail to adjust demonstrative choice with distance as humans do, revealing a specific gap in spatial reasoning.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The prompt never establishes the VLM as the deictic center (speaker/viewer), so the model may lack the perspective grounding needed to judge proximity—undermining the claim that failures are specifically about spatial deixis.","rationale":"The reader correctly identified the single-prompt vulnerability and the small sample sizes as concerns, and the CONDITIONAL verdict is appropriate. However, the reader's framing of the prompt issue as a phrasing/translation artifact misses the more structural problem: the prompt never assigns the model a deictic role. This is not about whether the translation was good or whether alternative phrasings would help—it is about whether the experimental design provides the minimal information needed for the task to be meaningful at all. In human deixis experiments, the speaker's embodied presence provides the deictic center automatically; in a VLM setting, this must be explicitly established, and its absence is a confound that could fully explain the central finding. The paper's own acknowledgment in §7 (\"since our evaluation relies on a single prompt, it is difficult to rigorously determine whether the observed trends reflect the models' intrinsic capabilities or are artifacts of the prompt's phrasing\") gestures at this but does not name the specific structural gap. The qualitative finding that distal demonstratives remain below 5% across all distances for most models is somewhat robust to sample size concerns, but without the perspective-grounding control, we cannot distinguish 'VLMs can't do spatial deixis' from 'VLMs weren't told where they stand.' The verdict remains CONDITIONAL—this is a novel and interesting benchmark, but the central claim is not yet established with sufficient controls.","tokens_in":11158,"tokens_out":3417,"duration_ms":163294,"concrete_test":"Run two control conditions: (1) Add perspective-grounding to the prompt (e.g., \"You are standing at the near end of the desk. The disk is at some distance from you. Describe the shape...\") and recompute the demonstrative distributions. If the distribution shifts significantly toward human-like patterns, the original finding is an artifact of missing deictic role assignment. (2) Add a distance-estimation control task asking the model to classify the disk as near/medium/far or estimate its distance. If the model cannot reliably distinguish the three distance conditions from the images, the demonstrative failure reflects depth-perception limitation, not deixis-specific reasoning failure.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that VLMs fail to use demonstratives as a function of distance, implying a deficit in spatial deixis reasoning. However, the prompt (§4.2) reads: \"Analyze the image and identify the shape on the disk. Describe it by filling in the following three-word template exactly: [Demonstrative] [Color] [Shape].\" At no point does the prompt tell the model that it is the speaker, that it is standing at the near end of the desk, or that it should judge proximity from the camera's viewpoint. In the human memory game (Coventry et al., 2023), participants are physically present and inherently know they are the deictic center. The VLM has no such grounding. Without an established deictic center, the model has no reference point from which to judge whether an object is \"near\" (proximal) or \"far\" (distal). The observed failure to shift demonstratives with distance could therefore reflect a missing perspective instruction rather than a failure of spatial deixis understanding per se. The paper does not include any control validating that the VLM can perceive the distance differences in the synthetic images (e.g., a distance-estimation pre-task), nor does it test whether adding perspective-grounding instructions changes the distribution. This is distinct from the reader's concern about prompt phrasing/translation—it is about a structural absence of deictic role assignment. Additionally, the extremely small valid sample sizes (Table 1: Gemma 3 4B Japanese n=12, ~4 per distance condition) make the quantitative JS distances in Table 2 unreliable, though the qualitative pattern (distal <5% across distances) may be robust to sample size.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper introduces a multilingual benchmark to evaluate whether vision-language models (VLMs) can use spatial deictic expressions (demonstratives like 'this' and 'that') in a human-like manner. The benchmark is based on the 'memory game' paradigm from psycholinguistics (Gudde et al., 2018; Coventry et al., 2023), where a disk with a colored shape is placed at varying distances (0.25 m, 1.50 m, 2.75 m) on a desk. Four VLMs (Gemma 3 4B/12B, Qwen3-VL 8B/32B) are prompted in four languages (English, Japanese, Korean, Chinese) to describe the disk using a fixed template: [Demonstrative] [Color] [Shape]. The authors compute the probability distribution of demonstrative usage from model logits and compare it to human data from Coventry et al. (2023) using Jensen-Shannon distance. The main finding is that VLMs fail to reproduce human-like shifts in demonstrative selection as a function of distance, particularly in Japanese and Korean, where distal demonstratives are almost never used.","tokens_in":11372,"tokens_out":1037,"duration_ms":180138,"significance":"The paper addresses a genuine gap: while spatial reasoning in VLMs has been benchmarked extensively, the use of spatial deixis—expressions whose meaning depends on the speaker's situational context—has not been explicitly studied. The cross-linguistic dimension (languages with two vs. three demonstratives) adds value. The benchmark is constructed independently of the tested models, the human reference data comes from an external source, and the metric (Jensen-Shannon distance) is standard, so there is no circularity concern. The code is publicly available. These are real strengths. However, the significance of the findings is tempered by a structural issue in the experimental design (see Major Comment 1) and very small effective sample sizes, which together make it difficult to attribute the observed failures specifically to a deficit in spatial deixis reasoning.","major_comments":[{"comment":"§4.2 (Prompt): The prompt never establishes the VLM as the deictic center (speaker/viewer). The instruction reads: 'Analyze the image and identify the shape on the disk. Describe it by filling in the following three-word template exactly: [Demonstrative] [Color] [Shape].' At no point does the prompt tell the model that it should judge proximity from the camera's viewpoint, that the camera position corresponds to the speaker's position, or that 'this' refers to objects near the camera. In the human memory game (§3.1), participants are physically present and inherently know they are the deictic center. The VLM has no such grounding. Without an established deictic center, the model has no reference point from which to judge whether an object is 'near' (proximal) or 'far' (distal). The observed failure to shift demonstratives with distance could therefore reflect a missing perspective-ground","section":null}],"minor_comments":[{"comment":"§4.2: The prompt was translated by Google Gemini and then manually patched with medial demonstratives for Japanese and Korean. The paper should report the exact final prompts used in each language, ideally in an appendix, to allow reproduction and to rule out translation artifacts.","section":null},{"comment":"Table 1: The caption should clarify that these counts are out of 60 and that the probability distributions in Figure 2 are computed only on these valid samples.","section":null},{"comment":"Figure 2: The figure is difficult to read in monochrome; the legend (green/orange/light blue) is hard to distinguish. Consider using patterns or distinct labels.","section":null},{"comment":"§5.1: The phrase 'a general trend is to avoid distal demonstratives in languages with three demonstratives' could be quantified with a summary statistic across models/languages for clarity.","section":null},{"comment":"§7 (Limitations): The authors acknowledge the single-prompt limitation and small dataset size. These are appropriate acknowledgments, but they also underscore the need for the controls suggested in Major Comment 1 before the central claim can be fully supported.","section":null}],"recommendation":"major_revision","confidential_remarks":"The core idea is good and the paper is well-motivated, but the experimental design has a structural gap: the deictic center is never assigned to the VLM. This is the most important issue to address. If the authors add perspective-grounding to the prompt and run a control distance-estimation task, and the VLMs still fail to shift demonstratives, the paper's contribution would be substantially strengthened. The small sample sizes are a secondary concern but also need attention. I would encourage the authors to expand the dataset and consider multiple prompt variants. The paper is borderline; with these revisions it could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"This paper applies the psycholinguistic “memory game” paradigm (Coventry et al. 2023) to VLMs, asking models to produce demonstratives (this/that, kono/sono/ano, etc.) for objects at three distances in synthetic images. The core finding: VLMs don’t shift demonstrative choice with distance the way humans do, and they almost never use distal forms in three-demonstrative languages. The multilingual deictic angle is genuinely new—nobody has tested this before, and the cross-linguistic setup (English, Chinese, Japanese, Korean) is well-motivated by real typological differences in demonstrative systems. Shipping the benchmark and code is a plus. The qualitative pattern is probably real and worth reporting. The soft spots are real too. The dataset is tiny—60 images total, and some models only produced valid outputs for 12 of them (Gemma 3 4B, Japanese), meaning the JS distances in Table 2 rest on ~4 samples per distance band. The quantitative claims are not reliable at that scale. The stress-test note raises a point I think is valid and under-addressed: the prompt never tells the VLM it is the deictic center. In the human memory game, participants are physically present and inherently know they’re the speaker. The VLM gets “analyze the image and fill in the template” with no perspective grounding. The observed failure to shift demonstratives could partly reflect a missing deictic role assignment rather than a pure spatial reasoning deficit. The paper acknowledges the single-prompt limitation in §7 but doesn’t run the obvious control: add a perspective instruction and see if distributions change. That’s the experiment that would separate “VLMs can’t do deixis” from “VLMs weren’t told they’re the speaker.” The reader’s CONDITIONAL verdict is about right. The central observation is likely robust at the qualitative level, but the precision of the quantitative claims doesn’t hold up, and the missing deictic-center control is a genuine gap. This is for people working on VLM spatial reasoning or multilingual evaluation. It identifies a real capability gap but doesn’t fully diagnose it. Worth a serious referee who pushes for the perspective-grounding control and more data before accepting the quantitative claims.","headline":"New benchmark for multilingual spatial deixis in VLMs; results are directionally right but underpowered","tokens_in":12001,"tokens_out":539,"would_cite":false,"duration_ms":148648,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"VLMs ignore distance when choosing 'this' versus 'that'","keywords":[],"falsifier":"If a VLM, given a more naturalistic or embodied prompt setting (e.g., a first-person scene with conversational context rather than a static image and a rigid three-word template), were shown to shift demonstrative choice with distance in a human-like way, the central claim that VLMs lack distance-sensitive deictic competence would be substantially weakened.","tokens_in":11217,"feed_emoji":"📏","tokens_out":2376,"duration_ms":143615,"temperature":0.7,"pith_summary":"The paper constructs a benchmark to test whether vision-language models (VLMs) can use spatial deictic expressions — words like 'this' and 'that' whose meaning depends on physical distance to the referenced object — in a human-like way across four languages: English, Chinese, Japanese, and Korean. The benchmark adapts the 'memory game' paradigm from linguistics: objects are placed at controlled distances (0.25m, 1.50m, 2.75m) on a desk, and the model must describe each using a demonstrative plus color plus shape. The authors compare the resulting probability distributions of demonstrative choice against human data from a large cross-linguistic study. The central finding is that all tested models fail to reproduce the human pattern: humans shift from proximal ('this') to medial to distal ('that') demonstratives as objects move farther away, but VLMs largely do not. In Japanese and Korean, which have three demonstratives, models almost never use the distal form regardless of distance. The authors also find a modest negative correlation (r = −0.40) between object recognition accuracy and deviation from human demonstrative distributions, suggesting that models with better visual grounding tend to use demonstratives somewhat more like humans.","feed_headline":"VLMs ignore distance when choosing 'this' versus 'that'","feed_subtitle":"Models across four languages fail to shift demonstratives with object distance, unlike human speakers.","key_machinery":"The 'memory game' paradigm adapted for VQA: synthetic images generated in Blender showing a colored shape on a disk placed at one of three controlled distances, with the model prompted to output '[Demonstrative] [Color] [Shape].' Demonstrative probability distributions are computed from model logits and compared to human distributions using Jensen-Shannon distance.","core_discovery":"Vision-language models do not adjust their choice of spatial demonstratives ('this' vs. 'that' and their cross-linguistic equivalents) based on the physical distance to the referenced object, unlike humans across all four languages tested. The human-like shift from proximal to medial to distal demonstratives as distance increases is absent or severely muted in every model evaluated. In three-demonstrative languages (Japanese, Korean), models almost entirely avoid the distal form. Better object recognition correlates weakly with more human-like demonstrative use, but the correlation is modest, indicating that visual grounding alone does not close the gap.","pith_inferences":["If the deficit stems from a lack of genuine spatial representation rather than linguistic knowledge, VLMs might also fail on other distance-dependent linguistic phenomena — for instance, spatial adverbs like 'near'/'far,' or tense-aspect systems in languages where spatial distance modulates verbal morphology.","The fact that some models with strong object recognition still fail at demonstrative selection points to a gap in the language-vision alignment mechanism specifically, not just in visual perception — suggesting that the binding between spatial features in the visual encoder and distance-sensitive lexical choices in the language decoder is where the breakdown occurs.","Testing in embodied or first-person settings — where the model occupies a viewpoint within a scene rather than observing a static image — could reveal whether the deficit is specific to the static-image paradigm or reflects a deeper inability to model deictic reference from a speaker's perspective."],"forward_implications":["If VLMs cannot properly use deictic expressions, they may fail in real-world assistive or robotic settings where 'this' vs. 'that' carries critical spatial information about which object to act upon.","The absence of distance-sensitive demonstrative shifting could serve as a diagnostic signal for whether a VLM possesses genuine spatial understanding versus surface-level pattern matching.","The near-total avoidance of distal demonstratives in three-demonstrative languages suggests training data imbalances may distort cross-linguistic competence in measurable, language-specific ways.","The modest correlation between recognition accuracy and human-likeness implies that improving visual grounding may partially improve spatial language competence, but a substantial gap would remain."],"fun_headline_variants":["VLMs fail to shift 'this' vs 'that' with object distance across four languages","Vision-language models skip distance cues when picking spatial demonstratives","Multilingual VLMs don't adjust demonstratives by distance like humans do","Models ignore proximal-to-distal shifts that humans make across languages","Spatial deixis in VLMs lacks the distance sensitivity humans rely on"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The benchmark assumes that a single fixed prompt, translated from English by an automated tool and manually patched with medial demonstratives, can elicit demonstrative usage from VLMs that is fairly comparable to the human memory game paradigm — but the paper itself acknowledges that it cannot determine whether the observed trends reflect intrinsic model capabilities or artifacts of the prompt's phrasing.","fun_headline_variants_meta":{"raw":{"variants":["VLMs fail to shift 'this' vs 'that' with object distance across four languages","Vision-language models skip distance cues when picking spatial demonstratives","Multilingual VLMs don't adjust demonstratives by distance like humans do","Models ignore proximal-to-distal shifts that humans make across languages","Spatial deixis in VLMs lacks the distance sensitivity humans rely on"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":615,"prompt_tokens":518,"completion_tokens":97,"prompt_tokens_details":null},"tokens_in":518,"tokens_out":97,"duration_ms":34675,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T16:05:32.287249+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a VLM, given a more naturalistic or embodied prompt setting (e.g., a first-person scene with conversational context rather than a static image and a rigid three-word template), were shown to shift demonstrative choice with distance in a human-like way, the central claim that VLMs lack distance-sensitive deictic competence would be substantially weakened.","supporting_citations":[],"review_version":1}