REVIEW 7 cited by
Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Spatial expressions in situated communication can be ambiguous, as their meanings vary depending on the frames of reference (FoR) adopted by speakers and listeners. While spatial language understanding and reasoning by vision-language models (VLMs) have gained increasing attention, potential ambiguities in these models are still under-explored. To address this issue, we present the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to systematically assess the spatial reasoning capabilities of VLMs. We evaluate nine state-of-the-art VLMs using COMFORT. Despite showing some alignment with English conventions in resolving ambiguities, our experiments reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning.
Forward citations
Cited by 7 Pith papers
-
GenSpace: Benchmarking Spatially-Aware Image Generation
GenSpace benchmarks spatial awareness in image generation with a 3D reconstruction-based evaluator, showing models struggle with allocentric relations and metric measurements.
-
LLM-Based Social Simulations Require a Boundary
LLM-based social simulations are scientifically useful only within boundaries set by behavioral variance, and current validation practice under-checks variance.
-
RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills
RobotSmith autonomously designs, 3D-prints, and uses task-specific tools for robotic manipulation, raising task success from 2.8% (no tool) to 50% in simulation.
-
SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
SpaRE builds a 3.4M-question synthetic spatial QA dataset from captioning sources and reports large gains on VSR and What's Up after fine-tuning Qwen2-VL models.
-
Vision language models are unreliable at trivial spatial cognition
Three vision-language models give inconsistent left/right judgments on simple synthetic tabletop images when the same relation is probed with logically equivalent prompt variations.
-
Augmented Vision-Language Models: A Systematic Review
A structured taxonomy of inference-time augmentation techniques that connect vision-language models to external symbolic systems, tools, and knowledge sources.
-
Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
The authors argue that spatial reasoning in multimodal LLMs requires dedicated new recipes in data, architecture, and training objectives, and will not emerge from scaling alone.
Discussion (0). Continue with ORCID to comment.