REVIEW 8 cited by
Evaluating Spatial Understanding of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) show remarkable capabilities across a variety of tasks. Despite the models only seeing text in training, several recent studies suggest that LLM representations implicitly capture aspects of the underlying grounded concepts. Here, we explore LLM representations of a particularly salient kind of grounded knowledge -- spatial relationships. We design natural-language navigation tasks and evaluate the ability of LLMs, in particular GPT-3.5-turbo, GPT-4, and Llama2 series models, to represent and reason about spatial structures. These tasks reveal substantial variability in LLM performance across different spatial structures, including square, hexagonal, and triangular grids, rings, and trees. In extensive error analysis, we find that LLMs' mistakes reflect both spatial and non-spatial factors. These findings suggest that LLMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains.
Forward citations
Cited by 8 Pith papers
-
SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
Dense scene-graph-grounded rewards let a 7B multimodal LLM trained on 7K synthetic questions beat SFT and sparse-RL baselines and outscore GPT-4o on average across 12 spatial/real-world benchmarks.
-
AeroDuo: Aerial Duo for UAV-based Vision and Language Navigation
Two drones at different altitudes, one guided by a vision-language model and one by a local navigator, reach targets more often than single-drone baselines on a new UAV navigation benchmark.
-
FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations
A new benchmark with 2,000 floorplans and 16,000 symbolic spatial questions shows LLMs are strong at simple metrics but weak at geometric unions and collision-free planning.
-
A Study on Individual Spatiotemporal Activity Generation Method Using MCP-Enhanced Chain-of-Thought Large Language Models
An MCP-enhanced chain-of-thought LLM framework generates individual daily activity-travel chains whose aggregate patterns resemble real mobile signaling data in the Lujiazui case study.
-
USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents
USTBench is the first benchmark that decomposes urban spatiotemporal reasoning into understanding, forecasting, planning, and reflection, and shows LLMs struggle most with planning and reflection.
-
Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
Large language models, especially GPT-4 with few-shot prompts, can classify topological spatial relations between WKT-encoded geometries with roughly 0.6 to 0.66 accuracy, though errors cluster near conceptually simil...
-
Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agent
A robot replanner that compares scene graphs to successful demonstrations before each subtask, triggering LLM-based replanning on mismatch, raises task success in AI2-THOR.
-
The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical features
GPT-4o and Gemini 2.0 Flash approximate the geography of Austria but show systematic biases in coordinates and elevations and frequent errors in assigning federal states.
Discussion (0). Sign in to comment.