Do Multimodal Language Models Really Understand Direction? A Benchmark for Compass Direction Reasoning

Bin Sun; Hang Yin; Kan Li; Xin Liu; Zhifeng Lin

REVIEW 4 major objections 5 minor 27 references

Do Multimodal Language Models Really Understand Direction? A Benchmark for Compass Direction Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Most multimodal language models answer compass-direction questions no better than random guessing, and the paper shows that fine-tuning with diverse data plus chain-of-thought reasoning lifts a small model to 53% accuracy.

desk verdict A genuinely new compass-direction benchmark with a striking near-chance finding, held back by unreleased data and a missing human baseline. read the letter →

arxiv 2412.16599 v1 pith:RXNDWAJV submitted 2024-12-21 cs.AI

Hang Yin , Zhifeng Lin , Xin Liu , Bin Sun , Kan Li This is my paper · ORCID

classification cs.AI

keywords compassdirectionreasoningmultimodallanguagemodelspatialbenchmarkchain-of-thoughtfine-tuningLLaVAevaluationvisualquestionansweringdataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces the Compass Direction Reasoning (CDR) benchmark, a set of synthetic grid images and multiple-choice questions that test whether multimodal language models can map visual orientation to spatial (up/down/left/right) and compass (north/south/east/west) directions. Across six models, object recognition is strong but direction reasoning is weak: on relative compass tasks most models score near the 12.5% random-guessing baseline, far below the paper's assumed 100% human reference. The paper then shows that fine-tuning LLaVA-7B directly on CDR alone does not teach the rule, but mixing out-of-domain data with roughly a 2:1 ratio and adding chain-of-thought reasoning answers raises relative compass accuracy from 11.90% to 53.43%. The upshot is a demonstrated gap: current models have not internalized real-world compass rules, and step-by-step training with diverse data is a partial remedy.

What carries the argument

The load-bearing object is the CDR benchmark's question taxonomy, especially the Relative Compass Reasoning task. Images are 200x200 canvases with a 3x3 grid; icon, letter, and number stimuli place one central element and surrounding elements, and questions specify a coordinate convention (e.g., 'upward is West') or the central element's heading to force the model to rotate image-relative positions into compass labels. The mapping rule the paper exploits is the rotation between image axes and compass axes: once the model knows the transform from one spatial direction to one compass direction, all eight directions follow. Chain-of-thought fine-tuning works, in the paper's account, because it makes the model verbalize this rotation step-by-step instead of guessing the multiple-choice label directly.

What would settle it

Have a group of human annotators answer the same relative-compass questions and measure their agreement with the automatically generated labels; if agreement is far from perfect, the benchmark itself is ambiguous and the model scores cannot be interpreted as a pure measure of compass reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that multimodal language models can classify icons and describe spatial layout while failing to reason about compass directions that require mapping image-relative orientation onto geographic labels. On the Relative Compass Reasoning (icon) task, every evaluated model performs near or below random guessing, with GPT-4o-mini at 18.65% and the best, Claude-3-Sonnet, at 19.46%. Directly fine-tuning LLaVA-7B on 71,552 CDR relative-compass samples yields only 11.90%, showing that memorizing question-answer pairs is not enough; the model must acquire an abstract orientation rule. Adding 40,000 random out-of-domain samples lifts accuracy to 31.05%, and replacing the CDR portion with chain-of-thought answers that spell out the mapping from image directions to compass directions lifts it further to 53.43% when combined with the same 40K mix. The paper reads this as evidence that explicit step-by-step reasoning about orientation relationships, rather than more in-domain data, is what lets a smaller model begin to master compass reasoning.

Load-bearing premise

The entire evaluation rests on the assumption that the synthetic grid questions are an unambiguous measure of compass-direction reasoning; the paper does not report human agreement or ambiguity checks, so if the rotated-coordinate wording or icon orientations are genuinely confusing, the low model scores would be artifacts rather than evidence of missing direction understanding.

Editorial extensions

If this is right

Compass direction reasoning is not captured by existing spatial-reasoning benchmarks: high spatial accuracy does not transfer to compass accuracy.
Training on a large in-domain direction dataset alone can leave a model at chance, so accuracy on direction tasks is not simply a data-volume problem.
A mix of roughly two parts in-domain data to one part out-of-domain data gives the best generalization among the mixtures tested, with larger mixes degrading performance.
Chain-of-thought supervision that verbalizes the spatial-to-compass mapping is a more effective training signal than the same questions with single-label answers.
State-of-the-art closed and open models tested all fall far below human-level compass reasoning, so real-world navigation and orientation tasks remain open problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Going beyond the paper, a natural next test is whether the 53.43% result transfers to real photographs or map layouts; grid-based icons may make the learned rule easier to verbalize than continuous real-world scenes.
Going beyond the paper, the paper's implied curriculum predicts that models with stronger spatial reasoning should improve more on CDR after chain-of-thought training, a correlation that could be tested across model families.
Going beyond the paper, if the 2:1 mix ratio is the real mechanism, the dominant factor might be regularization rather than world knowledge; an ablation holding total data fixed while varying the out-of-domain fraction would separate those accounts.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Worth a look: this paper does something nobody has done cleanly before—benchmarking compass direction reasoning in multimodal models, as opposed to the usual relative spatial reasoning tasks. The task family (rotated coordinate systems, absolute and relative compass directions) is simple but effective, and the core empirical observation is believable: object classification is near-perfect, yet on relative compass reasoning most models sit at or near the 12.5% chance level. The letter and number variants rule out the worry that it's just an icon-labeling artifact, since those also sit near chance. That is a genuinely useful result for anyone building navigation or embodied agents.

The fine-tuning story is more incremental but still interesting: mixing out-of-domain data at a 2:1 ratio and adding hand-written CoT answers lifts LLaVA-7B from 11.90% to 53.43% on the icon task. The CoT teaching the explicit mapping rule is a reasonable mechanism, not a circularity, since the test split is held out.

Now the soft spots. The human baseline of 100% is asserted in Figure 1 with no protocol—no sample size, no participants, no ambiguity checks. At minimum they should report how it was measured; otherwise the 'random vs. human' framing is unverifiable. Second, the icon orientations are manually annotated from internet images with no inter-annotator agreement. The letter/number tasks mitigate this, but the headline fine-tuning result is on icons, so label noise could inflate or deflate the improvement. Third, there's a clear inconsistency: Section III-D says training on All gives 11.90%, but Table III lists the 7B Base as 11.90% and All as 10.69%. Small, but it erodes trust. Fourth, the dataset is not released, which limits the benchmark's immediate utility. No error bars on the fine-tuning numbers, though temperature 0 and a fixed test set reduce variance.

Overall, the central claim holds up in its broad strokes—models do fail at compass reasoning—but the paper needs a real human baseline, a dataset release, and a fix for the numbers before I'd treat the fine-tuning gains as established.

Recommendation: send it to review. The benchmark idea is good, the finding is publishable, and a good referee can force the authors to close the validity gaps. My own verdict would be conditional acceptance after those fixes.

Referee Report

4 major / 5 minor

Summary. The paper proposes the Compass Direction Reasoning (CDR) benchmark for evaluating multimodal language models on spatial and compass direction reasoning. CDR contains three image modalities (icons, letters, numbers) and seven question types spanning object classification, absolute spatial/compass direction, and relative spatial/compass reasoning. The authors evaluate six MLMs zero-shot and report that most perform near or below random guessing (12.5%) on the relative compass reasoning tasks; for example, LLaVA-7B achieves 11.90% on Relative Compass Reasoning (icon). They then fine-tune LLaVA-7B on CDR data, alone and mixed with out-of-domain data, and with chain-of-thought (CoT) answers, reporting that a combination of CoT and mixed data raises accuracy to 53.43%. The central claim is that current MLMs lack robust compass direction reasoning and that mixdata plus explicit step-by-step reasoning is a partial remedy.

Significance. If the CDR benchmark is valid and made publicly available, it addresses a genuine gap: existing spatial reasoning benchmarks mostly test relative positions within images rather than mapping those positions to real-world compass directions. The finding that six commercial and open models perform at or near chance on relative compass reasoning is a useful, falsifiable result, and the fine-tuning study provides an actionable intervention (CoT with mixed data) that improves accuracy from near chance to 53.43% on the icon subtask. The paper's strengths include a large synthetic dataset with balanced answer distributions, systematic task progression, and controlled fine-tuning comparisons. However, the significance is currently conditional on validating the benchmark's low-ambiguity assumption and on releasing the dataset; without human-level validation and public data, the empirical claims cannot be fully assessed or reused by the community.

major comments (4)

[Section II-A and Figure 1] The human baseline of 100% accuracy in Figure 1 is asserted without any description of an experimental protocol: no number of participants, task instructions, or reporting procedure is given. Because the central claim that MLMs perform 'at random guessing levels' is meaningful only if the benchmark questions are trivially unambiguous for humans, the paper must include a proper human-subject evaluation with several annotators, report their accuracy, and report inter-annotator agreement on the direction labels. Without this, the low model scores could partly reflect task ambiguity rather than missing compass-direction reasoning.
[Section II-A] The icon images are manually annotated for orientation into eight compass bins, but the manuscript provides no ambiguity checks, no inter-annotator agreement, and no statistics about the angles of the icons relative to the bin boundaries. Many icons (e.g., a pointing finger, a car) may not point exactly along one of the eight directions, and the discretization could introduce label noise. This is load-bearing because the headline fine-tuning result (53.43%) is measured on the icon subset, and the near-chance accuracy of LLaVA-7B (11.90%) could be an artifact of noisy or ambiguous labels. The authors should report the distribution of annotated angles, an agreement measure, and ideally re-run the key evaluations after excluding icons whose orientation is ambiguous.
[Data availability] The CDR benchmark is the main contribution of the paper, yet the manuscript does not state whether the dataset, annotation files, fine-tuning code, or evaluation scripts will be released. For a benchmark paper, public availability is essential for reproducibility and community adoption. The authors should provide a clear data-release statement, including a URL or hosting plan, and specify any licensing restrictions (especially for the icon images sourced from Iconfont).
[Section III-D and Table III] There is a direct inconsistency between the text and Table III. The text says 'Using the all Relative Compass Reasoning training data in (71552 samples), the model only get 11.90% accuracy', but Table III lists '7B Base' as 11.90% and 'All' as 10.69%. The correct reading is that fine-tuning on the full CDR data yields no improvement over the base model (indeed a slight drop). This misreporting should be corrected, and the narrative around 'limited improvements' should be aligned with the actual numbers. In addition, the fine-tuning results appear to be single runs with no variance reported; please provide results over multiple seeds or at least state the number of runs and observed variance.

minor comments (5)

[Throughout] There are several typos and formatting issues: 'wu et al.' should be 'Wu et al.', 'llava v1 5 mix665k.json' should be 'LLaVA-v1.5-mix665k.json', 'V A' spacing in LLaVA appears inconsistently, and the number-type example in Table I contains 'a a Cartesian coordinate system'. Please proofread the manuscript carefully.
[Figure 2] Figure 2 contains corrupted labels such as '/glyph1197ortheast' and '/glyph1197orth' that appear to be PDF-extraction artifacts. The figure should be regenerated with correct text rendering.
[Table III] The column headers in Table III are ambiguous: '7B Base All' is a list of three separate conditions (7B Base, All, All+20K, etc.) but is presented as a single row label. Please make the table structure clearer, for example by separating the base model column from the fine-tuning conditions.
[Section III-D] The claim that All+40K corresponds to an 'optimal mix ratio (approximately 2:1)' is not substantiated: 71,552 CDR samples plus 40,000 mixdata samples gives roughly 1.8:1, and the selection of 40K is empirically driven without a statistical test. Please clarify the ratio and discuss whether the difference from All+20K or All+60K is significant.
[Section IV] The conclusion states that CDR contains 'over 100K training and testing samples', but the statistics in Section II-B sum to roughly 50K+50K+71K training and 4K+4K+4K+5K testing for the different tasks; the total should be stated per task to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CDR is an empirical benchmark and the fine-tuning gains are measured on held-out test splits; the sole self-citation is not load-bearing.

full rationale

The paper's central claims are empirical evaluations rather than derivations from first principles, so there is no fitted-value circularity or self-definitional reduction. CDR questions are constructed from recorded grid positions and manual icon-orientation labels, and model accuracies are computed on test splits, not on the training data used for fine-tuning. In Section III-D, the improvement from 11.90% to 53.43% on Relative Compass Reasoning (icon) is a held-out test measurement after training on CDR training subsets, so it is not a prediction forced by construction. The only author self-citation, reference [14] (MODE), appears in a general list of vision-language capabilities and plays no role in the benchmark design or the conclusions. The paper's unsupported assertion of a 100% human baseline and the lack of inter-annotator agreement are benchmark-validity concerns, not circularity: an ambiguous benchmark could lower scores without making the paper's claims definitionally equivalent to its inputs. No step in the paper reduces a stated result to its own assumptions by equation or self-citation chain.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No hidden physical constants or invented entities are introduced. The main assumptions are about task validity and the coordinate-to-compass mapping, while the only fitted quantity is the fine-tuning mix ratio selected on the test set.

free parameters (1)

mix data ratio for fine-tuning (All+40K) = approximately 2:1; 40K out-of-domain samples with the full CDR Relative Compass training set
The paper selects All+40K as best performing on the test set (31.05% vs 20.87% for All+20K and 24.50% for All+60K) and uses it as the basis for CoTAll+40K; no separate validation set is described, so the reported best result depends on this test-set selection.

assumptions (4)

standard math Cartesian coordinate system on the plane is used to define relative spatial and compass directions.
Assumed background for defining relative positions and orientations in the questions; standard mathematical framework.
domain assumption Spatial reasoning is the foundation for compass reasoning, based on human cognitive principles.
Motivates the task hierarchy in the Introduction but is not independently established or tested in this paper.
domain assumption The synthetic grid images are simple, intuitive, and low-ambiguity, and manual annotations of icon orientation are correct.
Section II-A describes manual annotation and low-ambiguity image selection without reporting ambiguity checks or inter-annotator agreement; benchmark validity rests on this assumption.
domain assumption The specified upward-to-compass mapping mirrors real-world direction rules.
Used in Compass and Relative Compass tasks; if this mapping is arbitrary or inconsistently applied, the results do not measure real-world compass reasoning.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Do Multimodal Language Models Really Understand Direction? A Benchmark for Compass Direction Reasoning." pith.science (2026). https://pith.science/paper/RXNDWAJV

@misc{pith2026241216599,
  author       = {Pith},
  title        = {Pith review of: Do Multimodal Language Models Really Understand Direction? A Benchmark for Compass Direction Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RXNDWAJV}},
  note         = {Machine review of arXiv:2412.16599}
}

read the original abstract

Direction reasoning is essential for intelligent systems to understand the real world. While existing work focuses primarily on spatial reasoning, compass direction reasoning remains underexplored. To address this, we propose the Compass Direction Reasoning (CDR) benchmark, designed to evaluate the direction reasoning capabilities of multimodal language models (MLMs). CDR includes three types images to test spatial (up, down, left, right) and compass (north, south, east, west) directions. Our evaluation reveals that most MLMs struggle with direction reasoning, often performing at random guessing levels. Experiments show that training directly with CDR data yields limited improvements, as it requires an understanding of real-world physical rules. We explore the impact of mixdata and CoT fine-tuning methods, which significantly enhance MLM performance in compass direction reasoning by incorporating diverse data and step-by-step reasoning, improving the model's ability to understand direction relationships.

Figures

Figures reproduced from arXiv: 2412.16599 by the authors.

**Figure 2.** Distribution of CDR answers in spatial and compass reasoning tasks. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 15 canonical work pages

[1]

World models,

D. Ha and J. Schmidhuber, “World models,” arXiv preprint arXiv:1803.10122, 2018

arXiv 2018
[2]

Reasoning with language model is planning with world model,

S. Hao, Y . Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu, “Reasoning with language model is planning with world model,” arXiv preprint arXiv:2305.14992, 2023

arXiv 2023
[3]

World model on million-length video and language with ringattention,

H. Liu, W. Yan, M. Zaharia, and P. Abbeel, “World model on million-length video and language with ringattention,” arXiv preprint arXiv:2402.08268, 2024

arXiv 2024
[4]

Penetrative ai: Making llms comprehend the physical world,

H. Xu, L. Han, Q. Yang, M. Li, and M. Srivastava, “Penetrative ai: Making llms comprehend the physical world,” in Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications, 2024, pp. 1–7

work page 2024
[5]

Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts,

Z. Shi, Q. Zhang, and A. Lipani, “Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts,” in Proc. Conf. Artif. Intell. (AAAI). AAAI Press, 2022, pp. 11 321–11 329

work page 2022
[6]

Eval- uating spatial understanding of large language models,

Y . Yamada, Y . Bao, A. K. Lampinen, J. Kasai, and I. Yildirim, “Eval- uating spatial understanding of large language models,” Trans. Mach. Learn. Res., vol. 2024, 2024

work page 2024
[7]

Spartqa:: A textual question answering benchmark for spatial reasoning,

R. Mirzaee, H. R. Faghihi, Q. Ning, and P. Kordjmashidi, “Spartqa:: A textual question answering benchmark for spatial reasoning,” arXiv preprint arXiv:2104.05832, 2021

arXiv 2021
[8]

Can large language models be good path planners? A benchmark and investigation on spatial-temporal reasoning,

M. Aghzal, E. Plaku, and Z. Yao, “Can large language models be good path planners? A benchmark and investigation on spatial-temporal reasoning,” CoRR, vol. abs/2310.03249, 2023

arXiv 2023

Show all 27 references

[9]

Advancing spatial reasoning in large language models: An in-depth evaluation and enhancement using the stepgame benchmark,

F. Li, D. C. Hogg, and A. G. Cohn, “Advancing spatial reasoning in large language models: An in-depth evaluation and enhancement using the stepgame benchmark,” in Proc. Conf. Artif. Intell. (AAAI) . AAAI Press, 2024, pp. 18 500–18 507

2024
[10]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

2023 arXiv
[11]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

2021
[12]

Scaling up vision-language pre-training for image captioning,

X. Hu, Z. Gan, J. Wang, Z. Yang, Z. Liu, Y . Lu, and L. Wang, “Scaling up vision-language pre-training for image captioning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2022, pp. 17 980–17 989

2022
[13]

Visual instruction tuning,

H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , 2023

2023
[14]

MODE: a multimodal open-domain dialogue dataset with explanation,

H. Yin, P. Lu, Z. Li, B. Sun, and K. Li, “MODE: a multimodal open-domain dialogue dataset with explanation,” Appl. Intell. , vol. 54, no. 7, pp. 5891–5906, 2024. [Online]. Available: https://doi.org/10. 1007/s10489-024-05479-x

2024
[15]

Canny, The complexity of robot motion planning

J. Canny, The complexity of robot motion planning . MIT press, 1988

1988
[16]

S. M. LaValle, Planning algorithms. Cambridge university press, 2006

2006
[17]

Intelligent control and decision- making demonstrated on a simple compass-guided robot,

L. S. Lopes, N. Lau, and L. P. Reis, “Intelligent control and decision- making demonstrated on a simple compass-guided robot,” in Smc 2000 conference proceedings. 2000 ieee international conference on systems, man and cybernetics. ’cybernetics evolving to systems, humans, orga...

2000
[18]

Plasticity of human spatial cognition: Spatial language and cognition covary across cultures,

D. B. Haun, C. J. Rapold, G. Janzen, and S. C. Levinson, “Plasticity of human spatial cognition: Spatial language and cognition covary across cultures,” Cognition, vol. 119, no. 1, pp. 70–80, 2011

2011
[19]

S. C. Levinson, Space in language and cognition: Explorations in cognitive diversity. Cambridge University Press, 2003, vol. 5

2003
[20]

Mind’s eye of llms: Visualization-of-thought elicits spatial reasoning in large language models,

W. Wu, S. Mao, Y . Zhang, Y . Xia, L. Dong, L. Cui, and F. Wei, “Mind’s eye of llms: Visualization-of-thought elicits spatial reasoning in large language models,” arXiv preprint cs.CL/2404.03622 , 2024

2024 arXiv
[21]

Visual spatial reasoning,

F. Liu, G. Emerson, and N. Collier, “Visual spatial reasoning,” Trans- actions of the Association for Computational Linguistics , vol. 11, pp. 635–651, 2023

2023
[22]

What’s” up

A. Kamath, J. Hessel, and K.-W. Chang, “What’s” up” with vision- language models? investigating their struggle with spatial reasoning,” arXiv preprint arXiv:2310.19785 , 2023

2023 arXiv
[23]

Improved baselines with visual instruction tuning,

H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 26 296–26 306

2024
[24]

The claude 3 model family: Opus, sonnet, haiku,

A. Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” Claude-3 Model Card , vol. 1, 2024

2024
[25]

Gpt-4o mini: A smaller, cheaper ai model,

OpenAI, “Gpt-4o mini: A smaller, cheaper ai model,” 2024, accessed: 2024-08-28. [Online]. Available: https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/

2024
[26]

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser et al. , “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530 , 2024

2024 arXiv
[27]

Ali iconfont,

Iconfont, “Ali iconfont,” 2024, accessed: 2024-09-02. [Online]. Available: https://www.iconfont.cn/

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

World models,

D. Ha and J. Schmidhuber, “World models,” arXiv preprint arXiv:1803.10122, 2018

arXiv 2018

[2] [2]

Reasoning with language model is planning with world model,

S. Hao, Y . Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu, “Reasoning with language model is planning with world model,” arXiv preprint arXiv:2305.14992, 2023

arXiv 2023

[3] [3]

World model on million-length video and language with ringattention,

H. Liu, W. Yan, M. Zaharia, and P. Abbeel, “World model on million-length video and language with ringattention,” arXiv preprint arXiv:2402.08268, 2024

arXiv 2024

[4] [4]

Penetrative ai: Making llms comprehend the physical world,

H. Xu, L. Han, Q. Yang, M. Li, and M. Srivastava, “Penetrative ai: Making llms comprehend the physical world,” in Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications, 2024, pp. 1–7

work page 2024

[5] [5]

Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts,

Z. Shi, Q. Zhang, and A. Lipani, “Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts,” in Proc. Conf. Artif. Intell. (AAAI). AAAI Press, 2022, pp. 11 321–11 329

work page 2022

[6] [6]

Eval- uating spatial understanding of large language models,

Y . Yamada, Y . Bao, A. K. Lampinen, J. Kasai, and I. Yildirim, “Eval- uating spatial understanding of large language models,” Trans. Mach. Learn. Res., vol. 2024, 2024

work page 2024

[7] [7]

Spartqa:: A textual question answering benchmark for spatial reasoning,

R. Mirzaee, H. R. Faghihi, Q. Ning, and P. Kordjmashidi, “Spartqa:: A textual question answering benchmark for spatial reasoning,” arXiv preprint arXiv:2104.05832, 2021

arXiv 2021

[8] [8]

Can large language models be good path planners? A benchmark and investigation on spatial-temporal reasoning,

M. Aghzal, E. Plaku, and Z. Yao, “Can large language models be good path planners? A benchmark and investigation on spatial-temporal reasoning,” CoRR, vol. abs/2310.03249, 2023

arXiv 2023