Pith. sign in

REVIEW 4 major objections 4 minor 58 references

RoadBench: A benchmark built on road markings shows multimodal models fail fine-grained spatial reasoning under urban road scenarios.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:48 UTC pith:ITL76X73

load-bearing objection RoadBench fills a real gap in evaluating MLLMs on fine-grained road-marking spatial reasoning, but the paper needs cleanup and the unvalidated external labels deserve scrutiny before the headline claim is taken on faith. the 4 major comments →

arxiv 2511.18011 v2 pith:ITL76X73 submitted 2025-11-22 cs.CV

RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios

Jun Zhang , Xin Zhang , Jie Feng , Long Chen , Junhui Wang , Zhicheng Liu , Depeng Jin , Yong Li This is my paper
classification cs.CV
keywords road markingsfine-grained spatial understandingmultimodal large language modelsbenchmarkbird's-eye viewfirst-person viewlane countingurban scenarios
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes RoadBench, a benchmark of 9,121 hand-verified test cases that probes multimodal large language models (MLLMs) on fine-grained spatial understanding and reasoning in urban road scenes, using road markings as the central test object. The six tasks range from local lane counting and lane-designation recognition on satellite (bird's-eye view) and dash-cam (first-person view) images to global road-network correction and road-type classification. Evaluating 14 mainstream MLLMs, the authors find that the best models achieve low F1 scores on lane counting, none beat a traffic-commonsense rule-based baseline on lane designation recognition, and several fall below random-choice baselines. The paper argues this shows current MLLMs lack the fine-grained spatial perception and domain-knowledge integration needed for real-world urban applications such as HD map generation and autonomous driving.

Core claim

RoadBench is presented as the first benchmark that centers on road markings—thin, elongated, interdependent spatial elements—as the unit of evaluation. It comprises two viewpoints (BEV satellite and FPV vehicle-camera images), six tasks organized by spatial scope, and 9,121 manually verified test cases from multiple Chinese cities. On it, 14 closed- and open-source MLLMs are tested; the central finding is that all models underperform on fine-grained road-marking tasks, with the best lane-counting F1 score at 0.369 and no model exceeding a rule-based lane-designation baseline (which achieves Hamming Loss 0.141 vs best model 0.152). The authors conclude that MLLMs fall short of simple non-visu

What carries the argument

The central object is the road marking system—lane dividers, directional arrows, and related painted symbols—treated as fine-grained spatial elements with strong inter-dependency. The benchmark's mechanism is a hierarchy of six tasks that escalate from local recognition (lane counting, lane designation) to global reasoning (road-network correction, road-type classification), requiring the model to integrate image content with traffic domain knowledge. Evaluation uses multi-class metrics (Precision/Recall/F1, RMSE), Hamming Loss and accuracy for multi-label lane designations, and RMSE and Fréchet distance for geometric outputs, with rule-based and random baselines as reference points.

Load-bearing premise

The ground-truth labels (lane counts, designations, junction positions, road types) are taken from a commercial map provider's database without independent re-labeling; if that database contains systematic errors—stale lane counts, wrong junction topology, or coordinate mismatches—the reported model scores and the 'rule-based beats MLLMs' conclusion may not generalize.

What would settle it

Independently re-annotate a random sample (say 300 cases) of RoadBench with a second set of human labelers and compare against the provided ground truth; if the disagreement rate on lane counts or junction positions exceeds the gap between the best MLLM and the rule-based baseline, the headline conclusion collapses. Alternatively, run the same six tasks on images from left-hand-traffic countries; if MLLM scores improve dramatically there, the failures are specific to China's marking conventions rather than general fine-grained spatial ability.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If RoadBench is a fair instrument, then existing MLLMs cannot be trusted for fine-grained spatial annotations in HD-map generation or end-to-end driving pipelines without substantial improvement.
  • The benchmark separates perceptual failure (reading thin markings in BEV images) from reasoning failure (integrating domain knowledge such as bidirectional-road conventions), giving developers a diagnostic ladder.
  • The result that a rule-based, image-free baseline beats MLLMs on lane-designation recognition implies that models have not yet absorbed basic traffic-layout common sense, pointing to a concrete training-data gap.
  • The coordinate-output task shows MLLMs produce geometrically unreliable junction points and polylines, suggesting structured numeric output and spatial grounding need dedicated treatment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial note: the provided abstract states eight tasks and 3,040 test cases, while the body, tables, and conclusion state six tasks and 9,121 test cases; this internal inconsistency should be reconciled before the benchmark is cited as a fixed resource.
  • Because all images come from five Chinese cities with right-hand traffic, the benchmark's baselines and difficulty may not transfer directly to left-hand-traffic regions or cities with different marking conventions; a left-hand-traffic extension would test whether the failures are perceptual or convention-dependent.
  • The quality-control stage rejects unrecognizable cases instead of re-labeling them, so the benchmark measures performance on human-legible images; an adversarial evaluation with harder, partially occluded cases might produce even lower model scores.
  • A targeted probe—varying image resolution or adding synthetic occlusions—could determine whether the BEV weakness is a resolution artifact or a structural reasoning limit, informing whether better cameras or better models are the fix.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces RoadBench, a benchmark for evaluating MLLMs on fine-grained spatial understanding and reasoning about road markings in urban settings. Six tasks are proposed: BEV lane counting, BEV lane designation recognition, BEV road network correction, FPV lane counting, FPV lane designation recognition, and FPV road type classification, with test cases sourced from satellite imagery and in-vehicle camera images. Labels are derived from an online map service database (Amap), with human proofreading and rejection of ambiguous cases. The authors evaluate 14 MLLMs on these tasks and report that performance is generally low, and that in several tasks MLLMs underperform simple rule-based or random baselines. The paper also includes analyses of prompt-format effects, environmental conditions, and failure cases.

Significance. If the empirical claims hold, RoadBench addresses a genuine gap in spatial benchmarks for MLLMs: existing urban benchmarks mostly target whole-image understanding or isolated objects, whereas RoadBench focuses on fine-grained, interdependent road markings and tasks that scale from local perception to global reasoning. The benchmark design is thoughtful, the task set is systematic, and the inclusion of rule-based baselines is a useful diagnostic. The paper also provides detailed per-task results, prompt templates, and failure-case analyses, which strengthen its value as a resource. However, the current manuscript contains internal inconsistencies in the benchmark statistics and does not provide sufficient validation of the external ground-truth labels. These issues are load-bearing because the central conclusion—that MLLMs are weak at fine-grained urban spatial reasoning and even fall below simple baselines—depends directly on the correctness of the labels and on the reliability of the reported numbers.

major comments (4)
  1. [Abstract versus Sections 3.1 and 4.1] The paper's metadata-level abstract states that RoadBench comprises eight tasks, 3,040 test cases, and 20 evaluated MLLMs, while the body abstract, Section 3.1, and Section 4.1 consistently describe six tasks, 9,121 test cases, and 14 evaluated MLLMs. The contribution size and the set of evaluated models are central to the benchmark claim, so this contradiction must be resolved and the correct values reported consistently in the camera-ready version.
  2. [Section 3.3 (Quality Control) and Tables 2/71-76] The ground-truth labels for lane counts, lane designations, junction positions, and road types are taken from the Amap database without independent verification. The quality-control stage rejects erroneous or unrecognizable cases rather than re-labeling them, and the paper reports no inter-annotator agreement, no rejection statistics, and no holdout re-labeling check. If the external map database contains systematic errors—outdated lane counts, incorrect junction topology, or coordinate misalignment—the reported model scores and, in particular, the conclusion that rule-based baselines beat MLLMs could be artifacts of label noise. The authors should provide a validation protocol, e.g., re-annotating a random sample by independent annotators and reporting agreement, along with rejection rates and any coordinate-matching error analysis.
  3. [Section 4.2 and Tables 72/75] The claim that 'none of the MLLMs can outperform the baseline' in lane designation rests on very small margins in some cases. For example, in the FPV lane designation task, the rule-based baseline has Hamming Loss 0.1282 and accuracy 0.6019, while GPT-5 has 0.1287 and 0.5932, and GPT-5-Mini has 0.1301 and 0.5941. These differences could lie within sampling variability. No confidence intervals, bootstrap estimates, or significance tests are reported for model-versus-baseline comparisons. The authors should report per-example bootstrap intervals or an appropriate paired test to support the strong wording that MLLMs 'fall short of' simple baselines.
  4. [Appendix Table 74 and Table 2] Table 74 lists F1 = 0.0000 for LLaMA-3.2-11B-Vision and LLaMA-3.2-90B-Vision in FPV lane counting, despite positive Precision and Recall values and despite Table 2 reporting F1 = 0.273 and F1 = 0.305 for those same models. Since F1 cannot be zero when both precision and recall are positive, these entries are numerically inconsistent. The authors need to correct the appendix table and audit the underlying computation.
minor comments (4)
  1. [Section 4.1] The phrase 'randomly select two lanes from {2,3,4}' should read 'randomly select a lane count from {2,3,4}'.
  2. [Section 8.1 / Table 810] The text says adverse lighting conditions caused only a 0.2%–1.9% drop for GPT-5, but Table 810 shows F1 improving by 0.2% and accuracy dropping by 0.8%. Please express changes consistently with respect to metric direction (higher is better for F1/accuracy, lower is better for RMSE/Hamming loss).
  3. [Table 1] The use of star symbols and cell placeholders in Table 1 makes the comparison hard to read. Consider replacing with textual descriptors or a cleaner symbol legend.
  4. [General] No URL or release plan for the benchmark data and evaluation code is provided. Since the benchmark is a main contribution, a public release link should be included in the final version.

Circularity Check

0 steps flagged

No significant circularity: benchmark conclusions rest on external labels and hand-coded baselines, not on self-referential derivation.

full rationale

RoadBench's central claim is an empirical benchmark result, not a derivation from first principles. The label-generation chain is: satellite/FPV images are matched to an external map-provider database (Amap) for lane counts, designations, junctions, and road types; reference lines come from OpenStreetMap; manual proofreading rejects rather than relabels ambiguous cases; then 14 MLLMs are prompted and scored against these external labels, with hand-coded rule/random baselines for comparison. None of these steps define the target quantity in terms of the MLLM outputs, and no parameter is fitted that is later reported as a prediction. The rule-based lane-designation baseline is a generic traffic-common-sense mapping from the already-provided lane count, not a fitted model, and it does not guarantee the Amap ground truth (its accuracy is only about 0.60), so the finding that MLLMs underperform it is not forced by construction. The authors' prior benchmarks are cited only as related work and gap justification; no load-bearing result or uniqueness theorem is imported from those self-citations. The abstract/body inconsistencies and the absence of independent Amap-label validation are real data-quality and presentation concerns, but they do not make the benchmark's conclusions circular.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The evaluation rests on external map data (ground truth), manual QC, and hand-designed baselines; the benchmark introduces no free parameters in a scientific model, but the evaluation protocol contains hand-chosen thresholds and a restricted baseline answer set.

free parameters (2)
  • Evaluation distance thresholds for road network correction = RMSE@20%, FD@50% reported in main table; 10%/20%/50% tried
    Thresholds chosen post hoc from {10%,20%,50%} to 'select the appropriate threshold', affecting headline numbers.
  • Random lane-count baseline support set {2,3,4} = uniform over {2,3,4}
    Hand-chosen answer set; actual lane counts include 5 (see failure cases), so the random baseline is evaluated against a distribution that omits a valid label.
axioms (4)
  • domain assumption Ground truth from Amap/OSM databases is accurate
    All tasks' labels and reference network data are taken from a commercial map provider without independent verification; see Section 3.3.
  • domain assumption Manual proofreading implies human solvability and correctness
    Section 4.1 uses absence of human baseline on the grounds that QC retained only human-identifiable cases.
  • domain assumption Chinese traffic conventions: right-hand driving, left-to-right lane ordering, lane designation rule table
    Prompts and the rule-based baseline in Section 7.2 encode these; applies to all Chinese-city images.
  • domain assumption Evaluation metrics (Hamming loss, RMSE with nearest-neighbor matching, Fréchet distance) capture the intended capability
    Metric choice in Section 4.1 affects headline rankings; no validation that e.g., F1/HL align with human judgment beyond the asserted label accuracy.

pith-pipeline@v1.3.0-alltime-deepseek · 25201 in / 14348 out tokens · 142447 ms · 2026-08-03T20:48:33.485053+00:00 · methodology

0 comments
read the original abstract

Multimodal large language models (MLLMs) have demonstrated powerful capabilities in general spatial understanding and reasoning. However, their fine-grained spatial understanding and reasoning capabilities in complex urban scenarios have not received significant attention in the fields of both research and industry. To fill this gap, we focus primarily on road markings as a typical example of fine-grained spatial elements under urban scenarios, given the essential role of the integrated road traffic network they form within cities. Around road markings and urban traffic systems, we propose \textbf{RoadBench}, a systematic benchmark that comprehensively evaluates MLLMs' fine-grained spatial understanding and reasoning capabilities using Bird's-Eye View (BEV) and First-Person View (FPV) image inputs. This benchmark comprises eight tasks consisting of 3,040 strictly manually verified test cases, constructed from 2,137 unique BEV images and 721 unique FPV images collected from five Chinese cities with relatively consistent traffic conventions. These tasks form a systematic evaluation framework that bridges understanding at local spatial scopes to global reasoning. They not only test MLLMs' capabilities in recognition, joint understanding, and reasoning but also assess their ability to integrate image information with domain knowledge. After evaluating 20 mainstream MLLMs, we confirm that RoadBench is a challenging benchmark for MLLMs while revealing significant shortcomings in existing MLLMs' fine-grained spatial understanding and reasoning capabilities within urban scenarios. In certain tasks, their performance even falls short of simple rule-based or random selection baselines. These findings, along with RoadBench itself, will contribute to the comprehensive advancement of spatial understanding capabilities for MLLMs.

Figures

Figures reproduced from arXiv: 2511.18011 by Depeng Jin, Jie Feng, Junhui Wang, Jun Zhang, Long Chen, Xin Zhang, Yong Li, Zhicheng Liu.

Figure 1
Figure 1. Figure 1: Examples of road markings in BEV and FPV images with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overview of RoadBench [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The RoadBench curation pipelines to construct the datasets for the six benchmark tasks. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 61
Figure 61. Figure 61: An example image input to MLLM in the BEV lane [PITH_FULL_IMAGE:figures/full_fig_p012_61.png] view at source ↗
Figure 62
Figure 62. Figure 62: An example image input to MLLM in the BEV lane [PITH_FULL_IMAGE:figures/full_fig_p013_62.png] view at source ↗
Figure 63
Figure 63. Figure 63: An example image input to MLLM in the BEV road [PITH_FULL_IMAGE:figures/full_fig_p014_63.png] view at source ↗
Figure 64
Figure 64. Figure 64: An example image input to MLLM in the FPV lane [PITH_FULL_IMAGE:figures/full_fig_p015_64.png] view at source ↗
Figure 65
Figure 65. Figure 65: An example image input to MLLM in the FPV lane [PITH_FULL_IMAGE:figures/full_fig_p016_65.png] view at source ↗
Figure 66
Figure 66. Figure 66: An example image input to MLLM in the FPV road [PITH_FULL_IMAGE:figures/full_fig_p017_66.png] view at source ↗
Figure 97
Figure 97. Figure 97: Example images of BEV lane counting failure cases. [PITH_FULL_IMAGE:figures/full_fig_p020_97.png] view at source ↗
Figure 98
Figure 98. Figure 98: Example images of BEV lane designation failure cases. [PITH_FULL_IMAGE:figures/full_fig_p021_98.png] view at source ↗
Figure 99
Figure 99. Figure 99: Example images of BEV road network correction failure [PITH_FULL_IMAGE:figures/full_fig_p022_99.png] view at source ↗
Figure 910
Figure 910. Figure 910: Example images of FPV lane counting failure cases. [PITH_FULL_IMAGE:figures/full_fig_p023_910.png] view at source ↗
Figure 911
Figure 911. Figure 911: Example images of FPV lane designation failure cases. [PITH_FULL_IMAGE:figures/full_fig_p024_911.png] view at source ↗
Figure 912
Figure 912. Figure 912: Example images of FPV road type classification failure [PITH_FULL_IMAGE:figures/full_fig_p025_912.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 13 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 1, 2

    Pith/arXiv arXiv 2023

  2. [2]

    Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025

    Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025. 2

    Pith/arXiv arXiv 2025

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1, 2, 6

    Pith/arXiv arXiv 2025

  4. [4]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020. 2

    1901

  5. [5]

    Driving by the rules: A benchmark for integrating traffic sign regulations into vectorized hd map

    Xinyuan Chang, Maixuan Xue, Xinran Liu, Zheng Pan, and Xing Wei. Driving by the rules: A benchmark for integrating traffic sign regulations into vectorized hd map. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 6823–6833, 2025. 3, 4

    2025

  6. [6]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024. 2

    2024

  7. [7]

    Spatialllm: From multi- modality data to urban spatial intelligence.arXiv preprint arXiv:2505.12703, 2025

    Jiabin Chen, Haiping Wang, Jinpeng Li, Yuan Liu, Zhen Dong, and Bisheng Yang. Spatialllm: From multi- modality data to urban spatial intelligence.arXiv preprint arXiv:2505.12703, 2025. 3

    Pith/arXiv arXiv 2025

  8. [8]

    Spa- tialrgpt: Grounded spatial reasoning in vision-language mod- els.Advances in Neural Information Processing Systems, 37: 135062–135093, 2024

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spa- tialrgpt: Grounded spatial reasoning in vision-language mod- els.Advances in Neural Information Processing Systems, 37: 135062–135093, 2024. 2

    2024

  9. [9]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 6

    Pith/arXiv arXiv 2025

  10. [10]

    A survey on multimodal large language models for autonomous driving

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei- Da Liao, et al. A survey on multimodal large language models for autonomous driving. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 958–979, 2024. 1, 2

    2024

  11. [11]

    Mm-spatial: Exploring 3d spatial understanding in multimodal llms

    Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner, Yinfei Yang, Afshin Dehghan, et al. Mm-spatial: Exploring 3d spatial understanding in multimodal llms. InProceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 7395–7408, 2025. 3

    2025

  12. [12]

    Palm-e: An embod- ied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embod- ied multimodal language model. InInternational Conference on Machine Learning, pages 8469–8488. PMLR, 2023. 1

    2023

  13. [13]

    Computing discrete fr´echet distance

    Thomas Eiter, Heikki Mannila, et al. Computing discrete fr´echet distance. 1994. 4

    1994

  14. [14]

    Citygpt: Empowering urban spatial cognition of large language models

    Jie Feng, Tianhui Liu, Yuwei Du, Siqi Guo, Yuming Lin, and Yong Li. Citygpt: Empowering urban spatial cognition of large language models. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing V . 2, pages 591–602, 2025. 1, 3

    2025

  15. [15]

    Urbanllava: A multi-modal large language model for urban intelligence with spatial reasoning and understanding

    Jie Feng, Shengyuan Wang, Tianhui Liu, Yanxin Xi, and Yong Li. Urbanllava: A multi-modal large language model for urban intelligence with spatial reasoning and understanding. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 6209–6219, 2025. 1, 3

    2025

  16. [16]

    City- bench: Evaluating the capabilities of large language models for urban tasks

    Jie Feng, Jun Zhang, Tianhui Liu, Xin Zhang, Tianjian Ouyang, Junbo Yan, Yuwei Du, Siqi Guo, and Yong Li. City- bench: Evaluating the capabilities of large language models for urban tasks. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 2, pages 5413–5424, 2025. 1, 3, 4

    2025

  17. [17]

    Mme: A comprehen- sive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehen- sive evaluation benchmark for multimodal large language models, 2024. 3

    2024

  18. [18]

    Regiongpt: Towards region understanding vision language model

    Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 13796–13806,

  19. [19]

    Rs-vheat: Heat conduction guided ef- ficient remote sensing foundation model

    Huiyang Hu, Peijin Wang, Hanbo Bi, Boyuan Tong, Zhaozhi Wang, Wenhui Diao, Hao Chang, Yingchao Feng, Ziqi Zhang, Yaowei Wang, et al. Rs-vheat: Heat conduction guided ef- ficient remote sensing foundation model. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9876–9887, 2025. 3

    2025

  20. [20]

    Seed-bench: Benchmark- ing multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmark- ing multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024. 3

    2024

  21. [21]

    Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR,

  22. [22]

    Manipllm: Embodied multimodal large language model for object-centric robotic manipulation

    Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yux- ing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18061–18070, 2024. 1

    2024

  23. [23]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, 9 Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024. 3

    2024

  24. [24]

    Mathvista: Evaluating mathemati- cal reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemati- cal reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 2

    Pith/arXiv arXiv 2023

  25. [25]

    Labelu, 2025

    OpenDataLab. Labelu, 2025. Accessed: 2025-09-01. 5

    2025

  26. [26]

    Geopix: A multimodal large language model for pixel-level image understanding in remote sensing.IEEE Geoscience and Remote Sensing Magazine, 2025

    Ruizhe Ou, Yuan Hu, Fan Zhang, Jiaxin Chen, and Yu Liu. Geopix: A multimodal large language model for pixel-level image understanding in remote sensing.IEEE Geoscience and Remote Sensing Magazine, 2025. 2, 3

    2025

  27. [27]

    Nuplanqa: A large-scale dataset and benchmark for multi-view driving scene understanding in multi-modal large language models

    Sung-Yeon Park, Can Cui, Yunsheng Ma, Ahmadreza Moradi- pari, Rohit Gupta, Kyungtae Han, and Ziran Wang. Nuplanqa: A large-scale dataset and benchmark for multi-view driving scene understanding in multi-modal large language models. arXiv preprint arXiv:2503.12772, 2025. 3, 4

    Pith/arXiv arXiv 2025

  28. [28]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 2

    2021

  29. [29]

    Does spatial cognition emerge in frontier models?arXiv preprint arXiv:2410.06468, 2024

    Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Krae- henbuehl, and Vladlen Koltun. Does spatial cognition emerge in frontier models?arXiv preprint arXiv:2410.06468, 2024. 3

    Pith/arXiv arXiv 2024

  30. [30]

    Charting new territories: Exploring the geographic and geospatial capabilities of multimodal llms

    Jonathan Roberts, Timo L¨uddecke, Rehan Sheikh, Kai Han, and Samuel Albanie. Charting new territories: Exploring the geographic and geospatial capabilities of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 554–563, 2024. 1

    2024

  31. [31]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Ta- tiana Matejovicova, Alexandre Ram´e, Morgane Rivi`ere, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Pith/arXiv arXiv

  32. [32]

    Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning,

    V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, J...

  33. [33]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, XianPeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. InConference on Robot Learning, pages 4698–4726. PMLR, 2025. 1

    2025

  34. [34]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. 2

    Pith/arXiv arXiv 2023

  35. [35]

    Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 37:75392–75421, 2024

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 37:75392–75421, 2024. 3

    2024

  36. [36]

    Spatial457: A diagnostic benchmark for 6d spatial reasoning of large mutimodal mod- els

    Xingrui Wang, Wufei Ma, Tiezheng Zhang, Celso M de Melo, Jieneng Chen, and Alan Yuille. Spatial457: A diagnostic benchmark for 6d spatial reasoning of large mutimodal mod- els. InProceedings of the Computer Vision and Pattern Recog- nition Conference, pages 24669–24679, 2025. 3

    2025

  37. [37]

    Slow perception: Let’s perceive geometric figures step-by- step.arXiv preprint arXiv:2412.20631, 2024

    Haoran Wei, Youyang Yin, Yumeng Li, Jia Wang, Liang Zhao, Jianjian Sun, Zheng Ge, Xiangyu Zhang, and Daxin Jiang. Slow perception: Let’s perceive geometric figures step-by- step.arXiv preprint arXiv:2412.20631, 2024. 2

    Pith/arXiv arXiv 2024

  38. [38]

    A comprehensive survey of large language models and multimodal large lan- guage models in medicine.Information Fusion, page 102888,

    Hanguang Xiao, Feizhong Zhou, Xingyue Liu, Tianqi Liu, Zhipeng Li, Xin Liu, and Xiaoxuan Huang. A comprehensive survey of large language models and multimodal large lan- guage models in medicine.Information Fusion, page 102888,

  39. [39]

    Are vlms ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives.arXiv preprint arXiv:2501.04003, 2025

    Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vlms ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives.arXiv preprint arXiv:2501.04003, 2025. 1, 3, 4

    Pith/arXiv arXiv 2025

  40. [40]

    Dynam- icvl: Benchmarking multimodal large language models for dy- namic city understanding.arXiv preprint arXiv:2505.21076,

    Weihao Xuan, Junjue Wang, Heli Qi, Zihang Chen, Zhuo Zheng, Yanfei Zhong, Junshi Xia, and Naoto Yokoya. Dynam- icvl: Benchmarking multimodal large language models for dy- namic city understanding.arXiv preprint arXiv:2505.21076,

    arXiv

  41. [41]

    Thinking in space: How multimodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025. 3

    2025

  42. [42]

    A survey on multimodal large language models.National Science Review, 11(12):nwae403,

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models.National Science Review, 11(12):nwae403,

  43. [43]

    Vldrive: Vision- augmented lightweight mllms for efficient language-grounded autonomous driving

    Ruifei Zhang, Wei Zhang, Xiao Tan, Sibei Yang, Xiang Wan, Xiaonan Luo, and Guanbin Li. Vldrive: Vision- augmented lightweight mllms for efficient language-grounded autonomous driving. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 5923–5933,

  44. [44]

    Open eyes, then reason: Fine-grained vi- sual mathematical understanding in mllms.arXiv preprint arXiv:2501.06430, 2025

    Shan Zhang, Aotian Chen, Yanpeng Sun, Jindong Gu, Yi- Yu Zheng, Piotr Koniusz, Kai Zou, Anton van den Hengel, 10 and Yuan Xue. Open eyes, then reason: Fine-grained vi- sual mathematical understanding in mllms.arXiv preprint arXiv:2501.06430, 2025. 2

    Pith/arXiv arXiv 2025

  45. [45]

    Earthmarker: A visual prompting multi- modal large language model for remote sensing.IEEE Trans- actions on Geoscience and Remote Sensing, 2024

    Wei Zhang, Miaoxin Cai, Tong Zhang, Yin Zhuang, Jun Li, and Xuerui Mao. Earthmarker: A visual prompting multi- modal large language model for remote sensing.IEEE Trans- actions on Geoscience and Remote Sensing, 2024. 1

    2024

  46. [46]

    Urbanmllm: Joint learning of cross-view imagery for urban understanding

    Xin Zhang, Tianjian Ouyang, Yu Shang, Qingmin Liao, and Yong Li. Urbanmllm: Joint learning of cross-view imagery for urban understanding. 2025. 1

    2025

  47. [47]

    Interndrive: A multimodal large language model for autonomous driving scenario understand- ing

    Ye Zhang and Yiming Nie. Interndrive: A multimodal large language model for autonomous driving scenario understand- ing. InProceedings of the 2024 4th International Conference on Artificial Intelligence, Automation and High Performance Computing, pages 294–305, 2024. 1

    2024

  48. [48]

    Embodied-r: Collaborative framework for acti- vating embodied spatial reasoning in foundation models via reinforcement learning

    Baining Zhao, Ziyou Wang, Jianjie Fang, Chen Gao, Fanhang Man, Jinqiang Cui, Xin Wang, Xinlei Chen, Yong Li, and Wenwu Zhu. Embodied-r: Collaborative framework for acti- vating embodied spatial reasoning in foundation models via reinforcement learning. InProceedings of the 33rd ACM In- ternational Conference on Multimedia, pages 11071–11080,

  49. [49]

    Cityeqa: A hierarchical llm agent on embodied ques- tion answering benchmark in city space.EMNLP, 2025

    Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, and Jincai Huang. Cityeqa: A hierarchical llm agent on embodied ques- tion answering benchmark in city space.EMNLP, 2025. 3, 4

    2025

  50. [50]

    Urbench: A comprehensive benchmark for evaluating large multimodal models in multi-view urban sce- narios

    Baichuan Zhou, Haote Yang, Dairong Chen, Junyan Ye, Tianyi Bai, Jinhua Yu, Songyang Zhang, Dahua Lin, Conghui He, and Weijia Li. Urbench: A comprehensive benchmark for evaluating large multimodal models in multi-view urban sce- narios. InProceedings of the AAAI Conference on Artificial Intelligence, pages 10707–10715, 2025. 1, 3, 4 11 RoadBench: Benchmark...

    2025

  51. [51]

    (Section 6)

    The prompts of all the 6 benchmark tasks. (Section 6)

  52. [52]

    (Section 7)

    Additional experimental setup descriptions and complete results. (Section 7)

  53. [53]

    (Section 8)

    Two results supporting further analysis about the impact of reference line prompting methods in the BEV tasks and the impact of scene environment conditions in the FPV tasks. (Section 8)

  54. [54]

    (Section 9)

    Analysis of failure cases in all the 6 benchmark tasks. (Section 9)

  55. [55]

    U-turn",

    Prompts of Benchmark Tasks Examples of prompts for each task in RoadBench and typical MLLM responses are listed below. It is worth noting that in the prompt design for RoadBench, the requirement for MLLM to return YAML format with a line of comments balances the need for explicit reasoning with the demand for structured output. 6.1. BEV Lane Counting An e...

  56. [56]

    BEV Lane Counting The complete experimental results for the BEV lane count- ing task are presented in Table 71

    Additional Experimental Setup Descriptions and Complete Results 7.1. BEV Lane Counting The complete experimental results for the BEV lane count- ing task are presented in Table 71. This table reports the performance of each model or baseline method based on Precision, Recall, F1-Score, and Root Mean Square Error (RMSE) metrics. 7.2. BEV Lane Designation R...

    arXiv 2053

  57. [57]

    Results of Further Analysis 8.1. The Impact of Reference Line Prompting Meth- ods in the BEV Tasks The experimental results of the impact of reference line prompting methods in the BEV tasks are shown in Table 87 and Table 88. For all results, the relative change ratios of metrics for other prompting methods were calculated relative to the benchmark defau...

    arXiv 1958

  58. [58]

    left-turn

    Case Analysis This section presents a detailed analysis of failure cases across different tasks in RoadBench, highlighting common failure patterns and limitations of current MLLMs in fine- grained spatial understanding and reasoning under urban road scenarios. In the following analyzes, we selected the evaluation results of the MLLMs that ranked highly in...

    2019