REVIEW 4 major objections 4 minor 24 references
Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities?
T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper claims that many current 3D LLM benchmarks can be passed by ordinary 2D vision-language models given rendered images of the point cloud, so those benchmarks do not measure true 3D reasoning.
desk verdict The 2D-Cheating diagnosis is valuable and probably right, but the headline 3D MM-Vet gap is contaminated by an evaluator swap and needs a controlled re-test before it can carry the paper's weight. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the VLM3D pipeline plus the 2D-Cheating comparison it enables: point clouds are rendered into flat images at chosen viewpoints, the query is augmented with few-shot examples, and an off-the-shelf VLM (GPT-4o or Qwen2-VL-72B) answers. For scene benchmarks the paper adds viewpoint selection, including a Best-of-N oracle that renders five views and picks the one with the highest average score, which isolates whether any flat view of the scene contains the needed information. This rendering-plus-oracle setup is the instrument that separates tasks solvable from surface appearance from tasks that seem to need the full 3D structure.
What would settle it
Compute the margin on tasks where the VLM currently loses (e.g., ScanQA and SQA3D) after giving the VLM the best possible flat view and matching the answer format exactly, for instance by rendering high-resolution real-photo-style images or by improving the VLM's instruction following; if the VLM then reaches or passes the 3D model on those tasks, the presumed 3D-specific tasks are also 2D-cheatable.
Extended reading notes
Core claim
The central discovery is that a task's difficulty for a 2D VLM is being mistaken for evidence that the task measures 3D capability. Concretely, VLM3D with GPT-4o reaches 58.1 on 3D MM-Vet against a 3D SOTA of 43.2, and likewise edges out the 3D specialist on ObjaverseXL-LVIS captioning, using nothing but rendered point-cloud images plus few-shot prompting. On ScanQA, providing a viewpoint selected by Best-of-N over five candidates lifts the VLM from 12.8 to 28.2 METEOR and from 51.2 to 71.2 CIDEr, passing the 3D baseline, while multi-view inputs help far less. The authors infer that benchmarks where such a 2D model still loses to a 3D LLM are the ones that exercise unique 3D abilities, and they formalize this into principles for future 3D benchmarks.
Load-bearing premise
The paper assumes a task genuinely requires unique 3D capability when a 2D VLM scores much worse than a 3D LLM, but a VLM can score worse for reasons unrelated to 3D, such as fine-grained object recognition, ambiguous language, or unfavorable image viewpoint.
Editorial extensions
If this is right
- Scores on 3D MM-Vet and ObjaverseXL-LVIS captioning cannot be read as measures of 3D-specific reasoning, because a 2D model with rendered images beats the 3D SOTA.
- A new 3D LLM could improve on these benchmarks by getting better at 2D recognition or language priors without any advance in 3D understanding.
- Scene benchmarks like ScanQA and SQA3D are more resistant to single-view cheating, but their text-similarity metrics mis-score reasonable answers, so part of the reported gap is metric artifact.
- Benchmark builders should follow the paper's principles: complex point clouds, detail-oriented and context-specific questions, and flexible LLM-based evaluation with multiple acceptable answers.
- Evaluation of 3D LLMs should report a 2D/1D baseline explicitly, so that gains attributable to 3D representation are separated from gains available to any VLM.
Reading between the lines
- The paper treats a VLM's failure as the marker of true 3D need; a stronger test would require showing the VLM fails even with an oracle view and with the same answer metric, since its failure could come from fine-grained recognition or language mismatch rather than 3D reasoning.
- Their own oracle-view results suggest that integrating information across several viewpoints is itself a hard skill for current VLMs; a benchmark could legitimately measure that as a 3D-adjacent capability instead of excluding all multi-view information.
- The principles could be turned into a concrete stress test: build a small benchmark of questions about occluded, counterintuitive, or layout-specific properties, then confirm that VLM scores collapse while 3D specialists hold, which the paper leaves to future work.
- If 2D-Cheating is as widespread as the numbers suggest, the same rendering attack could be used as a cheap screening tool by model developers: before training a 3D LLM, check whether a strong VLM with rendered views already solves the benchmark.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes VLM3D, a pipeline that renders 3D point clouds into images and feeds them with few-shot prompts to off-the-shelf VLMs (GPT-4o, Qwen2-VL-72B), and uses it to probe whether current 3D LLM benchmarks can be 'cheated' by models that only see 2D renderings. On two object-level benchmarks (3D MM-Vet and ObjaverseXL-LVIS Caption), VLM3D reportedly exceeds published SOTA 3D LLMs, while on two scene-level benchmarks (ScanQA and SQA3D) it generally underperforms 3D baselines except when an oracle viewpoint is selected via Best-of-N. The paper interprets these results as evidence that object-level benchmarks partly measure 2D image understanding, articulates principles for designing tasks that genuinely test 3D understanding (complex point clouds, context-specific inquiry, flexible LLM-based evaluation), and advocates separating 3D, 2D, and 1D abilities in evaluation. It includes a human-evaluation study of existing metrics and a GPT-4o-based pass-rate analysis of the proposed principles.
Significance. If the central claim holds, the paper provides a valuable and inexpensive baseline for the 3D LLM evaluation community: future benchmarks should compare 3D LLMs against 2D VLMs fed rendered views, and the paper's proposed principles give a constructive direction. The human-evaluation correlation results (Table 5) are useful evidence that text-similarity metrics are poorly aligned with human judgment for open-ended tasks, and the public release of code and data supports reproducibility. However, the headline quantitative evidence on 3D MM-Vet is weakened by the evaluator mismatch, and the inverse inference from VLM underperformance to 'genuine 3D capability' is not independently validated. The conceptual contribution is solid, but the empirical support for the strongest version of the claim is currently insufficient.
major comments (4)
- [Appendix D.3 / Table 1] The 3D MM-Vet comparison in Table 1 is confounded by an evaluator mismatch: the published ShapeLLM score of 43.2 was obtained under the original GPT-3.5 LLM evaluator, while VLM3D's 58.1 was scored with GPT-4o-mini. The paper reports no calibration or side-by-side comparison of the two judges. Because the +14.9 gap on 3D MM-Vet is the flagship evidence for 2D-Cheating, the claim is not supported until both models are scored with the same judge, prompt, and rubric.
- [§1 and §4.1] The paper's inverse inference—that tasks where VLMs underperform 3D LLMs 'truly involve unique 3D capabilities'—equates VLM difficulty with 3D specificity without independent validation. A VLM can fail for reasons unrelated to 3D reasoning (e.g., fine-grained instance recognition, language ambiguity, or poor rendering), so the classification of benchmarks as 'genuinely 3D' is not established. Provide a control, such as showing that VLM errors persist even when the same information is available in a 2D-friendly format, or that 3D LLM successes are specifically tied to 3D structure.
- [§3.3 / Tables 2–4] All baselines are taken from published numbers rather than re-run under the same pipeline, and no error bars or significance tests are reported. Several differences are small (e.g., ScanQA METEOR +1.6, SQA3D EM -2.4) and could be within run-to-run variance. Since these differences are used to separate '2D-cheatable' from 'genuine 3D' tasks, the paper should report variance across multiple runs and, where feasible, re-run the 3D baselines with the same evaluation code.
- [§5.2 / Table 6] The quantitative validation of the proposed principles uses GPT-4o both as the VLM whose behavior motivated the principles and as the judge of whether tasks comply with them. This circularity could explain the correlation between pass rates and VLM/3DLLM gaps. The LLM-based principle evaluation should be validated against human ratings (as the paper does for benchmark metrics in §3.4) or replaced with a judge independent of the VLM family used in VLM3D.
minor comments (4)
- [Tables 2 and 3] The header 'ROUGH' is a typo for 'ROUGE-L'; please correct this consistently, and also correct the typo 'BLUE_1' in Appendix C.1 to 'BLEU_1'.
- [§2.2.3] The BoN selection rule is described as 'the highest average score among all viewpoints,' but Figure 5 shows a curve; please state explicitly how k and n are swept and how the oracle score is defined when multiple (k, n) configurations are reported.
- [§3.4] The human evaluation uses 50 randomly sampled tasks per benchmark; please report the number of human judges, the instruction given, and the inter-annotator agreement, as these affect the interpretability of the Spearman correlations in Table 5.
- [Appendix D.3] The sentence 'We replaced the GPT-3.5 model originally used for evaluation in the 3D-MM-Vet benchmark with GPT-4o-mini' should specify whether this replacement was also applied to the baseline 3D SOTA scores; if not, the reported comparison must be treated as inconsistent until both are scored under the same evaluator.
Circularity Check
No significant circularity: the core 2D-Cheating result is an external, empirical comparison rather than a derivation that reduces to its own inputs.
full rationale
The paper's central claim is that some 3D LLM benchmark tasks can be solved by 2D VLMs given rendered point-cloud images, and that such tasks therefore do not isolate unique 3D capabilities. The evidence for this is an external experiment: VLM3D, a pipeline that renders point clouds and feeds them to GPT-4o or Qwen2-VL, is compared against published 3D SOTA scores on fixed benchmarks. No parameter is fitted to the target conclusion, and the VLM scores are not constructed from the benchmark labels or from the 3D models' outputs. The oracle-view procedure uses ground truth to select the best viewpoint, but the paper explicitly labels it an oracle and an upper bound, so it is not masquerading as a prediction or as evidence for typical VLM performance. The LLM-based validation of the proposed principles in Table 6 uses GPT-4o as a judge, which is self-referential in the sense that a VLM is evaluating task properties, but this validation is not load-bearing for the central empirical comparison and does not constitute a circular derivation; it is better characterized as a methodological limitation. The main identified weakness, namely the inference that tasks where a VLM underperforms a 3D LLM are necessarily testing unique 3D capabilities, is a substantive assumption about what 3D-specific reasoning requires, not a definitional identity or an equation that reduces to its inputs. Likewise, the discrepancy between the GPT-3.5 evaluator used for the published 3D MM-Vet baseline and the GPT-4o-mini evaluator used for VLM3D is a possible confound affecting the validity of the comparison, but it is not a circularity in the paper's derivation chain. No load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations were found. The paper is self-contained as an empirical study, so the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Best-of-N oracle view selection =
k=4 or 5 viewpoints, n=20 samples per view
assumptions (4)
- domain assumption VLM performance on rendered point-cloud images is a valid proxy for the 2D information available in a benchmark task.
- domain assumption Tasks where VLMs perform worse than 3D LLMs are tasks that truly require unique 3D capabilities.
- domain assumption GPT-4o can reliably judge whether benchmark tasks satisfy the proposed design principles.
- domain assumption Published 3D SOTA and baseline scores are directly comparable to VLM3D scores, despite different evaluation pipelines and model training.
Cite this review
Pith. "Pith review of Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities?." pith.science (2026). https://pith.science/paper/DAQE7THI
@misc{pith2026250208503,
author = {Pith},
title = {Pith review of: Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities?},
year = {2026},
howpublished = {\url{https://pith.science/paper/DAQE7THI}},
note = {Machine review of arXiv:2502.08503}
}
read the original abstract
In this work, we identify the "2D-Cheating" problem in 3D LLM evaluation, where these tasks might be easily solved by VLMs with rendered images of point clouds, exposing ineffective evaluation of 3D LLMs' unique 3D capabilities. We test VLM performance across multiple 3D LLM benchmarks and, using this as a reference, propose principles for better assessing genuine 3D understanding. We also advocate explicitly separating 3D abilities from 1D or 2D aspects when evaluating 3D LLMs. Code and data are available at https://github.com/LLM-class-group/Revisiting-3D-LLM-Benchmarks
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Daich Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. 2021. https://api.semanticscholar.org/CorpusID:245334889 Scanqa: 3d question answering for spatial scene understanding . 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19107--19117
work page 2021
-
[4]
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. 2020. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621--11631
work page 2020
-
[5]
Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. https://arxiv.org/abs/1702.04405 Scannet: Richly-annotated 3d reconstructions of indoor scenes . Preprint, arXiv:1702.04405
arXiv 2017
-
[6]
Matt Deitke , Dustin Schwenk , Jordi Salvador , Luca Weihs , Oscar Michel , Eli VanderBilt , Ludwig Schmidt , Kiana Ehsani , Aniruddha Kembhavi , and Ali Farhadi . 2022. https://doi.org/10.48550/arXiv.2212.08051 Objaverse: A Universe of Annotated 3D Objects . arXiv e-prints, arXiv:2212.08051
-
[7]
Agrim Gupta , Piotr Doll \'a r , and Ross Girshick . 2019. https://doi.org/10.48550/arXiv.1908.03195 LVIS: A Dataset for Large Vocabulary Instance Segmentation . arXiv e-prints, arXiv:1908.03195
-
[8]
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. 2022. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06336
arXiv 2022
Show all 24 references
- [9]
-
[10]
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. 2023. https://api.semanticscholar.org/CorpusID:265351495 An embodied generalist agent in 3d world . ArXiv, abs/2311.12871
2023 arXiv
-
[11]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning
2023
-
[12]
Xianzheng Ma, Yash Bhalgat, Brandon Smart, Shuai Chen, Xinghui Li, Jian Ding, Jindong Gu, Dave Zhenyu Chen, Songyou Peng, Jia-Wang Bian, et al. 2024. When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models. arXiv preprint ...
2024
- [13]
- [14]
-
[15]
OpenAI. 2022. https://openai.com/index/chatgpt/ Introducing chatgpt . openai.com
2022
-
[16]
OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774
2023 arXiv
-
[17]
OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o . openai.com
2024
- [18]
- [19]
-
[20]
Charles Spearman. 1904. https://doi.org/10.2307/1412159 The proof and measurement of association between two things . The American Journal of Psychology, 15(1):72--101
1904 doi
-
[21]
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language mode...
2024 arXiv
-
[22]
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. 2021. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 913--922
2021
- [23]
- [24]
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.