REVIEW 4 major objections 5 minor 37 references
3D Arena: An Open Platform for Generative 3D Evaluation
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read With 123,243 crowd votes, 3D Arena ranks image-to-3D models and finds users favor visual presentation over structural quality.
desk verdict Useful platform paper with a real dataset and a real confound: the headline texture/splat ELO gaps are aggregate between-model comparisons, and the paper's own caveat never makes it into the abstract. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the arena-style pairwise comparison platform: users view two anonymized 3D outputs side-by-side, rotate and zoom them, and vote for a preferred one, with model identities hidden until after voting. Votes are aggregated through the ELO rating system, a chess-style algorithm that updates each model's score based on pairwise outcomes. The evaluation is standardized by the iso3d dataset, a curated set of 100 isolated-object prompts that controls the input distribution across models. The rendering pipeline uses separate viewers for meshes and Gaussian splats with default lighting, which the paper acknowledges as a potential source of format-dependent presentation differences.
What would settle it
Re-run the same pairwise comparisons with rendering presentation matched across formats (for example, showing meshes with unlit splat-style shading and splats with lit mesh-style shading) and with a professionally recruited voter panel; if the 16.6 ELO splat advantage and the 144.1 ELO texture advantage shrink or reverse, the headline preference findings are measurement artifacts of presentation and population.
Extended reading notes
Core claim
The paper establishes that a large-scale, crowdsourced arena using anonymous side-by-side comparisons can produce a statistically reliable leaderboard for image-to-3D generation, and that the resulting preference data reveals a systematic bias toward visual presentation over technical structure. Specifically, splat-based outputs outrank mesh outputs by 16.6 ELO points, and textured models outrank untextured models by 144.1 ELO points, with very small p-values. A controlled comparison using the same underlying model in both formats shows a 78 ELO advantage for the splat version, isolating the format effect. The paper interprets this as evidence that immediate visual features dominate preference formation, consistent with dual-process cognition and the aesthetic-usability effect, and it highlights a gap between stated professional priorities and revealed voter behavior.
Load-bearing premise
The load-bearing premise is that the ELO gaps between formats and texture conditions reflect genuine quality preferences rather than systematic biases introduced by the two different rendering pipelines, the default lighting, or the particular self-selected pool of volunteer voters.
Editorial extensions
If this is right
- If the preference patterns hold, model developers optimizing for leaderboard rank will rationally prioritize vivid rendering, texture, and splat-style presentation over mesh topology and downstream usability.
- The 78 ELO gap between identical models in different formats implies that output format is a confound that any comparative benchmark must control or report explicitly.
- Evaluation practice should shift toward multi-criteria assessment, separating rendered visual appearance from wireframe structure and task-specific utility.
- The 100-prompt iso3d set can serve as a fixed, reusable evaluation suite for image-to-3D models, making future comparisons across studies more consistent.
- The revealed-preference gap between visual appeal and professional usability suggests that single-score leaderboards are insufficient for guiding real-world 3D asset production.
Reading between the lines
- If the 144.1 ELO texture advantage is largely presentation-driven, then current leaderboards likely underrate untextured but topologically cleaner models for professional workflows; a wireframe-only leaderboard would probably reorder the field substantially.
- As splat renderers improve and become more common, the format advantage may widen, pushing benchmarks toward format-aware normalization or per-format leaderboards.
- A task-oriented evaluation, such as measuring animation readiness or editability, would likely produce a different ordering than the visual-preference ranking, and comparing the two orderings could quantify the utility gap directly.
- The platform's open submission model could be extended to support controlled rendering variations of the same model, turning the 78 ELO format effect into a continuous diagnostic for presentation bias.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. 3D Arena is a crowdsourced pairwise human-preference platform for image-to-3D generation, operating since June 2024. The paper reports 123,243 votes from 8,096 users across 19 models, introduces the iso3d 100-prompt dataset, describes fraud detection and ELO-based ranking, and analyzes the vote data to claim that Gaussian splat outputs are preferred over meshes (16.6 ELO) and textured models over untextured ones (144.1 ELO). It also proposes multi-criteria and task-oriented evaluation recommendations.
Significance. The platform is a genuinely useful community resource, and the vote count is an order of magnitude larger than prior 3D preference collections; the iso3d prompt set and the live leaderboard are concrete contributions. The TRELLIS-versus-TRELLIS-3DGS paired comparison is a good attempt at controlling model identity. If the preference effects were established by controlled, auditable comparisons, the findings on presentation versus structural quality would be of wide interest to the generative-3D community. However, the headline ELO differences are currently confounded and non-auditable, so the paper's central analytical claims need substantial revision before the findings can be accepted.
major comments (4)
- [§4.2.2 and Abstract] The 144.1 ELO textured-over-untextured advantage is computed by pooling votes across entire model families, so texture status is confounded with model identity, architecture, training data, and geometry quality. The paper itself acknowledges this at the end of §4.2.2 ('Additional controlled studies would be required to disentangle the relative contributions of these factors'), yet the abstract and §4.3 present the 144.1 difference as an established preference for texture. Please provide a within-model controlled comparison (same generation, textured versus untextured rendering, same renderer) or an analysis adjusted for model identity, and either report adjusted and unadjusted effects separately or remove the causal wording from the abstract and conclusion.
- [§3.2.2 and §4.2.1] The 16.6 ELO mesh-versus-splat advantage is not a clean format effect: meshes are rendered with Babylon.js under dynamic lighting, while Gaussian splats are rendered with gsplat.js using unlit, vibrant defaults, and only three splat models appear on the leaderboard, including the top two entries. The TRELLIS versus TRELLIS-3DGS comparison (78 ELO) changes both representation and rendering, so it does not isolate a pure format preference. Please report same-renderer, same-model format comparisons and format-level effects adjusted for model strength, or explicitly refrain from claiming a format preference.
- [§3.3 and §4.2] The ELO and significance statements are not auditable. The manuscript gives no K-factor, no confidence intervals for ELO differences, no description of how the p-values in §4.2.1 and §4.2.2 are derived, and no release of vote-level or pair-level data. Since the leaderboard ELO values, the 16.6 and 144.1 differences, and the associated p-values are the central quantitative claims, please publish the anonymized vote log (or a disaggregated per-model-pair vote matrix) and report bootstrap confidence intervals for all headline ELO differences.
- [§4.3] The cognitive interpretation (System 1/System 2 processing, the aesthetic-usability effect, and the 150-200 ms visual-processing claim) is presented as if established by the voting data, but the study has no manipulation of processing time, no direct measures of cognitive load, and no behavioral controls. These are plausible post-hoc hypotheses consistent with aggregate vote counts, not tested findings. Please reframe this section as speculative and note the need for controlled behavioral experiments before such mechanisms are claimed.
minor comments (5)
- [§4.1] The text reads 'V ote distribution' — a typo for 'Vote distribution'.
- [§3.3] The claim of '99.75% user authenticity' is inconsistent with '31 accounts flagged among 8,096 total users', which corresponds to 99.62% unflagged. Please clarify the definition and the arithmetic.
- [Table 1] The superscript markers '1' and '2' in 'Strawberrry 1', 'Strawb3rry 1', and 'IM-MA 2' are never defined in the table caption or the text; either add footnotes or remove the markers.
- [§2.2] The comparison with 3DGen-Bench relies on priority language ('our 3D Arena platform launched earlier'); please provide launch-dated citations or repository timestamps for both platforms, or soften the claim to avoid a non-central priority dispute.
- [§3.4] The sentence 'anonymous submissions were initially permitted and remain included in internal rankings 1' contains a dangling footnote marker that is not resolved anywhere in the manuscript.
Circularity Check
No significant circularity: the ELO rankings and preference findings are direct summaries of the collected pairwise votes, not fitted predictions or self-citation-derived results.
full rationale
3D Arena is an observational measurement paper. The ELO ratings in Table 1 are computed from pairwise votes via a standard ELO update from a common initial rating, and the mesh/splat and textured/untextured contrasts in Sections 4.2.1 and 4.2.2 are aggregate summaries of those same votes. There is no fitted parameter that is later renamed a prediction, and no derivation that takes its own conclusion as an input. The paper explicitly flags its main inferential limitation in Section 4.2.2: 'Additional controlled studies would be required to disentangle the relative contributions of these factors.' That is an identification caveat about confounding, not circularity. The TRELLIS versus TRELLIS-3DGS comparison is a pairwise measurement of two outputs of the same model, not an equation that reduces to its own input. No load-bearing self-citation appears: the cited methods (Chatbot Arena, Bradley-Terry, ELO) are external standard tools, and no prior result by the same author is invoked to force a conclusion. The headline scale claims are descriptive counts (123,243 votes from 8,096 users), and the preference findings are direct empirical summaries of the collected data. I therefore find no significant circularity.
Assumptions & free parameters
free parameters (3)
- ELO initial rating =
1200
- ELO K-factor =
not reported
- Fraud detection p-value threshold =
p < 0.00001
assumptions (5)
- domain assumption Crowdsourced pairwise votes are a valid signal of 3D quality preference.
- domain assumption Anonymous side-by-side comparison removes model identity bias.
- domain assumption Each vote is an independent event for ELO/Bradley-Terry inference.
- standard math Binomial test for fraud detection assumes votes follow community consensus probabilities.
- domain assumption Human visual features are processed faster than geometric details.
Cite this review
Pith. "Pith review of 3D Arena: An Open Platform for Generative 3D Evaluation." pith.science (2026). https://pith.science/paper/VWR75ULS
@misc{pith2026250618787,
author = {Pith},
title = {Pith review of: 3D Arena: An Open Platform for Generative 3D Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/VWR75ULS}},
note = {Machine review of arXiv:2506.18787}
}
read the original abstract
Evaluating Generative 3D models remains challenging due to misalignment between automated metrics and human perception of quality. Current benchmarks rely on image-based metrics that ignore 3D structure or geometric measures that fail to capture perceptual appeal and real-world utility. To address this gap, we present 3D Arena, an open platform for evaluating image-to-3D generation models through large-scale human preference collection using pairwise comparisons. Since launching in June 2024, the platform has collected 123,243 votes from 8,096 users across 19 state-of-the-art models, establishing the largest human preference evaluation for Generative 3D. We contribute the iso3d dataset of 100 evaluation prompts and demonstrate quality control achieving 99.75% user authenticity through statistical fraud detection. Our ELO-based ranking system provides reliable model assessment, with the platform becoming an established evaluation resource. Through analysis of this preference data, we present insights into human preference patterns. Our findings reveal preferences for visual presentation features, with Gaussian splat outputs achieving a 16.6 ELO advantage over meshes and textured models receiving a 144.1 ELO advantage over untextured models. We provide recommendations for improving evaluation methods, including multi-criteria assessment, task-oriented evaluation, and format-aware comparison. The platform's community engagement establishes 3D Arena as a benchmark for the field while advancing understanding of human-centered evaluation in Generative 3D.
Figures
Reference graph
Works this paper leans on
-
[1]
Learning repre- sentations and generative models for 3d point clouds
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning repre- sentations and generative models for 3d point clouds. In International conference on machine learning, pages 40–49. PMLR, 2018
work page 2018
-
[2]
Tomas Akenine-Moller, Eric Haines, and Naty Hoffman. Real-time rendering. AK Peters/crc Press, 2019
work page 2019
-
[3]
Estimation of the timing of human visual perception from magnetoencephalog- raphy
Kaoru Amano, Naokazu Goda, Shin’ya Nishida, Yoshimichi Ejima, Tsunehiro Takeda, and Yoshio Ohtani. Estimation of the timing of human visual perception from magnetoencephalog- raphy. Journal of Neuroscience, 26(15):3981–3991, 2006
work page 2006
-
[4]
Sf3d: Stable fast 3d mesh reconstruction with uv-unwrapping and illumination disentanglement
Mark Boss, Zixuan Huang, Aaryaman Vasishta, and Varun Jampani. Sf3d: Stable fast 3d mesh reconstruction with uv-unwrapping and illumination disentanglement. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 16240–16250, 2025
work page 2025
-
[5]
RALPH ALLAN BRADLEY and MILTON E. TERRY . Rank analysis of incomplete block designs: The method of paired comparisons. Biometrika, 39(3-4):324–345, 12 1952
work page 1952
-
[6]
Meshanything: Artist-created mesh generation with autoregressive transformers
Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, et al. Meshanything: Artist-created mesh generation with autoregressive transformers. arXiv preprint arXiv:2406.10163, 2024
arXiv 2024
-
[7]
3dtopia-xl: Scaling high-quality 3d 9 asset generation via primitive diffusion
Zhaoxi Chen, Jiaxiang Tang, Yuhao Dong, Ziang Cao, Fangzhou Hong, Yushi Lan, Tengfei Wang, Haozhe Xie, Tong Wu, Shunsuke Saito, et al. 3dtopia-xl: Scaling high-quality 3d 9 asset generation via primitive diffusion. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26576–26586, 2025
work page 2025
-
[8]
Chatbot arena: An open platform for evaluating llms by human preference
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, 2024
2024
Show all 37 references
-
[9]
Common sense machines: 3d ai for creators and developers
Common Sense Machines. Common sense machines: 3d ai for creators and developers. https://www.csm.ai/, 2024. Accessed: 2024
2024
-
[10]
Arpad E. Elo. The Rating of Chess Players, Past and Present. Ishi Press International, Bronx, N.Y ., 1978
1978
-
[11]
Dual-process theories of reasoning: Contemporary issues and develop- mental applications
Jonathan St BT Evans. Dual-process theories of reasoning: Contemporary issues and develop- mental applications. Developmental review, 31(2-3):86–102, 2011
2011
-
[12]
Fardo, Victor H
Fernando A. Fardo, Victor H. Conforto, Francisco C. de Oliveira, and Paulo S. Rodrigues. A formal evaluation of psnr as quality measurement parameter for image segmentation algorithms, 2016
2016
-
[13]
T3bench: Benchmarking current progress in text-to-3d generation, 2024
Yuze He, Yushi Bai, Matthieu Lin, Wang Zhao, Yubin Hu, Jenny Sheng, Ran Yi, Juanzi Li, and Yong-Jin Liu. T3bench: Benchmarking current progress in text-to-3d generation, 2024
2024
-
[14]
Visual features for perception, attention, and working memory: Toward a three-factor framework
Liqiang Huang. Visual features for perception, attention, and working memory: Toward a three-factor framework. Cognition, 145:43–52, 2015
2015
-
[15]
Spar3d: Stable point-aware reconstruction of 3d objects from single images
Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M Rehg, and Varun Jampani. Spar3d: Stable point-aware reconstruction of 3d objects from single images. arXiv preprint arXiv:2501.04689, 2025
2025 arXiv
-
[16]
Hyper3d: Ai-powered 3d generation platform
Hyper3D. Hyper3d: Ai-powered 3d generation platform. https://hyper3d.ai/, 2024. Accessed: 2024
2024
-
[17]
Genai arena: An open evaluation platform for generative models
Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models. Advances in Neural Information Processing Systems, 37:79889–79908, 2024
2024
-
[18]
Real3d: Scaling up large reconstruction models with real-world images
Hanwen Jiang, Qixing Huang, and Georgios Pavlakos. Real3d: Scaling up large reconstruction models with real-world images. arXiv preprint arXiv:2406.08479, 2024
2024 arXiv
-
[19]
Shap-e: Generating conditional 3d implicit functions
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023
2023 arXiv
-
[20]
Apparent usability vs
Masaaki Kurosu and Kaori Kashimura. Apparent usability vs. inherent usability: experimental analysis on the determinants of the apparent usability. In Conference companion on Human factors in computing systems, pages 292–293, 1995
1995
-
[21]
Karlo-v1.0.alpha on coyo-100m and cc15m
Donghoon Lee, Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Baek, and Saehoon Kim. Karlo-v1.0.alpha on coyo-100m and cc15m. https://github.com/ kakaobrain/karlo, 2022
2022
-
[22]
A comprehensive survey on 3d content generation, 2024
Jian Liu, Xiaoshui Huang, Tianyu Huang, Lu Chen, Yuenan Hou, Shixiang Tang, Ziwei Liu, Wanli Ouyang, Wangmeng Zuo, Junjun Jiang, and Xianming Liu. A comprehensive survey on 3d content generation, 2024
2024
-
[23]
Meshformer: High-quality mesh genera- tion with 3d-guided reconstruction model
Minghua Liu, Chong Zeng, Xinyue Wei, Ruoxi Shi, Linghao Chen, Chao Xu, Mengqi Zhang, Zhaoning Wang, Xiaoshuai Zhang, Isabella Liu, et al. Meshformer: High-quality mesh genera- tion with 3d-guided reconstruction model. arXiv preprint arXiv:2408.10198, 2024
2024 arXiv
-
[24]
Meshy: The ultimate ai 3d model generator for creators
Meshy LLC. Meshy: The ultimate ai 3d model generator for creators. https://www.meshy. ai/, 2024. Accessed: 2024
2024
-
[25]
The role of visual attention in decision making
Jacob Lund Orquin and Simone Mueller Loose. The role of visual attention in decision making. In 24th Subjective Probability, Utility and Decision-making Conference, 2013. 10
2013
-
[26]
Computer animation: algorithms and techniques
Rick Parent. Computer animation: algorithms and techniques. Newnes, 2012
2012
-
[27]
Lgm: Large multi-view gaussian model for high-resolution 3d content creation
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1–18. Springer, 2024
2024
-
[28]
Triposr: Fast 3d object reconstruction from a single image
Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.02151, 2024
2024 arXiv
-
[29]
Tripo3d: Generate 3d model powered by ai
Tripo AI. Tripo3d: Generate 3d model powered by ai. https://www.tripo3d.ai/, 2024. Accessed: 2024
2024
-
[30]
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600– 612, 2004
2004
-
[31]
Unique3d: High-quality and efficient 3d mesh generation from a single image
Kailu Wu, Fangfu Liu, Zhihan Cai, Runjie Yan, Hanyang Wang, Yating Hu, Yueqi Duan, and Kaisheng Ma. Unique3d: High-quality and efficient 3d mesh generation from a single image. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024
2024
-
[32]
Structured 3d latents for scalable and versatile 3d generation, 2025
Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation, 2025
2025
-
[33]
In- stantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruc- tion models
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. In- stantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruc- tion models. arXiv preprint arXiv:2404.07191, 2024
2024 arXiv
-
[34]
Hi3dgen: High-fidelity 3d geometry generation from images via normal bridging, 2025
Chongjie Ye, Yushuang Wu, Ziteng Lu, Jiahao Chang, Xiaoyang Guo, Jiaqing Zhou, Hao Zhao, and Xiaoguang Han. Hi3dgen: High-fidelity 3d geometry generation from images via normal bridging, 2025
2025
-
[35]
3dgen-bench: Comprehensive benchmark suite for 3d generative models
Yuhan Zhang, Mengchen Zhang, Tong Wu, Tengfei Wang, Gordon Wetzstein, Dahua Lin, and Ziwei Liu. 3dgen-bench: Comprehensive benchmark suite for 3d generative models. arXiv preprint arXiv:2503.21745, 2025
2025 arXiv
-
[36]
Challenges and opportunities in 3d content generation, 2024
Ke Zhao and Andreas Larsen. Challenges and opportunities in 3d content generation, 2024
2024
-
[37]
Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation, 2025
Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang, Sheng Zhang, Xianghui Yang, Huiwen Shi, Sicong Liu, Junta Wu, Yihang Lian, Fan Yang, Ruining Tang, Zebin He, Xinzhou Wang, Jian Liu, Xuhui Zuo, Zhuo Chen, Biwen Lei, Haohan W...
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.