REVIEW 2 major objections 4 minor 5 cited by
FlipSet, a 28-item benchmark that asks vision-language models what a monkey sees on the back of a rotated card, finds that 91.3% of 103 models score below the 25% chance level and that 75.9% of errors merely copy the camera's viewpoint.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 02:54 UTC pith:H7TB5IEF
load-bearing objection Promising diagnostic design and a likely-real egocentric-bias finding, but the stated physical setup would keep the monkey from seeing any characters—invalidating the labels as described. the 2 major comments →
Egocentric Bias in Vision-Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, FlipSet shows that 91.3% of 103 vision-language models perform below the 25% random baseline on a task that only requires reading a short character string and rotating it 180 degrees from a monkey's viewpoint. Mean accuracy is 8.96%, median 5.36%, and egocentric responses account for 75.88% of all answers. In the control battery, the same models average 90.4% on theory-of-mind recognition (does the monkey see a different string?), 26.1% on isolated mental rotation (what does the string become under 180 degrees?), and 10.3% on the full L2 task. The authors interpret the gap between the product of the component scores and the observed L2 accuracy—a deficit present in
What carries the argument
FlipSet's central device is a controlled multiple-choice item: an upright white card showing a 2D string such as '81', with a plush monkey on the opposite side facing the card's back. The question 'What does the monkey see on the card?' requires mentally rotating the string 180° (to '18'). Every item's four options are designed to diagnose the failure mode: correct perspective-transformed answer, egocentric camera-view answer, a contour-confusable distractor, and an unrelated random distractor, with 12 counterbalanced layouts per item to remove position bias. A companion control set reuses the same images under three prompts—theory-of-mind visibility judgment, pure mental rotation, and full
Load-bearing premise
The claims rest on two assumptions: that the monkey actually sees the rotated character string, and that expected L2 performance equals theory-of-mind accuracy multiplied by mental-rotation accuracy; if either is wrong, the egocentric-bias and compositional-deficit conclusions lose their footing.
What would settle it
Run the same FlipSet battery with the character string printed on both sides of the card, so the monkey genuinely sees the rotated characters. If model accuracy jumps from near 10% to at or above chance, the original opaque-card geometry—not a compositional deficit—explains the failure; if accuracy stays near chance, the integration deficit is genuine. A complementary check is to ask human raters what the monkey sees in the original images; if they say the back is blank, the ground-truth labels are invalid.
If this is right
- Chain-of-thought prompting does not fix the egocentric bias and often amplifies it, implying the limitation is not a lack of verbal reasoning steps.
- Models with near-perfect theory-of-mind scores and above-chance mental rotation still fail the integrated task, so the two component skills do not automatically compose in current architectures.
- Mental rotation accuracy correlates strongly with L2 accuracy (r = 0.746) while theory of mind does not (r = 0.010), pointing to spatial transformation as the bottleneck skill.
- FlipSet's 28-item, zero-shot protocol with counterbalanced answer positions offers a reusable diagnostic for tracking perspective-taking progress in future vision-language models.
Where Pith is reading between the lines
- Beyond the paper: the ground-truth labels assume the monkey sees the rotated string, but the described setup—an upright, opaque card with characters printed on the camera-facing side—would show the monkey a blank back. Re-rendering the stimuli with the string visible on both sides (or a transparent card) would test whether the reported egocentric bias is partly a visual-geometry artifact.
- Beyond the paper: the product-rule baseline treats theory of mind and mental rotation as independent and free to combine. Because the full L2 task adds prompt comprehension and coordination demands, any real task cost will automatically read as a 'compositional deficit'; adding a two-step control task would calibrate this baseline.
- Beyond the paper: the benchmark only uses 180-degree rotations. Extending FlipSet to 90- and 270-degree rotations would test whether model error scales with rotation angle the way human response time does, linking the result to classical mental-rotation findings.
- Beyond the paper: the egocentric bias may partly reflect training statistics, since front-view text is far more common than rotated text. Fine-tuning on multi-view or egocentric-to-allocentric data is a testable intervention that, if it reduces egocentric errors, would support a data-driven rather than architectural explanation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FlipSet, a benchmark for Level-2 visual perspective taking (L2 VPT) in vision-language models. Each item shows a card with a character string and a plush monkey on the opposite side; the model must choose what the monkey sees. The authors report that 91.3% of 103 VLMs perform below 25% chance, with roughly 75.88% of errors being egocentric (outputting the camera viewpoint). Control experiments on 24 models separate theory of mind (ToM), mental rotation (MR), and L2 VPT, reporting high ToM accuracy, near-chance MR, and catastrophically low L2 VPT. They further claim a compositional deficit: L2 VPT is below the product ToM × MR. The paper frames these results as evidence that VLMs lack mechanisms for integrating social awareness with spatial transformation.
Significance. If the benchmark and its labels are valid, this is a valuable large-scale diagnostic result with clear implications for model architecture: it would demonstrate a systematic and specific failure in situated social reasoning. The scale (103 models), the error-type taxonomy, and the separation of ToM and MR are strong points. However, the entire quantitative edifice rests on the stimulus geometry producing the ground-truth 180° rotations, and the compositional-deficit conclusion relies on an unvalidated product-rule baseline. Both are load-bearing and currently unsupported.
major comments (2)
- [Methods (Main Experiments)] The described stimulus geometry is inconsistent with the ground-truth labels. The text says the card is 'placed upright on a wooden floor' and the monkey sits 'on the opposite side, facing the card's back.' An upright opaque card shows the monkey a blank back; a transparent card would show a mirror reflection (e.g., 'd' would appear 'b', not 'p'). The 180° rotations in Table 1 (d→p, nod→pou) correspond to the six/nine paradigm where the card lies flat between the two agents, not to the described vertical-card layout. Since all accuracy, egocentric-error, and ToM/MR/L2 statistics are computed against these labels, the benchmark's validity is at stake. The paper releases neither stimuli images nor a precise diagram. Please correct the geometry, provide the actual layout, or re-run the evaluation under the intended setup.
- [Control Results / Figure 3] The compositional-deficit claim rests on the asserted product rule L2 = ToM × MR. This rule is not derived and is not a neutral baseline: the L2 task is four-way multiple choice while the ToM task is binary, and L2 adds the demand of recognizing which option corresponds to the transformed string. Any such added task demand will automatically produce L2 < ToM × MR even without an integration deficit. For example, with ToM = 1.0 and MR = 0.505, the product is 0.505, but a 0.339 L2 score is not evidence of a binding failure unless the product rule is justified or replaced with a matched-task baseline. Please provide a formal task model or empirical calibration; otherwise the 'deficit' is an artifact of task design.
minor comments (4)
- [Control Experiment] The ToM task is described as 'recognizing that another agent's view differs', but the question is actually a Level-1 visibility judgment ('Is the monkey seeing a different string...?'). This conflation should be acknowledged more explicitly, as it may overstate what the high ToM accuracy measures.
- [Results / Figure 3] The claim that MR performance is 'above chance' (mean 26.1% vs 25%) is not statistically supported. With only 28 items per model per task, the standard error for a single model is about 8 percentage points; a t-test or confidence interval across the 24 models is needed before concluding MR is above chance.
- [Model Evaluations] The selection criterion for the 24 models used in the control experiments is not described. Since these models drive the compositional-deficit analysis, a clear sampling procedure or a justification for the subset is needed to rule out selection bias.
- [Appendix A1] Figure A1 shows fluctuations in confusable errors of up to 12 percentage points across answer layouts. The claim that 'answer position has limited influence' is supported only by visual inspection; a statistical test (e.g., ANOVA or chi-square) would strengthen the claim. Also, the main text contains typos such as 'In this respectm' in the Introduction.
Circularity Check
No significant circularity: the central results are direct empirical measurements; the compositional-deficit baseline is an asserted comparison, not a fitted or self-cited input.
full rationale
The paper reports an empirical benchmark evaluation. The main claims (91.3% of models below chance; 75.88% egocentric errors; ToM 90.4%, MR 26.1%, L2 VPT 10.3%) are direct measurements of model outputs against fixed labels; no parameter is fitted and no equation is derived whose output equals an input. The egocentric option is deliberately the camera view, but that does not force models to select it, so the high egocentric rate is an observed result, not a construction artifact. The 'compositional deficit' compares observed L2 VPT to the asserted product-rule baseline ToM×MR. That baseline is a normative assumption, not fitted from the L2 data and not imported from a self-citation; the finding that L2 falls below it is a comparison, not a tautology. The baseline's appropriateness is debatable (e.g., it ignores differing chance levels and task demands), but that is a modeling/validity concern, not circularity. Several references in Related Work and Discussion are self-citations (Gao et al. 2024; Y. Li et al. 2025; B. Wang et al. 2025; Luo, Gao, & Deng 2025), but they are used for motivation or interpretation and are not load-bearing for the empirical claims. A serious construct-validity issue exists outside circularity: the Methods describe an upright opaque card with the monkey facing the blank back, which is inconsistent with ground-truth labels that require the monkey to see the 180°-rotated string; this threatens all reported numbers but is a correctness/geometry problem rather than a circular derivation.
Axiom & Free-Parameter Ledger
axioms (3)
- domain assumption The monkey, seated on the opposite side of an upright card, sees the 180°-rotated version of the printed string.
- ad hoc to paper If a model can integrate theory-of-mind and mental rotation, L2 VPT accuracy equals the product ToM × MR.
- domain assumption The 180° rotation of Latin digits/letters has a unique, well-defined ground truth (e.g., 'q'→'b', 'nod'→'pou') for all models.
read the original abstract
Visual perspective taking--inferring how the world appears from another's viewpoint--is foundational to social cognition. We introduce FlipSet, a diagnostic benchmark for Level-2 visual perspective taking (L2 VPT) in vision-language models. The task requires simulating 180-degree rotations of 2D character strings from another agent's perspective, isolating spatial transformation from 3D scene complexity. Evaluating 103 VLMs reveals systematic egocentric bias: the vast majority perform below chance, with roughly three-quarters of errors reproducing the camera viewpoint. Control experiments expose a compositional deficit--models achieve high theory-of-mind accuracy and above-chance mental rotation in isolation, yet fail catastrophically when integration is required. This dissociation indicates that current VLMs lack the mechanisms needed to bind social awareness to spatial operations, suggesting fundamental limitations in model-based spatial reasoning. FlipSet provides a cognitively grounded testbed for diagnosing perspective-taking capabilities in multimodal systems.
Figures
Forward citations
Cited by 5 Pith papers
-
DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving
DriveSpatial benchmark shows the best of 15 VLMs trails humans by 28.4 points on spatiotemporal driving tasks, with cognitive scene construction as the main failure mode.
-
DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving
DriveSpatial benchmark shows the strongest of 15 VLMs trails humans by 28.4 points on spatiotemporal tasks, with cognitive scene construction as the primary weakness.
-
Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images
MLLMs exhibit a large perception-reasoning gap on perspective-conditioned spatial reasoning in omnidirectional images, with accuracy falling from 57% on basic direction tasks to under 1% on compositional reasoning, th...
-
Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images
A new benchmark reveals MLLMs achieve only 13% or lower accuracy on advanced perspective-conditioned spatial tasks in omnidirectional images, with RL reward shaping raising a 7B model from 31% to 60% in controlled settings.
-
Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images
MLLMs display a large perception-reasoning gap on perspective-conditioned spatial reasoning tasks from omnidirectional images, with sharp accuracy drops on advanced tasks like egocentric rotation, though partial gains...
Reference graph
Works this paper leans on
-
[1]
Alayrac,J.-B.,Donahue,J.,Luc,P.,Miech,A.,Barr,I.,Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022). Flamingo: A visual language model for few- shot learning.Advances in neural information processing systems,35, 23716–23736
2022
-
[2]
Ye, S., Tian, H., Liu, Z., Gu, L., Wang, X., Li, Q., Ren, Y., Chen,Z.,Luo,J.,Wang,J.,Jiang,T.,Wang,B.,... Wang,W. (2025). Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. https://arxiv.org/abs/2412.05271
Pith/arXiv arXiv 2025
-
[3]
Cheng, K., Li, Y., Xu, F., Zhang, J., Zhou, H., & Liu, Y. (2024). Vision-language models can self-improve reasoning via reflection.arXiv preprint arXiv:2411.00855
Pith/arXiv arXiv 2024
-
[4]
Zhu, Y., Zhang, W., Luo, Z., Zhao, D., & Bing, L. (2024). Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. https://arxiv.org/abs/ 2406.07476
Pith/arXiv arXiv 2024
-
[5]
Edwards, K., & Low, J. (2019). Level 2 perspective-taking distinguishes automatic and non-automatic belief-tracking. Cognition,193, 104017
2019
-
[6]
Flavell, J. H. (2013). Perspectives on perspective taking. In Piaget’s theory(pp. 107–139). Psychology Press
2013
-
[7]
Fu, S., Bonnen, T., Guillory, D., & Darrell, T. (2025). Hidden in plain sight: Vlms overlook their visual representations. arXiv preprint arXiv:2506.08008
Pith/arXiv arXiv 2025
-
[8]
Gallese, V. (2007). Before and below ‘theory of mind’: Em- bodied simulation and the neural correlates of social cog- nition.Philosophical Transactions of the Royal Society B, 362(1480), 659–669
2007
-
[9]
Gallese, V., & Goldman, A. (1998). Mirror neurons and the simulation theory of mind-reading.Trends in Cognitive Sciences,2(12), 493–501
1998
-
[10]
Gao, Q., Li, Y., Lyu, H., Sun, H., Luo, D., & Deng, H. (2024). Visionlanguagemodelsseewhatyouwantbutnotwhatyou see.arXiv preprint arXiv:2410.00324
Pith/arXiv arXiv 2024
-
[11]
Gao, Q., Pi, X., Liu, K., Chen, J., Yang, R., Huang, X., Fang, X., Sun, L., Kishore, G., Ai, B., et al. (2025). Do vision- language models have internal world models? towards an atomic evaluation.arXiv preprint arXiv:2506.21876
Pith/arXiv arXiv 2025
-
[12]
Gunia, A., Moraresku, S., & Vlček, K. (2021). Brain mecha- nismsofvisuospatialperspective-takinginrelationtoobject mental rotation and the theory of mind.Behavioural Brain Research,407, 113247
2021
-
[13]
Jiang, Y., Wang, Y., Zhao, R., Parag, T., Chen, Z., Liao, Z., & Unnikrishnan, J. (2025). Videop2r: Video under- standing from perception to reasoning.arXiv preprint arXiv:2511.11113
Pith/arXiv arXiv 2025
-
[14]
Li, J., Li, D., Savarese, S., & Hoi, S. (2023). Blip-2: Boot- strapping language-image pre-training with frozen image encoders and large language models.CONFERENCE. Li,X.,Wang,Y.,&Feng,J.(2024).Spatial-vqa:Benchmarking spatial reasoning in vision–language models.Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP)...
arXiv 2023
-
[15]
Luo, D., Li, Y., & Deng, H. (2025). The philosophical foundations of growing ai like a child.arXiv preprint arXiv:2502.10742
arXiv 2025
-
[16]
Moll, H., & Meltzoff, A. N. (2011). How does it look? level 2 perspective-taking at 36 months of age.Child Development, 82(2), 661–673
2011
-
[17]
Park, J. S., Bhagavatula, C., Mottaghi, R., Farhadi, A., & Choi,Y.(2020).Visualcomet:Reasoningaboutthedynamic context of a still image. https://arxiv.org/abs/2004.10796
Pith/arXiv arXiv 2020
-
[18]
(1954).The construction of reality in the child
Piaget, J. (1954).The construction of reality in the child. Routledge
1954
-
[19]
(1977).The development of thought: Equilibration of cognitive structures
Piaget, J. (1977).The development of thought: Equilibration of cognitive structures. Viking Press
1977
-
[20]
Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision.arXiv preprint arXiv: 2103.00020
Pith/arXiv arXiv 2021
-
[21]
N., & Metzler, J
Shepard, R. N., & Metzler, J. (1971). Mental rotation of three-dimensional objects.Science,171(3972), 701–703
1971
-
[22]
K., Kaur, G., & Batra, D
Singh, Y. K., Kaur, G., & Batra, D. (2023). Rotated CLEVR: Assessing spatial reasoning of vision–language models. Proceedingsofthe17thPacificRimInternationalConference on Artificial Intelligence
2023
-
[23]
Sun, H., Gao, Q., Lyu, H., Luo, D., Li, Y., & Deng, H. (2024). Probingmechanicalreasoninginlargevisionlanguagemod- els.arXiv preprint arXiv:2410.00318
Pith/arXiv arXiv 2024
-
[24]
Sun, H., Yu, S., Li, Y., Gao, Q., Lyu, H., Deng, H., & Luo, D. (2025). Probing perceptual constancy in large vision language models.arXiv preprint arXiv:2502.10273
arXiv 2025
-
[25]
J., & Pinker, S
Tarr, M. J., & Pinker, S. (1989). Mental rotation and orientation-dependenceinshaperecognition.CognitivePsy- chology,21(2), 233–282
1989
-
[26]
Ramos, S., Yvinec, E., Casbon, M., Pot, E., Penchev, I., ... Hussenot, L. (2025). Gemma 3 technical report. https: //arxiv.org/abs/2503.19786 Wang,B.,Li,Y.,Zhou,Q.,Leong,H.Y.,Zhao,T.,Ye,L.,Deng, H., Luo, D., & Vasconcelos, N. (2025). Do vision language models infer human intention without visual perspective- taking? towards a scalable" one-image-probe-all...
Pith/arXiv arXiv 2025
-
[27]
Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., & Lin, J. (2024). Qwen2-vl:Enhancingvision-languagemodel’sperceptionof theworldatanyresolution.https://arxiv.org/abs/2409.12191 Wang,R.,Todd,G.,Xiao,Z.,Yuan,X.,Côté,M. -A.,Clark,P., &Jansen,P.(2024).Canlanguagemodelsserveastext-based world simulators?arXiv...
Pith/arXiv arXiv 2024
-
[28]
Xu, G., Jin, P., Wu, Z., Li, H., Song, Y., Sun, L., & Yuan, L. (2024). Llava-cot: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440
Pith/arXiv arXiv 2024
-
[29]
Xu, Z., Jain, S., & Kankanhalli, M. (2024). Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817
Pith/arXiv arXiv 2024
-
[30]
Yin, B., Wang, Q., Zhang, P., Zhang, J., Wang, K., Wang, Z., Zhang, J., Chandrasegaran, K., Liu, H., Krishna, R., et al. (2025). Spatial mental modeling from limited views. Structural Priors for Vision Workshop at ICCV’25
2025
-
[31]
Zacks, J. M. (2008). Neuroimaging studies of mental rotation: A meta-analysis and review.Journal of Cognitive Neuro- science,20(1), 1–19
2008
-
[32]
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019). From recognition to cognition: Visual commonsense reasoning. https://arxiv.org/abs/1811.10830
Pith/arXiv arXiv 2019
-
[33]
Zhang, J., Hu, J., Khayatkhoei, M., Ilievski, F., & Sun, M. (2024). Exploring perceptual limitation of multimodal large language models.arXiv preprint arXiv:2402.07384
Pith/arXiv arXiv 2024
-
[34]
Yang, Y., Pang, R., & Yang, Y. (2024). Improve vision language model chain-of-thought reasoning.arXiv preprint arXiv:2410.16198
Pith/arXiv arXiv 2024
-
[35]
Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., & Li, C. (2024). Video instruction tuning with synthetic data. https://arxiv.org/abs/2410.02713
Pith/arXiv arXiv 2024
-
[36]
Zhang, Z., Hu, F., Lee, J., Shi, F., Kordjamshidi, P., Chai, J., & Ma, Z. (2024). Do vision–language models represent space and how? evaluating spatial frame of reference under ambiguities. Zhao,X.,Malle,B.,&Gweon,H.(2016).Isitanine,orasix? prosocial and selective perspective taking in four-year-olds. Proceedings of the 38th Annual Meeting of the Cognitiv...
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.