REVIEW 4 major objections 4 minor 28 references
Through the Looking Glass: Common Sense Consistency Evaluation of Weird Images
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Through the Looking Glass (TLG) claims that a compact attention-pooling classifier over vision-language-generated atomic facts can detect commonsense-breaking images, reaching 73.54% on WHOOPS! and 87.57% on WEIRD.
desk verdict Solid incremental paper with a useful new dataset and an honest appendix that exposes the very marker-word confound that undercuts its headline mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the attention-pooling classifier over atomic facts: for an image $I$, an LVLM samples $N$ facts $F=\{f_1,\dots,f_N\}$; a frozen text encoder turns each fact into token hidden states $H_i$; masked average pooling condenses each fact to $V_i$; a learned attention head computes $A=\operatorname{softmax}(W_a V+b_a)$; the representations are combined as $v_{\text{weighted}}=\sum_i A_i V_i/\sum_i A_i$; and a logistic layer maps $v_{\text{weighted}}$ to the probability that the image breaks common sense. The attention weights carry the argument: TLG up-weights facts that violate common sense (e.g., 'The man is using a vacuum cleaner on the beach' in Figure 3), so the learned head can be read as a per-fact realism ranker.
What would settle it
Run TLG with all marker words ('weird', 'unreal', 'normal', 'real', 'digital', '3D', 'artistic' and their variants) deleted from the generated facts, or with a prompt that forbids evaluative language; if accuracy on WHOOPS! and WEIRD falls to near chance, the model was using lexical tell-tales rather than fact contradictions.
Extended reading notes
Core claim
The central discovery is that commonsense consistency of images can be judged in text space: when an LVLM is asked to state brief facts about a strange image, it produces a mixture of true facts and hallucinations, and that mixture is different from the one it produces for normal images. TLG learns this difference with a compact classifier. The pipeline generates $N$ atomic facts by diverse beam search, encodes each fact with a frozen text encoder, condenses each fact by masked average pooling, and trains an attention-pooling head plus a logistic layer to decide 'normal' versus 'strange'. In five-fold cross-validation under the WHOOPS! protocol, TLG reaches 73.54% on WHOOPS! and 87.57% on WEIRD, surpassing the fine-tuned BLIP2 FlanT5-XXL baseline (73.00% and 72.31%) and linear probing of a 13B LVLM (73.50% and 85.26%), and exceeding the human agreement measured on WEIRD (82.22%). The authors conclude that a compact classifier on textual fact representations is a better route than adapting an entire LVLM for anomaly detection.
Load-bearing premise
The load-bearing premise is that the classifier learns from contradictions among the generated atomic facts; if it is instead exploiting lexical markers that strange images tend to elicit—such as 'weird', 'digital', or '3D'—the method is not measuring common-sense consistency and would likely fail on a benchmark whose weird images do not trigger those words.
Editorial extensions
If this is right
- Fine-tuning an entire LVLM for anomaly detection is unnecessary: a compact head on frozen text representations reaches state of the art on both benchmark datasets.
- The learned attention weights can serve as an explicable per-fact strangeness score, allowing TLG to rank individual atomic facts by how much they strain common sense.
- TLG transfers across datasets: trained on one benchmark and tested on the other, it still exceeds most specialized baselines, with the largest gain when trained on the smaller WHOOPS! set and tested on WEIRD (83.05%).
- On WEIRD, TLG beats a proprietary zero-shot vision-language model by roughly six points, suggesting that a compact open-weight recipe can outperform a much larger closed system on this distribution.
Reading between the lines
- The paper does not control for lexical markers: Appendix D shows strange images elicit words like 'weird', 'unreal', 'digital', '3D', and 'artistic' far more often than normal images. If TLG is keying on those words rather than on fact contradictions, its accuracy would collapse on a benchmark whose strange images avoid such vocabulary.
- A cheap bag-of-words baseline over the same facts could rival TLG on these distributions; running it would separate lexical cueing from the relational consistency signal the paper argues for.
- If the consistency signal is real, TLG should also work as a hallucination gauge for generated captions: feed a model's own factual claims into the attention head and use its scores to flag fabricated statements, not just weird images.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Through the Looking Glass (TLG), a method for scoring whether an image violates common sense. For a given image, an LVLM generates a set of atomic facts via diverse beam search; the facts are encoded with a frozen DeBERTa-based text encoder; and a small attention-pooling classifier is fine-tuned to predict whether the image is normal or strange. The authors report 73.54% accuracy on the WHOOPS! benchmark and 87.57% on their newly introduced WEIRD benchmark, claiming state-of-the-art performance on both, and they release the WEIRD dataset, which contains 824 human-filtered image pairs.
Significance. If the central claim is upheld, the paper has clear practical value: it shows that a compact classifier over LVLM-generated textual facts can beat full-model fine-tuning and zero-shot proprietary systems on this task, with a modest parameter count and released code and data. The diversity-based fact generation and the use of an NLI-tuned text encoder are simple, reproducible design choices, and the paper compares against a broad set of baselines. However, the significance is currently conditional: the paper's own Appendix D documents a strong lexical-marker imbalance between normal and strange images, and no control experiment shows that the classifier relies on semantic contradictions rather than on these markers. The WHOOPS! state-of-the-art margin is also very small and not significance-tested. These gaps must be closed before the method can be accepted as a common-sense consistency measure.
major comments (4)
- [Appendix D, Tables 6 and 7; Section 4] The marker-word imbalance is a direct threat to the central claim that TLG detects common-sense inconsistency among atomic facts. For the main backbone, LLaVA-1.6-Mistral-7B on WHOOPS!, the marker 'weird' appears 12 times in facts from strange images versus 1 time in facts from normal images, and 'digital' appears 68 versus 37 times; on WEIRD the corresponding counts are 57 versus 1 and 270 versus 201. Since TLG is trained on exactly these facts with normal/strange labels, the DeBERTa encoder can solve the task by exploiting these lexical cues without performing any consistency reasoning. The paper should include control experiments, such as removing or masking marker words, training a marker-only classifier, and reporting accuracy on marker-free facts. Without such controls, the claim that TLG measures 'common sense consistency' is not supported.
- [Section 6, Table 2] The claim of state-of-the-art performance on WHOOPS! rests on a 0.54-percentage-point margin (73.54% for TLG versus 73.00% for BLIP2 FlanT5-XXL) on a 204-sample dataset evaluated with 5-fold cross-validation. No standard deviations or significance tests are reported in the main text; Appendix A only provides a figure without numerical values. Given the small sample size, this margin is likely within the noise. The authors should report per-fold accuracies and a paired significance test, and should temper the 'state-of-the-art' claim on WHOOPS! unless it is statistically supported.
- [Appendix F, Table 8; Section 6] The fact generator and text encoder are selected on the same benchmarks used for the headline results. The paper states that the highest performance on both benchmarks was achieved with LLaVA-1.6-Mistral-7B and deberta-v3-large-tasksource-nli, and then reports these exact numbers as TLG's performance. This means the reported accuracies are the result of model selection on the evaluation data, making them optimistic and weakening the comparison with baselines that did not undergo the same selection procedure. An independent validation split or nested cross-validation is needed to obtain unbiased estimates.
- [Section 4, Eqs. (2)-(5); Section 2] The method as described does not actually compute contradictions among atomic facts. The attention-pooling classifier in Eqs. (2)-(5) maps each fact to a scalar, softmax-normalizes the scalars, and forms a weighted average of fact representations; there is no pairwise comparison, no entailment check, and no explicit contradiction signal. The paper's framing in Section 2, that the classifier identifies 'high contradiction among atomic facts', is therefore not reflected in the architecture. Figure 3 shows high attention weights for lexically strange statements, but this is anecdotal and does not establish that contradiction between facts drives the decision. The authors should either implement an explicit contradiction-based aggregation or provide an analysis showing that the learned attention is causally tied to fact-level inconsistency.
minor comments (4)
- [Section 5, Experimental Setup] The number of atomic facts N per image is never specified, despite being a central hyperparameter of the method; this should be stated explicitly for reproducibility.
- [Figure 3, Eq. (3)] Figure 3 displays attention scores that include negative values (e.g., -0.28), but Eq. (3) defines attention weights as the output of a softmax, which are nonnegative. Please clarify whether the figure shows pre-softmax logits or attention weights.
- [Appendix D, Table 7] The legend of Table 7 should define whether the marker-word counts are per image or aggregated over all facts, and the column header should align with the category names in Table 6; as written, the counts are hard to interpret quantitatively.
- [Section 5, Hyperparameters] The diversity_penalty is said to be set to 1.0 because it is 'optimal', but no search range or selection criterion is reported; a brief note on how this value was chosen would improve reproducibility.
Circularity Check
No derivational circularity: TLG is a supervised classifier over LVLM-generated facts; the only self-citation is in related work and is not load-bearing, while the marker-word and model-selection caveats are validity risks, not Eq.-level circularity.
full rationale
TLG's derivation chain is a standard supervised pipeline: an LVLM generates atomic facts, a frozen DeBERTa encoder produces token representations, and a learned attention-pooling classifier maps the pooled representation to a weird/normal probability (Eqs. 1-5). The training labels come from WHOOPS! and the authors' WEIRD dataset, and the reported numbers are 5-fold cross-validation accuracies. No fitted parameter is renamed as a prediction, and no result is defined in terms of the target label. The method therefore does not reduce to its inputs by construction. The paper cites Rykov et al. (2025), an overlapping-author preprint, but only in Related Work as a similar atomic-fact approach ('Rykov et al. (2025) proposed an approach, in which LVLM is used to first generate atomic facts... Our approach is similar to the preceding methods'). The present contribution replaces pairwise NLI aggregation with a trained attention classifier, so the self-citation is not load-bearing and no uniqueness theorem or ansatz is imported from it. Two caveats are worth stating explicitly but they are not circularity. First, Appendix D documents a marker-word imbalance: Table 7 shows that strange images generate many more 'weird', 'digital', '3D', and 'artistic' markers, and the paper itself notes 'the presence of digital markers can be misleading for the model.' This is a potential shortcut/confound for the supervised classifier, but exploiting a distributional cue is a generalization and validity risk, not a circular derivation. Second, Table 8 selects the best LVLM/encoder pair on the same two benchmarks used for the headline accuracy, which introduces optimistic selection bias. Again, this is a statistical reporting concern rather than an equation-level equivalence. Overall, no specific circular step can be exhibited, so the paper receives a low score reflecting only the minor, non-load-bearing self-citation.
Assumptions & free parameters
free parameters (3)
- diversity_penalty =
1.0
- number_of_atomic_facts N =
not reported
- TLG classifier hyperparameters (learning rate, epochs, batch size) =
not reported
assumptions (3)
- domain assumption LVLM-generated atomic facts for an image carry a signal that distinguishes common-sense-consistent from strange images.
- domain assumption The human-annotated weird/normal labels on WHOOPS! and WEIRD are a valid ground truth for common sense consistency.
- domain assumption Frozen DeBERTa-v3-large text representations of facts are sufficient inputs for the classifier.
Cite this review
Pith. "Pith review of Through the Looking Glass: Common Sense Consistency Evaluation of Weird Images." pith.science (2026). https://pith.science/paper/Z46ZTLX3
@misc{pith2026250507704,
author = {Pith},
title = {Pith review of: Through the Looking Glass: Common Sense Consistency Evaluation of Weird Images},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z46ZTLX3}},
note = {Machine review of arXiv:2505.07704}
}
read the original abstract
Measuring how real images look is a complex task in artificial intelligence research. For example, an image of a boy with a vacuum cleaner in a desert violates common sense. We introduce a novel method, which we call Through the Looking Glass (TLG), to assess image common sense consistency using Large Vision-Language Models (LVLMs) and Transformer-based encoder. By leveraging LVLMs to extract atomic facts from these images, we obtain a mix of accurate facts. We proceed by fine-tuning a compact attention-pooling classifier over encoded atomic facts. Our TLG has achieved a new state-of-the-art performance on the WHOOPS! and WEIRD datasets while leveraging a compact fine-tuning component.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...
-
[2]
Bowman, Gabor Angeli, Christopher Potts, and Christopher D
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://doi.org/10.18653/V1/D15-1075 A large annotated corpus for learning natural language inference . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015 , pages 632--642. The Ass...
-
[3]
Gonzalez, Ion Stoica, and Eric P
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\
2023
-
[5]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \' e lio Renard Lavaud, Marie - Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth \' e e Lacroix, and William El Sayed. 2023. https://doi.org/...
-
[6]
Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du. 2023. https://doi.org/10.48550/ARXIV.2311.01477 FAITHSCORE: evaluating hallucinations in large vision-language models . CoRR, abs/2311.01477
-
[7]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023 a . https://proceedings.mlr.press/v202/li23q.html BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machin...
2023
-
[8]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023 b . https://proceedings.mlr.press/v202/li23q.html BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machin...
2023
-
[9]
Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics
2004
Show all 28 references
-
[10]
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . https://doi.org/10.1109/CVPR52733.2024.02484 Improved baselines with visual instruction tuning . In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , p...
2024
- [11]
-
[12]
Ivan Maksimov, Vasily Konovalov, and Andrei Glinskii. 2024. https://doi.org/10.18653/v1/2024.semeval-1.42 D eep P avlov at S em E val-2024 task 6: Detection of hallucinations and overgeneration mistakes with an ensemble of transformer-based models . In Proceedings of the 18th ...
2024 doi
-
[13]
Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023 a . https://doi.org/10.18653/V1/2023.IJCNLP-MAIN.4 MQAG: multiple-choice question answering and generation for assessing information consistency in summarization . In Proceedings of the 13th International Joint Confere...
2023 doi
-
[14]
Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023 b . https://doi.org/10.18653/V1/2023.EMNLP-MAIN.557 Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models . In Proceedings of the 2023 Conference on Empirical Methods in Nat...
2023 doi
-
[15]
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...
2023 doi
-
[16]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. http://proceedings.mlr.press/v139/radford21a.html Learning transferable visual models...
2021
-
[17]
Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/V1/D19-1410 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on ...
2019 doi
-
[18]
Elisei Rykov, Kseniia Petrushina, Kseniia Titova, Alexander Panchenko, and Vasily Konovalov. 2025. https://arxiv.org/abs/2503.15948 Don't fight hallucinations, use them: Estimating image realism using nli over atomic facts . Preprint, arXiv:2503.15948
2025 arXiv
-
[19]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022....
2022
-
[20]
Damien Sileo. 2024. https://aclanthology.org/2024.lrec-main.1361 tasksource: A large collection of NLP tasks with a structured dataset preprocessing framework . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalu...
2024
-
[21]
Gemma Team. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma
2024 doi
-
[22]
Vijayakumar, Michael Cogswell, Ramprasaath R
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2016. https://arxiv.org/abs/1610.02424 Diverse beam search: Decoding diverse solutions from neural sequence models . CoRR, abs/1610.02424
2016 arXiv
-
[23]
Smith, Daniel Khashabi, and Hannaneh Hajishirzi
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.754 Self-instruct: Aligning language models with self-generated instructions . In Proceedings of the 61st Annual Mee...
2023 doi
-
[24]
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. http://aclweb.org/anthology/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computationa...
2018
- [25]
-
[26]
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.1109/CVPR.2019.00688 From recognition to cognition: Visual commonsense reasoning . In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019...
2019
-
[27]
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. https://doi.org/10.1109/ICCV51070.2023.01100 Sigmoid loss for language image pre-training . In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 11...
2023
-
[28]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[29]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.