Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports that even the strongest tested multimodal model detects only 26% of subtle "cheating" clues and invents clues on 92% of innocent photos, marking a concrete boundary of MLLM perception and reasoning.

desk verdict A genuinely new and practically relevant MLLM stress test, but the headline numbers rest on annotations and a human baseline that need better documentation before I'd trust the exact boundary. read the letter →

arxiv 2507.00045 v1 pith:NCVUURDF submitted 2025-06-23 cs.CV cs.AIcs.CL

Ming Li , Chenguang Wang , Yijun Liang , Xiyao Wang , Yuhang Zhou , Xiyang Wu , Yuqing Zhang , Ruiyi Zhang
show 1 more author
Tianyi Zhou
This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords CaughtCheatingmultimodallargelanguagemodelsvisualperceptionreasoningsuspiciouscluedetectionGuidedSearchtheoryhallucinationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether multimodal large language models (MLLMs) can play detective: look at an ordinary photo and notice a tiny, context-dependent clue that contradicts the story attached to it. To test this, the authors built CaughtCheating, a 100-photo benchmark of real social-media images in which a partner's claim is accompanied either by a subtle "deterministic" clue or by no clue at all. Their experiments report that every model they tried fails badly: the strongest, o3, detects the deterministic clue in only 26.0% of clued images and in only 8.0% of unclued images correctly says there is nothing suspicious, giving an overall F1 of 23.9%, while a human comparison reaches F1 59.6%. The paper's point is that near-ceiling scores on existing benchmarks conceal a real boundary: MLLMs cannot yet reliably perform open-ended search for unspecified, low-salience clues, and they hallucinate accusations on innocent images. If the measurements are right, CaughtCheating marks a concrete, practically relevant limit of current visual perception and reasoning.

What carries the argument

The load-bearing object is the CaughtCheating benchmark itself: 100 real photographs, balanced 50/50 between images with a hidden deterministic clue and images without one, annotated with a primary claim-violation question, the deterministic clue, weaker non-deterministic clues, and decomposed perception and reasoning questions. The explanatory machinery is Guided Search theory from cognitive science, which says visual search is steered by bottom-up salience, top-down feature guidance, and scene structure and meaning. The paper uses those three factors to explain why o3's exhaustive grid-search reasoning traces succeed on Waldo-style tasks but collapse on CaughtCheating: the clue is low-salience, there is no predefined template telling the model what to look for, and interpreting the clue requires social reasoning about what contradicts a stated claim.

What would settle it

Run a controlled human study on the same 100 images with at least 30 naive participants, no hints, and a pre-registered marking rule; if human F1 lands near the 20–25% range instead of 59.6%, the claimed human–model gap collapses. Independently, have fresh annotators label each clued image's deterministic clue and check agreement; low agreement would mean the accuracy scores are measured against a ground truth the authors themselves could not reliably reproduce.

Watch

Extended reading notes

Core claim

The central claim is that open-ended detection of subtle, context-dependent visual clues is beyond current agentic MLLMs, even when the same models solve closed tasks like locating Waldo, finding a camouflaged snake, or geolocating a photo. On CaughtCheating, o3 achieves 26.0% accuracy and 17.2% IoU on clued images and 8.0% accuracy on unclued images, for an F1 of 23.9%; the reported human reference is 56.0% clued accuracy, 63.6% unclued accuracy, and F1 59.6%. The paper also shows through decomposed questions that models often identify the clue when it is explicitly pointed out, yet fail to find it when the query does not name it, and that the strongest models invent suspicious clues on innocent photographs. The authors interpret this pattern as a missing "detective" ability: MLLMs rely on exhaustive, region-by-region search, which fails when the target is invisible to bottom-up salience, has no known appearance to guide top-down attention, and draws its meaning from social context.

Load-bearing premise

The results stand on hand-assigned labels about which images contain a "deterministic" clue and which are genuinely unclued, together with a human F1 of 59.6% that is reported without a description of who the participants were, how many there were, what instructions they received, or whether hints were allowed.

Editorial extensions

If this is right

  • Near-ceiling performance on existing multimodal benchmarks should not be read as detective-level perception, because tasks that name the target reward recognition rather than open-ended search.
  • A model that can identify a clue once told where to look cannot automatically find it in a free-form query; search guidance is a separate capability from perception and reasoning.
  • Agentic test-time scaling, such as o3's extended reasoning traces, does not by itself solve low-salience, unspecified-target detection.
  • Practical uses of MLLMs for photo verification, safety screening, or "spot the problem" applications would inherit both failure modes measured here: missed real clues and false accusations on innocent images.
  • Because the decomposed perception questions are answered far more accurately than the open question, a plausible route to improvement is teaching models to generate candidate hypotheses about context before scanning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the benchmark's 100 images are hand-selected and 50 of them lightly cropped to hide a visible person, so the measured difficulty may be partly a property of minimal image editing rather than of naturally occurring hidden clues; a larger set of uncropped, naturally ambiguous photos would test that.
  • Editorial inference: the paper's own limitation section notes that public social-media sourcing centers cisgender, heterosexual couples and hotel/dining scenes, so the "boundary" demonstrated is for one culturally specific form of social reasoning; other relationship norms or workplace/nightlife settings could behave differently.
  • Editorial inference: the strong models' low unclued accuracy (o3 at 8.0%) conflates two failures—perceptual hallucination and a cautious/refusal style that keeps generating possible suspicions—and the paper notes safety refusals on gender-related decomposed questions; separating refusal from perception needs neutral rephrased queries.
  • Editorial inference: a two-stage architecture that first proposes many candidate clues and then verifies each with an explicit "does this contradict the claim" step is a direct, testable extension suggested by the decomposed-perception result, and would predict a much higher clued accuracy if search guidance is the bottleneck.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces CaughtCheating, a benchmark of 100 real-world social-media photographs (50 'Clued' images with subtle suspicious clues and 50 'Unclued' images) designed to test whether agentic multimodal large language models (MLLMs) can detect context-dependent visual clues that contradict a stated claim, such as 'dining alone.' The authors evaluate a wide range of open-source and proprietary MLLMs and report that even the best model, GPT-o3, achieves only 26.0% Clued Accuracy and 17.2% Clued IoU, while unreported human participants reach about 56.0% Clued Accuracy. They also report that strong models frequently hallucinate suspicious clues on unclued images, and they use decomposed perception/reasoning questions and Guided Search theory to explain the failures. The data and code are publicly released.

Significance. If the quantitative results are reliable, CaughtCheating fills a real gap: existing benchmarks largely test object recognition or well-defined reasoning, whereas this benchmark requires open-ended search for initially unspecified, socially contextual clues, a task with direct practical relevance. The reported hallucination behavior on unclued images is a useful and falsifiable finding that matters for real deployment. The paper's strengths include public data/code, a judge-validation step (Cohen's kappa 0.82 and 0.943 against human evaluators), decomposed questions that separate perceptual from reasoning failures, and detailed o3 reasoning traces. The qualitative direction — that current agentic MLLMs are far below human-level performance on this task — is plausible and important. However, the exact boundary numbers are supported only weakly, because the gold clue labels lack inter-annotator validation, the human baseline is undocumented and internally inconsistent, and the sample size is small with no confidence intervals.

major comments (4)
  1. [Section 3.2 / Appendix B.2] The gold 'deterministic clue' labels and the 'unclued' ground-truth labels are load-bearing for the headline numbers, but the manuscript reports no inter-annotator agreement on these labels. The Cohen's kappa values in Section 3.4 (0.82 for Clued Acc and 0.943 for Unclued Acc) validate the GPT-4.1 judge against human evaluators; they do not validate the correctness of the deterministic clue selection or the absence-of-clue determination themselves. With only 50 clued images and highly context-dependent social cues, a different annotation team might plausibly select a different deterministic clue, and a model naming a different but equally decisive clue would be scored incorrect. Please add a second independent annotation pass (or a detailed annotation protocol with agreement statistics) and report agreement on the clue labels, not just on the final binary judgments.
  2. [Table 1 / Appendix C] The human row in Table 1 is internally inconsistent. Under the transformation in Appendix C, Clued Acc=56.0 and Unclued Acc=63.6 (with N_clued=N_unclued=50) imply Precision = 56/(56+18.2) ≈ 60.6 and F1 ≈ 58.2, but the table reports Precision=56.0 and F1=59.6. In addition, the paper does not report the number of human participants, their recruitment, the task instructions, or whether hints were allowed, although Figure 1 and Section 4.2 imply different hint conditions. Because Section 4.1 uses the human row to place the 'boundary' of MLLM capabilities, the human baseline must be documented fully and the reported P/R/F1 values corrected or derived from raw human judgments.
  3. [Section 4.1] The central quantitative claims, including 'GPT-o3 only achieves 26.0% accuracy and 17.2% IoU,' are computed from 100 images (50 clued, 50 unclued) with no confidence intervals, no repeated-run variance, and no significance tests. For the best models, a one-sample difference (26.0 vs 20.0 for Gemini-2.5-pro) is within the expected sampling noise for N=50, so the exact ordering and the precision of the word 'boundary' are not supported. Please report bootstrap confidence intervals or exact binomial intervals for the main metrics, and soften claims that distinguish between near-ceiling models.
  4. [Section 3.4 / Appendix D] The GPT-4.1 judge is validated for Clued Acc and Unclued Acc, but no human-model agreement is reported for the Clued IoU metric, which depends on a two-stage extraction-and-matching pipeline (Figures 9 and 10 of Appendix D). Since the 17.2% IoU figure is part of the central result, the IoU evaluation should either be validated against human annotations of the non-deterministic clue sets or reported with a caveat that its reliability is unmeasured.
minor comments (5)
  1. [Abstract and Section 1] The abstract and introduction contain typos ('acheived', 'Butcan', 'refelction', 'femine', 'Uncledimage'); please proofread the text.
  2. [Appendix B.3] Table 3 uses 'with-clue' and 'without-clue' while the main text uses 'Clued' and 'Unclued'; please unify the terminology across the manuscript.
  3. [Section 5] In case (c), 'an Uncledimage' should be 'an Unclued image' and 'really suspecious' should be 'really suspicious'.
  4. [Appendix A and Section 1] The appendix title 'Relate Work' should be 'Related Work', and the phrase 'detection-level benchmark' in Section 1 appears to be a typo for 'detective-level benchmark'.
  5. [Figure 6 caption] The caption 'The clue, such as a scattered lipstick, are clearly visible' should be 'The clues, such as a scattered lipstick, are clearly visible'.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical benchmark with hand-annotated labels and a human-validated automatic judge; no load-bearing step reduces to its own inputs.

full rationale

CaughtCheating is an empirical benchmark paper rather than a derivation. The central claim in Section 4.1 ('even the best performing model, GPT-o3, only achieves 26.0% accuracy and 17.2% IoU') is a measured outcome: open-ended MLLM responses are parsed by GPT-4.1 and compared with hand-annotated ground-truth clues. The paper validates its automatic judge against human evaluators, reporting 'kappa scores of 0.82 for Clued Acc and 0.943 for Unclued Acc' (Section 3.4), so the evaluation is an independent proxy rather than a self-certifying loop. The metric formulas in Appendix C are definitions used uniformly across models and humans; nothing is fitted to a subset of the data and then renamed a prediction. The Guided Search theory is imported from external cognitive-science sources and serves as post-hoc interpretation, not as evidence for the benchmark numbers. There are no load-bearing self-citations and no imported uniqueness theorem. The weaknesses noted by the reader - no inter-annotator agreement on the gold clue labels and an undocumented, internally inconsistent human baseline (a 56.0/63.6 human row implies F1 approximately 58.2 under the paper's own Appendix C transformation, not the reported 59.6) - are validity and reporting concerns about how well the numbers support the boundary claim, not circularity in the paper's derivation. Even if those concerns are correct, the model scores would not be equivalent to the inputs by construction. Hence no significant circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are introduced; the paper makes no parametric derivation. It does rely on domain assumptions: that the hand-authored clue labels are correct ground truth, that the reported human baseline is a valid reference despite no protocol details, and that Guided Search theory transfers from human attention to MLLM behavior. These assumptions are reasonable but not independently established in the preprint.

assumptions (3)
  • domain assumption The five-factor Guided Search theory of human visual attention transfers to MLLM perception and explains their failures.
    Section 2.2 applies Wolfe et al. (1989) to model behavior without quantitative validation that MLLM attention follows these factors.
  • domain assumption The hand-authored deterministic and non-deterministic clue labels, along with the unclued labels, are treated as objective ground truth.
    Section 3.2 and Section 3.4 make metric validity depend on these labels, but no inter-annotator agreement on the labeling step is reported.
  • domain assumption The reported human performance on the 100 images is a valid reference baseline.
    Table 1 reports human Clued Acc 56.0, Unclued Acc 63.6, and F1 59.6 without describing the human evaluation protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning." pith.science (2026). https://pith.science/paper/NCVUURDF

@misc{pith2026250700045,
  author       = {Pith},
  title        = {Pith review of: CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCVUURDF}},
  note         = {Machine review of arXiv:2507.00045}
}
read the original abstract

Recent agentic Multi-Modal Large Language Models (MLLMs) such as GPT-o3 have achieved near-ceiling scores on various existing benchmarks, motivating a demand for more challenging test tasks. These MLLMs have been reported to excel in a few expert-level tasks for humans, e.g., GeoGuesser, reflecting their potential as a detective who can notice minuscule cues in an image and weave them into coherent, situational explanations, leading to a reliable answer. But can they match the performance of excellent human detectives? To answer this question, we investigate some hard scenarios where GPT-o3 can still handle, and find a common scenario where o3's performance drops to nearly zero, which we name CaughtCheating. It is inspired by the social media requests that ask others to detect suspicious clues from photos shared by the poster's partner. We conduct extensive experiments and analysis to understand why existing MLLMs lack sufficient capability to solve this kind of task. CaughtCheating provides a class of challenging visual perception and reasoning tasks with great value and practical usage. Success in these tasks paves the way for MLLMs to acquire human-level detective perception and reasoning capabilities.

Figures

Figures reproduced from arXiv: 2507.00045 by the authors.

Figure 1
Figure 1. An example from CaughtCheating. Query: “My boyfriend said he’s dining alone at the restaurant and sent me this photo. Do you notice anything suspicious in this image that contradicts his claim?” Suspicious Clue: “There are other people, including a girl with long hair, visible in the spoon’s reflection.” In this example, most human participants, and the strong o3 are not capable of identifying this clue when not giv… view at source ↗
Figure 2
Figure 2. Demonstration of GPT-o3’s multimodal visual-reasoning breadth. (a) Visual search: locating Waldo in a densely populated illustration. (b) Visual search for camouflage: spotting a nearly invisible copperhead snake hidden among dry leaves. (c) GeoGuessr: identifying the upper terminal of Chair 1 at New Mexico, and estimating its latitude/longitude from a single image. (d) TimeGuesser: dating the photograph by matching… view at source ↗
Figure 3
Figure 3. An example of the annotation for the "Clued" category. Each image is annotated with a general question assessing overall suspicion and decomposed questions focused on a deterministic clue (here, the feminine bow hair accessory). Decomposed questions include perception-based inquiries (clue identification) and reasoning￾based inquiries (social implications and contradictions), all annotated with the expected answer "… view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Case studies of the models’ performance on the CaughtCheating examples. 3 representative models are selected, including GPT-o3, GPT-4o and InternVL2.5-1B, and 3 images are selected: (a) A difficult Clued image, (b) An easy Clued image, and (c) An Unclued image. The mod…
Figure 5
Figure 5. Figure 5: Example of cropping an image for with-clue category. The original photo shows part of the person sitting on the sofa (Before). By cropping (After), we can still infer there is a person, but identifying the clue is more subtle and challenging for MLLM. To ensure quality…
Figure 6
Figure 6. Figure 6: Example of generated image. The clues, such as a scattered lipstick, are clearly visible in the image. The clue is obvious rather than subtle, making the inference extremely unchallenging for MLLMs. image that contradicts [her/his] claim?” Among the potential clues, th…
Figure 7
Figure 7. Figure 7: The prompt used to request GPT4.1 to measure whether MLLMs successfully identify the key deterministic [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: The prompt used to request GPT4.1 to assess whether MLLMs correctly determine the absence of [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: The prompt used to request GPT4.1 to convert MLLM’s answer into a key points list for the following [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: The prompt used to request GPT4.1 to evaluate whether MLLMs identify each relevant non-deterministic [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: The prompt used to request GPT4.1 to evaluate whether the MLLM’s answer agrees with the expected [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: OpenAI o3 full reasoning process for Figure [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: OpenAI o3 full reasoning process for [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: OpenAI o3 full reasoning process for [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: OpenAI o3 full reasoning process for [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: OpenAI o3 full reasoning process for [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 17
Figure 17. Figure 17: OpenAI o3 full reasoning process for [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]
Figure 18
Figure 18. Figure 18: OpenAI o3 full reasoning process for Figure [PITH_FULL_IMAGE:figures/full_fig_p032_18.png]
Figure 19
Figure 19. Figure 19: OpenAI o3 full reasoning process for Figure [PITH_FULL_IMAGE:figures/full_fig_p033_19.png]
Figure 20
Figure 20. Figure 20: OpenAI o3 full reasoning process for Figure [PITH_FULL_IMAGE:figures/full_fig_p034_20.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A multilingual, multi-page document retrieval benchmark with 35K+ QA pairs shows MLLM retrievers lead but still fail on tables and low-resource languages.

Reference graph

Works this paper leans on

89 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. https://arxiv.org/abs/2502.13923 Qwen2.5-vl technical report . Preprint, arXiv:2502.13923

    arXiv 2025

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  3. [3]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024 a . Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330

    arXiv 2024

  4. [4]

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. 2024 b . M3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. arXiv preprint arXiv:2405.16473

    arXiv 2024

  5. [5]

    Ruxiao Chen, Chenguang Wang, Yuran Sun, Xilei Zhao, and Susu Xu. 2025. From perceptions to decisions: Wildfire evacuation decision prediction with behavioral theory-informed llms. arXiv preprint arXiv:2502.17701

    arXiv 2025

  6. [6]

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, and 1 others. 2024 c . Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187

    arXiv 2024

  7. [7]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024 d . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

    arXiv 2024

  8. [8]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024 e . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185--24198

    2024

Show all 89 references
  1. [9]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  2. [10]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  3. [11]

    Edirlei Soares de Lima, Marco A Casanova, Bruno Feij \'o , and Antonio L Furtado. 2025. Characterizing the investigative methods of fictional detectives with large language models. arXiv preprint arXiv:2505.07601

    2025 arXiv

  4. [12]

    Google DeepMind. 2025 a . https://deepmind.google/technologies/gemini/flash/ Gemini 2.0 flash

    2025

  5. [13]

    Google DeepMind. 2025 b . https://storage.googleapis.com/model-cards/documents/gemini-2.5-pro-preview.pdf Gemini 2.5 pro preview model card . Technical report, Google DeepMind

    2025

  6. [14]

    Huilin Deng, Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, and Yu Kang. 2025 a . Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning. arXiv preprint arXiv:2503.07065

    2025 arXiv

  7. [15]

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. 2025 b . Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement. arXiv preprint arXiv:2503.17352

    2025 arXiv

  8. [16]

    Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, Quanquan Gu, James Y Zou, Kai-Wei Chang, and Wei Wang. 2024. Enhancing large vision language models with self-training on image comprehension. Advances in Neural Information Processing Systems, 37:131369--131397

    2024

  9. [17]

    John Duncan and Glyn W Humphreys. 1989. Visual search and stimulus similarity. Psychological review, 96(3):433

    1989

  10. [18]

    Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. 2025. Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill? arXiv preprint arXiv:2504.06514

    2025 arXiv

  11. [19]

    Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, and Cha Zhang. 2025. Refocus: Visual editing as a chain of thought for structured image understanding. arXiv preprint arXiv:2501.05452

    2025 arXiv

  12. [20]

    Kanishk Gandhi, Jan-Philipp Fr \"a nken, Tobias Gerstenberg, and Noah Goodman. 2023. Understanding social reasoning in language models with language models. Advances in Neural Information Processing Systems, 36:13518--13529

    2023

  13. [21]

    Zhouhong Gu, Lin Zhang, Jiangjie Chen, Haoning Ye, Xiaoxuan Zhu, Zihan Li, Zheyu Ye, Yan Gao, Yao Hu, Yanghua Xiao, and 1 others. 2023. Piecing together clues: A benchmark for evaluating the detective skills of large language models. arXiv preprint arXiv:2307.05113

    2023 arXiv

  14. [22]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  15. [23]

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. 2025. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444

    2025 arXiv

  16. [24]

    Jingyuan Huang, Jen-tse Huang, Ziyi Liu, Xiaoyuan Liu, Wenxuan Wang, and Jieyu Zhao. 2025 a . Vlms as geoguessr masters: Exceptional performance, hidden biases, and privacy risks. arXiv preprint arXiv:2502.11163

    2025 arXiv

  17. [25]

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. 2025 b . Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749

    2025 arXiv

  18. [26]

    Laurent Itti and Christof Koch. 2001. Computational modelling of visual attention. Nature reviews neuroscience, 2(3):194--203

    2001

  19. [27]

    Laurent Itti, Christof Koch, and Ernst Niebur. 2002. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence, 20(11):1254--1259

    2002

  20. [28]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  21. [29]

    Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, and 1 others. 2024. Graph chain-of-thought: Augmenting large language models by reasoning on graphs. arXiv preprint arXiv:2404.07103

    2024 arXiv

  22. [30]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

    2025 arXiv

  23. [31]

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern reco...

    2017

  24. [32]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  25. [33]

    Seungpil Lee, Woochang Sim, Donghyeon Shin, Wongyu Seo, Jiwon Park, Seokki Lee, Sanha Hwang, Sejin Kim, and Sundong Kim. 2024. Reasoning abilities of large language models: In-depth analysis on the abstraction and reasoning corpus. ACM Transactions on Intelligent Systems and T...

    2024

  26. [34]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . https://arxiv.org/abs/2408.03326 Llava-onevision: Easy visual task transfer . Preprint, arXiv:2408.03326

    2024 arXiv

  27. [35]

    Ming Li, Han Chen, Chenguang Wang, Dang Nguyen, Dianqi Li, and Tianyi Zhou. 2024 b . Ruler: Improving llm controllability by rule-based data recycling. arXiv preprint arXiv:2406.15938

    2024 arXiv

  28. [36]

    Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. 2024 c . Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning. In Findings of the Association for Computational Linguistics ACL 2024, pages 16189--16211

    2024

  29. [37]

    Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, Yupeng Hou, Fuxiao Liu, and Tianyi Zhou. 2024 d . Mosaic-it: Free compositional data augmentation improves instruction tuning. arXiv preprint arXiv:2405.13326

    2024 arXiv

  30. [38]

    Ming Li, Yanhong Li, Ziyue Li, and Tianyi Zhou. 2025 a . How instruction and reasoning data shape post-training: Data quality through the lens of layer-wise gradients. arXiv preprint arXiv:2504.10766

    2025 arXiv

  31. [39]

    Ming Li, Yanhong Li, and Tianyi Zhou. 2024 e . What happened in llms layers when trained for fast vs. slow thinking: A gradient perspective. arXiv preprint arXiv:2410.23743

    2024 arXiv

  32. [40]

    Ming Li, Zhengyuan Yang, Xiyao Wang, Dianqi Li, Kevin Lin, Tianyi Zhou, and Lijuan Wang. 2025 b . What makes reasoning models different? follow the reasoning leader for efficient decoding. arXiv preprint arXiv:2506.06998

    2025 arXiv

  33. [41]

    Ming Li, Ruiyi Zhang, Jian Chen, Jiuxiang Gu, Yufan Zhou, Franck Dernoncourt, Wanrong Zhu, Tianyi Zhou, and Tong Sun. 2025 c . Towards visual text grounding of multimodal large language model. arXiv preprint arXiv:2504.04974

    2025

  34. [42]

    Yijun Liang, Ming Li, Chenrui Fan, Ziyue Li, Dang Nguyen, Kwesi Cobbina, Shweta Bhardwaj, Jiuhai Chen, Fuxiao Liu, and Tianyi Zhou. 2025. Colorbench: Can vlms see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness. arXi...

    2025

  35. [43]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. In The Twelfth International Conference on Learning Representations

    2023

  36. [44]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  37. [45]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

    2023

  38. [46]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255

    2023 arXiv

  39. [47]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2...

    2022

  40. [48]

    Weidi Luo, Qiming Zhang, Tianyu Lu, Xiaogeng Liu, Yue Zhao, Zhen Xiang, and Chaowei Xiao. 2025. Doxing via the lens: Revealing privacy leakage in image geolocation for agentic multi-modal large reasoning model. arXiv preprint arXiv:2504.19373

    2025

  41. [49]

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, and 1 others. 2025. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365

    2025 arXiv

  42. [50]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393

    2025 arXiv

  43. [51]

    OpenAI. 2025. https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf Openai o3 and o4-mini system card . Technical report, OpenAI

    2025

  44. [52]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and etc

    OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and etc. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

    2024 arXiv

  45. [53]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing sys...

    2022

  46. [54]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191

    2021 arXiv

  47. [55]

    Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. 2024. Multimath: Bridging visual and mathematical reasoning for large language models. arXiv preprint arXiv:2409.00147

    2024 arXiv

  48. [56]

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. 2025. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536

    2025 arXiv

  49. [57]

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, and 1 others. 2025. Humanity's last exam. arXiv preprint arXiv:2501.14249

    2025 arXiv

  50. [58]

    Leonardo Ranaldi and Andr \`e Freitas. 2024. Self-refine instruction-tuning for aligning reasoning in language models. arXiv preprint arXiv:2405.00402

    2024 arXiv

  51. [59]

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275

    2022 arXiv

  52. [60]

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022 a . Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100

    2022 arXiv

  53. [61]

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, and 1 others. 2024 a . Enhancing visual-language modality alignment in large vision language models via self-improvement. arXiv preprint ar...

    2024 arXiv

  54. [62]

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Yongyuan Liang, Yuhang Zhou, Xiaoyu Liu, Ziyi Zang, Ming Li, Chung-Ching Lin, Kevin Lin, and 1 others. 2025 a . Vicrit: A verifiable reinforcement learning proxy task for visual perception in vlms. arXiv preprint arXiv:2506.10128

    2025 arXiv

  55. [63]

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. 2025 b . Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint arXiv:2504.07934

    2025 arXiv

  56. [64]

    Xiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu, Yuancheng Xu, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. 2024 b . Scaling inference-time search with vision value model for improved visual comprehension. arXiv preprint arXiv:2412.03704

    2024 arXiv

  57. [65]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022 b . Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  58. [66]

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, and 1 others. 2024 c . Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems, 37...

    2024

  59. [67]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  60. [68]

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, and 1 others. 2025. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503.10460

    2025 arXiv

  61. [69]

    Jeremy M Wolfe, Kyle R Cave, and Susan L Franzel. 1989. Guided search: an alternative to the feature integration model for visual search. Journal of Experimental Psychology: Human perception and performance, 15(3):419

    1989

  62. [70]

    Jeremy M Wolfe and Todd S Horowitz. 2017. Five factors that guide attention in visual search. Nature human behaviour, 1(3):0058

    2017

  63. [71]

    Junde Wu, Jiayuan Zhu, and Yuyuan Liu. 2025. Agentic reasoning: Reasoning llms with tools for the deep research. arXiv preprint arXiv:2502.04644

    2025 arXiv

  64. [72]

    Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. 2024. Minds eye of llms: Visualization-of-thought elicits spatial reasoning in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  65. [73]

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333

    2024 arXiv

  66. [74]

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451

    2024 arXiv

  67. [75]

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. 2024. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440

    2024 arXiv

  68. [76]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809--11822

    2023

  69. [77]

    Bin Yu, Hang Yuan, Yuliang Wei, Bailing Wang, Weizhen Qi, and Kai Chen. 2025. Long-short chain-of-thought mixture supervised fine-tuning eliciting efficient reasoning in large language models. arXiv preprint arXiv:2505.03469

    2025 arXiv

  70. [78]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490

    2023 arXiv

  71. [79]

    Yuan Yuan, Muyu He, Muhammad Adil Shahid, Jiani Huang, Ziyang Li, and Li Zhang. 2025. Turnaboutllm: A deductive reasoning benchmark from detective games. arXiv preprint arXiv:2505.15712

    2025

  72. [80]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024 a . Mmmu: A massive mul...

    2024

  73. [81]

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, and 1 others. 2024 b . Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813

    2024 arXiv

  74. [82]

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720--6731

    2019

  75. [83]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, and 1 others. 2024 a . Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, page...

    2024

  76. [84]

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. 2024 b . Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198

    2024 arXiv

  77. [85]

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923

    2023 arXiv

  78. [86]

    Pei Zhou, Aman Madaan, Srividya Pranavi Potharaju, Aditya Gupta, Kevin R McKee, Ari Holtzman, Jay Pujara, Xiang Ren, Swaroop Mishra, Aida Nematzadeh, and 1 others. 2023. How far are large language models from agents with theory-of-mind? arXiv preprint arXiv:2310.03051

    2023 arXiv

  79. [87]

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622

    2024 arXiv

  80. [88]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  81. [89]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.