Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Position: Reasoning After Perception Means Reasoning Without Vision

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that multimodal AI's persistent failures on fine-grained visual tasks are not a reasoning deficit but a representational one: task-critical spatial detail is discarded when images are compressed into text-aligned tokens…

desk verdict A genuinely useful TET diagnostic with clean component ablations, but the formal information-collapse bound is an assumption in disguise; the empirical case deserves peer review. read the letter →

arxiv 2507.16863 v2 pith:PKP5SBAW submitted 2025-07-21 cs.CV cs.AIcs.CL

Hongcheng Gao , Zihao Huang , Jingyi Tang , Lin Xu , Xinhao Li , Haoyang Li , Yue Liu , Minhua Lin
show 7 more authors
Xinlong Yang Taihang Hu Ge Wu Balong Bi Hongyu Chen Olive Huang Wentao Zhang
This is my paper
classification cs.CVcs.AIcs.CL
keywords multimodallargelanguagemodelsvisualperceptionbottleneckreasoningwithininformationcollapseTuringEyeTestsemanticallycontractiveprojectionfine-grainedspatialcontinuousversusdiscreterepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper challenges the common assumption that stronger language reasoning — chain-of-thought, in-context learning, or tools — can compensate for the perceptual weaknesses of vision-language models. Its central claim is that these failures are structural: when reasoning is deferred to the language decoder, computation moves from the continuous visual manifold into a discrete, text-aligned space, so fine-grained geometric and topological signals are collapsed before any thinking occurs. The authors formalize this as an information-collapse bound guaranteeing a non-zero error floor for such tasks, and they support it with the Turing Eye Test (TET), on which fifteen current models score near zero. The constructive message is that reasoning must happen within perception — actively querying pixel-level representations during encoding — rather than after them.

What carries the argument

The load-bearing object is the semantically contractive vision-to-language projection $g_\phi$, which maps the continuous visual feature space into the text-aligned embedding space; Proposition 4.1 (Information Collapse) converts this contraction into a strict lower bound on prediction error for any task whose deciding detail lives in collapsed geometry or topology. The Turing Eye Test (TET) is the companion mechanism: four task families — hidden text, 3D-perspective captchas, Ishihara-style color plates, and fused Chinese ligatures — that are easy to verify from pixels but hard to specify in words, designed to isolate the perceptual bottleneck from semantic shortcuts. The fine-tuning and reinforcement-learning ablations on a decomposable open model family complete the machinery by showing that only vision-encoder updates move the error, pinning the bottleneck to encoding rather than language-side computation.

What would settle it

Directly measure the projection gap: build pairs of images that share semantic content but differ in the geometric detail a task requires, then compare distances in the visual feature space with distances in the text-aligned embedding space; if the text-aligned distances are not systematically much smaller, semantic contraction fails and the bound does not apply. Alternatively, the claim would be refuted empirically if any model using the standard perception-then-reasoning pipeline — with only text-side interventions such as chain-of-thought, in-context exemplars, or discrete tool calls — achieved high accuracy on the Turing Eye Test tasks.

Watch

Extended reading notes

Core claim

The paper's central claim is that persistent perceptual failures in multimodal large language models stem from representational access, not insufficient reasoning capacity: task-critical spatial information is discarded before reasoning begins. Because the vision-to-text projection is semantically contractive — it maps geometrically distinct but semantically similar images into nearby text-aligned embeddings — the projection gap is strictly positive for fine-grained geometric and topological tasks, leaving an irreducible error lower bound that text-space reasoning alone cannot beat. The Turing Eye Test demonstrates this: across fifteen models, accuracy on hidden-text, 3D-captcha, color-blind, and fused-ligature tasks is near zero, and neither chain-of-thought prompting, few-shot exemplars, nor pass@32 sampling recovers the lost signal. The paper's positive claim is that perception itself must become the site of reasoning, so that pixel-level evidence remains accessible and steerable until the decision is made.

Load-bearing premise

The whole argument rests on the assumption that the vision-to-text projection is semantically contractive — that it systematically maps geometrically distinct but semantically similar images onto nearly identical text-aligned embeddings — an assumption the paper states but does not directly measure on the tested models.

Editorial extensions

If this is right

  • Text-side scaling — more chain-of-thought tokens, more samples, or more tools that return discrete outputs — cannot fix failures on fine-grained spatial tasks, because the deciding detail is gone before reasoning starts.
  • Fine-tuning the vision encoder is the only configuration that substantially improves TET accuracy, so the bottleneck is located in perception, not in the language model.
  • Reinforcement learning applied only to the vision tower matches full-parameter optimization, and language-side optimization adds almost nothing, implying the error floor is architectural rather than algorithmic.
  • Escaping the bound requires architectures that reason within perception, such as Active Visual Querying, where the reasoning state repeatedly issues queries back to the pixel-level representation.
  • Benchmarks that reward semantic priors can overestimate visual understanding; diagnostic probes like TET are needed to separate perceptual failures from reasoning or knowledge failures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the semantic-contraction premise is correct, the error floor on a given model and task should be predictable from a cheap direct measurement: the ratio of text-aligned embedding distance to visual feature distance for semantically identical but geometrically distinct image pairs.
  • The same argument should transfer to other modalities projected into text-aligned spaces before reasoning — audio, video, and tactile signals — where hard-to-verbalize deciding details would be collapsed identically.
  • The paper's diagnosis suggests a concrete architectural test: a model that keeps pixel-level representations queryable throughout generation should show a measurably lower TET error floor than any post-hoc fix applied to the same frozen encoder, a comparison the paper motivates but does not fully run.
  • If the generalization caveat holds — that fine-tuned encoders memorize task-specific patterns rather than learn transferable geometric primitives — then the field's next question is whether any training objective can induce genuinely transferable spatial primitives, which would turn the diagnosis into a curriculum for new perception losses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper challenges the common assumption that stronger language-side reasoning (CoT, in-context learning, tool use) can compensate for perceptual failures in multimodal large language models (MLLMs). It argues that in the dominant Perception-then-Reasoning pipeline, deferring reasoning to language generation displaces computation from the continuous visual manifold to a discrete, text-aligned space, so that task-critical spatial information is collapsed before reasoning begins. Section 4 formalizes this as an information-collapse bound: under a 'semantically contractive' visual-to-text projection g_phi, the reconstruction error epsilon_proj is strictly positive and text-space reasoning h_psi cannot reduce the resulting error. The paper introduces the Turing Eye Test (TET), four hard-to-verbalize visual recognition benchmarks, and reports near-zero pass@1 and low pass@32 accuracy across 15 MLLMs, negligible gains from 3-shot in-context learning, large gains from vision-encoder fine-tuning but not from language-backbone or adapter fine-tuning, and analogous results under reinforcement learning. It concludes by advocating reasoning within perception and sketches an Active Visual Querying framework.

Significance. If the empirical findings are treated as diagnostics rather than as proof of an information-theoretic lower bound, the paper makes a useful and honest contribution: the broad multi-model TET evaluation, the component-wise fine-tuning ablations, the explicit generalization caveat, and the resolution-sensitivity analyses are informative and reported with appropriate caution. The observation that language-side interventions fail while vision-encoder updates succeed is a concrete, falsifiable result that will interest the multimodal community. However, the formal lower-bound result in Prop. 4.1 is conditional on an unmeasured premise, and the benchmark exercises distorted-character recognition and color grouping rather than the geometric and topological reasoning tasks named in the thesis; the strength of the central claim therefore exceeds what the current evidence establishes.

major comments (3)
  1. [§4.2, Eq. (3)-(4), Prop. 4.1] The formal lower bound is not derived; it is a restatement of the paper's hypothesis. Eq. (4) defines 'semantically contractive' as distance contraction for semantically similar pairs, but a uniformly contracting injective map also satisfies Eq. (4); for an injective map, an appropriate reconstruction g† can recover z exactly, so Eq. (4) does not imply εproj > 0. To obtain Prop. 4.1 one must assume non-injectivity on the task-relevant subspace, which is precisely the conclusion being argued. Moreover, Eq. (3) is an upper bound; the sentence claiming a non-zero lower bound after Eq. (3) asserts the result rather than proves it. Please measure εproj directly, replace the proposition with an explicitly conditional statement, or derive non-injectivity from architectural facts such as token count or quantization.
  2. [§5.4, Table 7] Table 7 creates a direct tension with Prop. 4.1's premise about gφ. Updating only the vision encoder fθ while freezing gφ raises accuracy from 0% to 87-99% on HiddenText, 3DCaptcha, and ColorBlind. If gφ irreversibly collapsed all task-critical geometric information for these stimuli, no re-parameterization of fθ feeding into the same gφ should succeed. The results locate the bottleneck in the frozen visual encoder fθ, not in the projection gφ, and therefore do not validate the proposition as stated. The paper should either reconcile the proposition with this observation or restrict the claim to 'representations produced by the frozen pretrained encoder' rather than the projection operation itself.
  3. [§5.1 and §1] TET does not exercise the reasoning bottleneck named in the thesis. The four task families (HiddenText, 3DCaptcha, ColorBlind, ChineseLigatures) are distorted-character recognition and visual grouping tasks; the paper's motivating examples in Sec. 1 and Table 3 are segment intersection, shape congruence, connectivity, mental rotation, and trajectory prediction, which require genuine spatial reasoning. Since the central claim is that post-perception text reasoning cannot remedy failures of reasoning about geometry and topology, the empirical validation needs at least one task family that requires such reasoning. Without it, the results support only the weaker conclusion that language-side interventions fail on low-level perceptual recognition tasks. A concrete test would be to add a TET family with segment-intersection or shape-congruence queries.
minor comments (5)
  1. [§5.4, Eq. (5)] The notation Δ∥y−fθ(x)∥²_Ω⁻¹ is used without defining Ω or deriving the approximation; as written it is an informal intuition rather than a proof and should be labeled accordingly.
  2. [§4.2, Eq. (2)] The reconstruction mapping g† is called 'optimal' but no optimality criterion is specified; please state the reconstruction loss and whether g† is allowed to depend on the task.
  3. [§5.4] The text refers to 'Theorem 4.1' but Sec. 4.2 states 'Proposition 4.1'; use one name consistently.
  4. [§5.5, Figure 3] The three panels of Figure 3 are not explicitly referenced in the text; please add pointers to the RL on Full Parameters, RL on LM Backbone, and RL on ViT panels.
  5. [References] Several references contain formatting errors or incomplete author fields, e.g., 'Team, G. . Gemini 2.5' has a stray period and missing author initials; please proofread the bibliography.

Circularity Check

1 steps flagged · score 6.0 of 10

Proposition 4.1's information-collapse bound restates its own antecedent: assuming gϕ is 'semantically contractive (collapsing geometrically distinct but semantically equivalent inputs)' already assumes the ε_proj > 0 it then 'proves'; Eq. 4 alone does not imply it.

  1. self definitional [Sec. 4.2, Eq. 4 and Prop. 4.1; premise first introduced as hypothesis in Sec. 3.2; contrasted with Sec. 5.4, Table 7]
    "Proposition 4.1 (Information Collapse). Let z∗ encode geometric or topological properties, and let gϕ be semantically contractive (collapsing geometrically distinct but semantically equivalent inputs). Then: (1) the projection gap εproj = ‖fθ(x)−g†(gϕ(fθ(x)))‖Z > 0 is strictly positive, and (2) the resulting prediction error admits a non-zero lower bound irreducible by text-space reasoning hψ alone."

    The proposition's antecedent literally contains its conclusion: 'semantically contractive' is parenthesized as 'collapsing geometrically distinct but semantically equivalent inputs,' and the conclusion is that εproj > 0 with an irreducible lower bound. Non-injectivity on the task-relevant subspace is exactly what a positive εproj requires, so the bound is a restatement of the assumption. Eq. 4 alone cannot force εproj > 0: an injective contractive map is exactly invertible on its image, so the optimal reconstruction g† can make εproj = 0. The paper also labels the premise only a hypothesis in Sec. 3.2 ('We hypothesize that the projection from visual to textual space acts as a filter governed by verbalizability...') and never measures εproj.

full rationale

The paper's formal first-principles support, Proposition 4.1, is circular in the argumentative sense: its asserted lower bound is obtained by assuming the very collapse it purports to derive. The proposition asks us to 'let gϕ be semantically contractive (collapsing geometrically distinct but semantically equivalent inputs)' and then concludes that εproj > 0 and that no text-space reasoning can eliminate the error; the parenthetical already supplies the non-injectivity needed for a positive projection gap. Eq. 4, which merely says distances in E are much smaller than distances in Z, does not imply εproj > 0 because an injective but strongly contractive map admits an exact inverse on its image. Sec. 3.2 introduces the premise explicitly as a hypothesis, and the paper reports no measurement of εproj, so the 'Information Collapse Bound' is not independently established. The empirical sections contain genuinely independent content: the near-zero TET results across 15 MLLMs, the component-wise fine-tuning ablations, the RL experiments, and the resolution-sensitivity analyses are real measurements supporting a representational-access diagnosis. However, Table 7 cuts against the formal lower bound: with a frozen gϕ, re-parameterizing only fθ reaches high accuracy, indicating the frozen projection does not necessarily destroy the task-critical information. The paper's own Generalization Caveat (Sec. 5.4) honestly narrows these gains, but does not rescue the formal claim. No load-bearing self-citation chain or imported 'uniqueness theorem' is used; the central circularity is the self-definitional lower bound. Overall score 6: a central theoretical 'prediction' reduces by construction to its input assumption, while the benchmark and ablation evidence remain largely independent.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted constants or invented physical entities appear. The load-bearing assumptions are the semantic contractivity of g_phi (stated, not proven) and the Lipschitz condition on the task; both are formal assumptions rather than derived facts. The proposed Active Visual Querying framework is a design sketch, not an invented entity.

assumptions (4)
  • domain assumption The task function is Lipschitz continuous with respect to the visual representation with constant L.
    Assumed in Sec. 4.2 right before Eq. 3 to connect representation error to prediction error. L is never estimated, so the bound remains qualitative.
  • ad hoc to paper The projection g_phi is semantically contractive for geometric and topological information.
    This is the central assumption of Proposition 4.1 (Eq. 4); it is not derived or empirically verified in the paper. The theorem's conclusion follows from this premise.
  • standard math The reconstruction mapping g-dagger exists and is optimal in the sense of minimizing reconstruction error.
    Invoked in Sec. 4.2 to define the effective visual information z-tilde. Existence follows from standard projection arguments, but optimality is an idealization.
  • domain assumption The image manifold X and embedding spaces Z and E admit norms and the composition g_phi o g-dagger is non-invertible on the geometric subspace.
    Implicit in the formalization of the 'projection gap' and the claim that perfect reconstruction is impossible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position: Reasoning After Perception Means Reasoning Without Vision." pith.science (2026). https://pith.science/paper/PKP5SBAW

@misc{pith2026250716863,
  author       = {Pith},
  title        = {Pith review of: Position: Reasoning After Perception Means Reasoning Without Vision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PKP5SBAW}},
  note         = {Machine review of arXiv:2507.16863}
}
read the original abstract

A common belief in multimodal research is that the perceptual weaknesses of vision--language models can be compensated by stronger language reasoning (e.g., chain-of-thought, in-context learning, or external tools). We challenge this assumption. We argue that for a broad class of visual tasks hard to specify in language, failures stem from a structural fatality where the temporal decision of \textit{when} to reason strictly dictates the spatial constraint of \textit{where} reasoning takes place. When visual reasoning is deferred to language generation, current architectures do not merely delay computation; they displace it from the continuous visual representation to a discrete textual space. Consequently, the sequential ``Perception-then-Reasoning'' paradigm degenerates perception into a passive, one-off feature encoding process, rendering it functionally equivalent to ``Reasoning-in-Text-Space'', where task-critical spatial signals are collapsed before reasoning begins. We substantiate this claim with the Turing Eye Test (TET): tasks that must be resolved in \emph{visual space} and are hard to verbalize; results show text-only reasoning cannot remedy these perceptual failures. Our findings suggest rethinking the architectural divide: shifting from reasoning \textit{about} perception to reasoning \textit{within} perception. This facilitates actively reasoning-driven perception that operates directly on pixel-level visual representations, rather than within a collapsed textual space.

Figures

Figures reproduced from arXiv: 2507.16863 by the authors.

Figure 1
Figure 1. (Left) The mainstream Perception-then-Reasoning paradigm: visual evidence is passively encoded into semantic signals, leading to Information Collapse. (Right) The proposed Reasoning within Perception paradigm: reasoning is executed ac￾tively on the original image before semantic compression. coding, after the visual stream has been compressed into a text-facing interface (e.g., a limited set of visual tokens, pooled… view at source ↗
Figure 2
Figure 2. Evaluation cases for each TET category: HiddenText, 3DCaptcha, ColorBlind, and ChineseLigatures. The text beneath each image represents the corresponding ground truth. The third line of Chinese characters, read from left to right, symbolizes marital bliss, a serendipitous union, a perfect match, dreams fulfilled, and flawless. 0 50 100 150 200 250 300 350 Step 0.4 0.5 0.6 0.7 0.8 Accuracy SFT on Full SFT on LLM SFT … view at source ↗
Figure 3
Figure 3. Architecture Ablation on Perception Space. We conduct architecture ablation experiments during both the cold start and RL stages, where Full denotes training the entire architecture, LLM/ViT indicates training only the corresponding module, and None represents that no cold-start training was performed. with reinforcement learning applied solely to the Vision Tower achieve performance on par with full-parameter op￾ti… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Grad-CAM of Qwen2.5-VL Series Models on HiddenText . 6. Toward Reasoning Within Perception Our diagnosis points to a clear prescription: visual reason￾ing must occur in visual space, not text space. Rather than advocating for a single architecture, we outline the desig…
Figure 5
Figure 5. Figure 5: Resolution Sensitivity. (a) Downsampling paradoxically improves accuracy, suggesting reliance on coarse semantics over fine structure. (b) Blurring degrades performance despite preserved global layout, confirming dependence on local texture cues. Mechanism Formulation.…
Figure 6
Figure 6. Figure 6: reports the pass@K curves across tasks, together with their standard deviations. As K increases, existing MLLMs show consistently flat performance trajectories: average accuracy exhibits little variation across tasks, and even the best-performing cases achieve gains of…
Figure 7
Figure 7. Figure 7: Model response on question of HiddenText. The goal is to identify the hidden word in an image. Gemini-2.5-Pro-0506 answers the hidden word as “castle”. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Grad-CAM of Qwen2.5-VL Series Models on 3DCaptcha. noise, illustrating that perceptual grouping—conceptually simple but hard to verbalize—requires active visual reasoning on pixel-level representations. As shown in [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Grad-CAM of Qwen2.5-VL Series Models on ColorBlind subset. current interfaces do not for fine-grained visual tasks. (Cartuyvels et al., 2021; Nagrani et al., 2021) “These Tasks Are Artificial.” One might dismiss TET tasks as contrived edge cases. We counter that these …
Figure 10
Figure 10. Figure 10: Grad-CAM of Qwen2.5-VL Series Models on ChineseLigature subset. visual tokens, and U is the residual update. This maintains pixel-level access but lacks explicit query generation (Q) and perception refinement (P)—the visual representation remains static. Recurrent Vis…
Figure 11
Figure 11. Figure 11: Grad-CAM of Qwen2.5-VL-7B before and after visual fine-tuning on HiddenText. Qwen2.5-VL-7B ViT LLM ViT LLM Answer: 74 GT: M3 Qwen2.5-VL-7B-FT Answer: M3 GT: M3 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Grad-CAM of Qwen2.5-VL-7B before and after visual fine-tuning on ColorBlind. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Grad-CAM of Qwen2.5-VL-7B before and after visual fine-tuning on 3DCaptcha. 0 25 50 75 100 125 150 175 Training Steps 0.0 0.5 1.0 1.5 2.0 2.5 Loss Full parameters Vision encoder only Vision encoder with adapter Language backbone only Vision-language adapter only (a) C…
Figure 14
Figure 14. Figure 14: Training Curves. Training loss curve for different settings on finetuning parameters for both our tasks and traditional tasks. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Example responses from the Gemini model on the four tasks of TET. For each task, the model fails to provide correct analysis and conclusion based on its flawed initial perception. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Frontier MLLMs remain far from mastering atomic visual perception: none reach 60% on a failure-derived, perception-only benchmark of ten capabilities.

  2. Moravec's Paradox: Towards an Auditory Turing Test

    cs.AI 2025-07 reject novelty 4.0 of 10

    The paper proposes an auditory Turing test of 917 challenges and reports that the best tested AI model, GPT-4o audio, scores 6.9% versus 52% for a nine-person human sample.

Reference graph

Works this paper leans on

62 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Introducing Claude 4

    Anthropic. Introducing Claude 4. Anthropic News, May 2025. URL https://www.anthropic.com/news/claude-4. Accessed: [Insert Access Date]

    work page 2025

  3. [3]

    DASH: Detection and Assessment of Systematic Hallucinations of VLMs

    Maximilian Augustin, Yannic Neuhaus, and Matthias Hein. Dash: Detection and assessment of systematic hallucinations of vlms. arXiv preprint arXiv:2503.23573, 2025

    work page Pith review arXiv 2025

  4. [4]

    Qwen technical report, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  5. [5]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  7. [7]

    Spider2-v: How far are multimodal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37: 0 107703--107744, 2024

    Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37: 0 107703--107744, 2024

    work page 2024

  8. [8]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

Show all 62 references
  1. [9]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025

    2025 arXiv

  2. [10]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67 0 (12): 0 22...

    2024

  3. [11]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  4. [12]

    V-star: Benchmarking video-llms on video spatio-temporal reasoning

    Zixu Cheng, Jian Hu, Ziquan Liu, Chenyang Si, Wei Li, and Shaogang Gong. V-star: Benchmarking video-llms on video spatio-temporal reasoning. arXiv preprint arXiv:2503.11495, 2025

    2025 arXiv

  5. [13]

    Emerging properties in unified multimodal pretraining

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025

    2025 arXiv

  6. [14]

    Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning, 2025

    Ling Fu, Zhebin Kuang, Jiajun Song, Mingxin Huang, Biao Yang, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Zhang Li, Guozhi Tang, Bin Shan, Chunhui Lin, Qi Liu, Binghong Wu, Hao Feng, Hao Liu, Can Huang, Jingqun Tang, Wei Chen, Lianwen Jin, Yuliang Liu, and Xiang Bai. ...

    2025 arXiv

  7. [15]

    Exploring hallucination of large multimodal models in video understanding: Benchmark, analysis and mitigation, 2025

    Hongcheng Gao, Jiashu Qu, Jingyi Tang, Baolong Bi, Yue Liu, Hongyu Chen, Li Liang, Li Su, and Qingming Huang. Exploring hallucination of large multimodal models in video understanding: Benchmark, analysis and mitigation, 2025. URL https://arxiv.org/abs/2503.19622

    2025 arXiv

  8. [16]

    Gonzalez and Richard E

    Rafael C. Gonzalez and Richard E. Woods. Digital Image Processing. Pearson, New York, 4th edition, 2018

    2018

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  10. [18]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025

    2025 arXiv

  11. [19]

    Tests for colour-blindness, 1951

    Shinobu Ishihara. Tests for colour-blindness, 1951

    1951

  12. [20]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  13. [21]

    Elite: Enhanced language-image toxicity evaluation for safety

    Wonjun Lee, Doehyeon Lee, Eugene Choi, Sangyoon Yu, Ashkan Yousefpour, Haon Park, Bumsub Ham, and Suhyun Kim. Elite: Enhanced language-image toxicity evaluation for safety. arXiv preprint arXiv:2502.04757, 2025 a

    2025 arXiv

  14. [22]

    Holisafe: Holistic safety benchmarking and modeling with safety meta token for vision-language model

    Youngwan Lee, Kangsan Kim, Kwanyong Park, Ilcahe Jung, Soojin Jang, Seanie Lee, Yong-Ju Lee, and Sung Ju Hwang. Holisafe: Holistic safety benchmarking and modeling with safety meta token for vision-language model. arXiv preprint arXiv:2506.04704, 2025 b

    2025

  15. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023

    2023

  16. [24]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023

    2023

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 26296--26306, 2024 a

    2024

  18. [26]

    Llavanext: Improved reasoning, ocr, and world knowledge, 2024 b

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024 b

    2024

  19. [27]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 0 2507...

    2022

  20. [28]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL https://arxiv.org/abs/2310.02255

    2024 arXiv

  21. [29]

    Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. In Proceedings of the Computer Vision and Pattern Recognition...

    2025

  22. [30]

    Mia-bench: Towards better instruction following evaluation of multimodal llms, 2025

    Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. Mia-bench: Towards better instruction following evaluation of multimodal llms, 2025. URL https://arxiv.org/abs/2407.01509

    2025 arXiv

  23. [31]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  24. [32]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  25. [33]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp.\ 618--626, 2017

    2017

  26. [34]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  27. [35]

    Introduction to techniques used in seed1.6, June 2025 a

    ByteDance Seed Team. Introduction to techniques used in seed1.6, June 2025 a . URL https://seed.bytedance.com/en/seed1_6. Technical Report

    2025

  28. [36]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024 a

    2024 arXiv

  29. [37]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025 b

    Gemini 2.5 Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025 b . URL https://arxiv.org/abs/2507.06261

    2025 arXiv

  30. [38]

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yan...

    2025 arXiv

  31. [39]

    Minicpm-o 2.6: A gpt-4o level mllm for vision, speech, and multimodal live streaming on your phone, 2025 c

    OpenBMB MiniCPM-o Team. Minicpm-o 2.6: A gpt-4o level mllm for vision, speech, and multimodal live streaming on your phone, 2025 c

    2025

  32. [40]

    Qvq: To see the world with wisdom, December 2024 b

    Qwen Team. Qvq: To see the world with wisdom, December 2024 b . URL https://qwenlm.github.io/blog/qvq-72b-preview/

    2024

  33. [41]

    Measuring multimodal mathematical reasoning with math-vision dataset, 2024 a

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset, 2024 a . URL https://arxiv.org/abs/2402.14804

    2024 arXiv

  34. [42]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024 b

    2024 arXiv

  35. [43]

    Mmlongbench: Benchmarking long-context vision-language models effectively and thoroughly, 2025

    Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng, Ginny Wong, Simon See, Pasquale Minervini, Yangqiu Song, and Mark Steedman. Mmlongbench: Benchmarking long-context vision-language models effectively and thoroughly, 2025. URL https://arxiv.or...

    2025

  36. [44]

    Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024. URL https://arxiv.org/abs/2407.15754

    2024 arXiv

  37. [45]

    Logicvista: Multimodal llm logical reasoning benchmark in visual contexts, 2024

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts, 2024. URL https://arxiv.org/abs/2407.04973

    2024 arXiv

  38. [46]

    Show-o2: Improved native unified multimodal models

    Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506.15564, 2025

    2025 arXiv

  39. [47]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing S...

    2024

  40. [48]

    Qwen2.5-omni technical report

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  41. [49]

    Cc-ocr: A comprehensive and challenging ocr benchmark for evaluating large multimodal models in literacy

    Zhibo Yang, Jun Tang, Zhaohai Li, Pengfei Wang, Jianqiang Wan, Humen Zhong, Xuejing Liu, Mingkun Yang, Peng Wang, Shuai Bai, et al. Cc-ocr: A comprehensive and challenging ocr benchmark for evaluating large multimodal models in literacy. arXiv preprint arXiv:2412.02210, 2024

    2024 arXiv

  42. [50]

    Mm-vet: Evaluating large multimodal models for integrated capabilities, 2024

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2024. URL https://arxiv.org/abs/2308.02490

    2024 arXiv

  43. [51]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  44. [52]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024 b

    2024 arXiv

  45. [53]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?, 2024 a

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?, 2024 a . URL https://arxiv.org/abs/2403.14624

    2024 arXiv

  46. [54]

    From redundancy to relevance: Enhancing explainability in multimodal large language models

    Xiaofeng Zhang, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. From redundancy to relevance: Enhancing explainability in multimodal large language models. arXiv e-prints, pp.\ arXiv--2406, 2024 b

    2024

  47. [55]

    Multitrust: A comprehensive benchmark towards trustworthy multimodal large language models

    Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, et al. Multitrust: A comprehensive benchmark towards trustworthy multimodal large language models. Advances in Neural Information Processing Systems, 37: ...

    2024

  48. [56]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  49. [57]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  50. [58]

    Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836, 2024

    2024 arXiv

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  52. [60]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  53. [61]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  54. [62]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.