Pith. sign in

REVIEW 5 major objections 5 minor 11 cited by

VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A vision-language model can be trained to answer most questions from a quarter-resolution image and request full resolution only for detail-hungry samples.

desk verdict Solid, useful dynamic-resolution RL method whose per-benchmark results are more mixed than the abstract claims; deserves review after honest reporting. read the letter →

arxiv 2507.13348 v1 pith:QSPMIAV2 submitted 2025-07-17 cs.CV cs.AIcs.CLcs.LG

Senqiao Yang , Junyi Li , Xin Lai , Bei Yu , Hengshuang Zhao , Jiaya Jia This is my paper
classification cs.CVcs.AIcs.CLcs.LG
keywords vision-languagemodelvisualtokencompressionadaptiveresolutionreinforcementlearningGRPOLLM-as-JudgeefficientinferenceOCR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VisionThink proposes that a vision-language model can decide for itself, per question, whether a downsampled image is enough to answer or whether it must request the original resolution. Starting from a 1/4-resolution image, the model answers directly in most cases and emits a special tool call only when fine detail is needed. The authors train this behavior with multi-turn GRPO, using an LLM-as-Judge to give correct/incorrect rewards because general VQA answers cannot be verified by fixed rules. Across nine benchmarks the trained model retains about 51.3% of visual tokens while averaging 103.7% of the base model's accuracy; on OCR-heavy benchmarks it upscales far more often (roughly 79% on ChartQA, 62% on OCRBench) than on general ones (31% on MME, 7% on DocVQA). If correct, this means token-compression efficiency does not have to come from a fixed pruning ratio applied to every image.

What carries the argument

The load-bearing mechanism is a learned two-round image-access policy. The model sees a downsampled image and either answers immediately or emits a special upscale token, framed as a tool call, that doubles the image resolution; a multi-turn extension of GRPO optimizes the whole trajectory. A binary LLM-as-Judge reward decides correctness, and a thresholded penalty term $P_{\text{control}}=0.1[\mathbb{1}_{\text{direct}}I(r<\theta)+\mathbb{1}_{\text{high}}I(r\geq\theta)]$ with $r=C_{\text{direct}}/(C_{\text{direct}}+C_{\text{high}})$ prevents the model from collapsing into always upscaling or always guessing at low resolution.

What would settle it

Take a random sample of 1,000 QA pairs from the reward-training distribution, including OCR-heavy cases, and compare the LLM judge's binary scores with human scores; if disagreements are substantially more frequent than the paper's reported zero-misclassification user study, or if they cluster in the fine-grained OCR cases that drive VisionThink's upscale decisions, the RL signal and the ChartQA† numbers would rest on a biased reward.

Watch

Extended reading notes

Core claim

The central claim is that a VLM can learn to route each sample to either a cheap direct answer from a 1/4-resolution image or an expensive high-resolution pass, and that this routing can be learned entirely by reinforcement learning with no cold-start stage. The paper's evidence is that VisionThink, built on Qwen2.5-VL-7B-Instruct, reaches 103.7% of the base model's average score over nine general-VQA benchmarks while keeping 51.3% of visual tokens, and that its upscale-request rate tracks the OCR content of the benchmark (high on ChartQA and OCRBench, low on MME and DocVQA). The same LLM-as-Judge training also improves quality when run at full resolution, so the reward design is claimed to work for general VQA rather than only math.

Load-bearing premise

The load-bearing premise is that the external language model that says 'right' or 'wrong' about an answer agrees with human judgment on general visual questions; the only direct check reported is a 1,000-case user study.

Editorial extensions

If this is right

  • Deploying a VLM with VisionThink's policy spends about half the visual tokens on average, with the exact saving determined per sample rather than by a fixed ratio.
  • OCR-heavy workloads still pay for full resolution, so accuracy on charts, documents, and text-heavy images is preserved instead of being traded away for speed.
  • The LLM-as-Judge reward makes general VQA, not just visual math, trainable by RL, so open-ended answer correctness can be optimized without hand-written verifiers.
  • Because the upscale decision is a token output, the mechanism stays compatible with other token-compression methods; integrating them should lower the direct-answer cost further.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The two-round policy is an instance of a more general idea: any VLM can be trained to treat its own uncertainty about image detail as a gating signal; a testable extension would replace resizing with cropping or patch selection and see whether the same RL training learns which regions to request.
  • If the LLM judge systematically favors answers that only a high-resolution image can support, the same bias would inflate both the RL reward and the reported ChartQA† score; measuring judge accuracy separately on OCR-heavy and general questions would separate the model's routing skill from the reward's preference.
  • The observed call-ratio pattern (79% ChartQA, 7% DocVQA) suggests a cheap classifier could predict upscale need from image statistics and question tokens, letting the efficiency transfer to smaller models that were not RL-trained.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. VisionThink proposes a sample-adaptive visual token compression paradigm for vision-language models. Instead of applying a fixed pruning ratio, the model first processes a low-resolution image and may emit a tool-call token to request the original resolution; training uses multi-turn GRPO with an LLM-as-Judge accuracy reward, a format reward, and a penalty mechanism (Eq. 5) that penalizes direct answers or resizing depending on the model's own per-resolution correct-answer counts. On nine benchmarks, the paper reports an average retained visual-token ratio of about 51.3% and an average relative performance of 103.7% of Qwen2.5-VL-7B-Instruct, with per-benchmark resize ratios ranging from about 7% (DocVQA) to 79% (ChartQA). The authors also show that the LLM-as-Judge reward improves general VQA performance when trained on 130K samples (VisionThink♠/‡).

Significance. If the reported results are robust, the work makes a useful contribution to efficient VLM inference by shifting from fixed-ratio token pruning to per-sample decisions about input resolution. The multi-turn GRPO formulation with an LLM-as-Judge reward is a transferable recipe for applying RL to open-ended VQA, and the ablations on penalty threshold (Fig. 8), prompt choice (Fig. 7), SFT vs RL (Table 8), and cold-start (Table 9) are helpful. The paper also provides code and model links. The significance is conditional on resolving two load-bearing issues: the reliability and consistency of the judge-based ChartQA evaluation, and the precise accounting of the reported 51.3% token retention. If those are addressed, the adaptive-resolution idea is a credible alternative to existing efficient-VLM methods.

major comments (5)
  1. [Appendix B.4, Table 2] The manuscript replaces the official lmms-eval ChartQA metric with GPT-4o-Judge plus human verification (ChartQA†), and the same judge provides the accuracy reward in Eq. (4). Because the judge labels both the RL training signal and the ChartQA† column of Table 2, a systematic judge bias would inflate the reported 103.7% average relative performance and the 'smartness' claim simultaneously. The only evidence of judge reliability is the 1,000-case user study in Appendix B.1.1, which reports no misclassifications but provides no sampling protocol, confidence intervals, or independent audit. Please report judge-human agreement (e.g., Cohen's kappa) on a stratified sample, including open-ended vs numeric and direct vs resize answers, and state explicitly whether all rows of Table 2, including Vanilla and the efficient-VLM baselines, were scored under the same ChartQA† protocol; if not, the comparison is not well-defined.
  2. [Section 4.3, Table 2] The headline 'Retain Approximately 51.3% Visual Tokens' is not reproducible because the paper does not define how this number is computed. It could be an unweighted average of per-benchmark retained ratios, a sample-weighted average, or a global token count ratio; for samples that request resizing, the ratio must combine the initial low-resolution tokens and the additional high-resolution tokens. Please provide the exact formula and the per-benchmark token counts (or FLOPs), and clarify the relation to the per-benchmark resize ratios in Fig. 5.
  3. [Section 4.2, Tables 1 and 2] VisionThink underperforms the vanilla model on five of the nine benchmarks in Table 2 (OCRBench, DocVQA, RealWorldQA, POPE, MathVista), with absolute drops of roughly 0.5-1.5 points, while the overall 103.7% average is driven by large gains on MME and MMVet. The abstract and Sec. 4.2 claim 'superiority' or 'comparable or even superior' without statistical support. Please report standard errors or confidence intervals for the point estimates, or at least temper the claims to a per-benchmark accuracy-efficiency tradeoff analysis.
  4. [Section 3.4, Eq. (5)] Equation (5) defines the penalty using r = C_direct/(C_direct+C_high), but the text never specifies how C_direct and C_high are obtained during training (e.g., over the 16 GRPO rollouts, over a sliding window of training steps, or over the whole dataset) or how the case C_direct=C_high=0 is handled. Since these counts are produced by the judge and by the policy being trained, the penalty is non-stationary and can steer the model toward the judge's notion of when to upscale; the data-selection rule in Sec. 3.5 has the same structure. Please make the computation precise and include an analysis of the sensitivity of the final resize ratios to the rollout count and to the threshold θ beyond the ablation in Fig. 8.
  5. [Section 4.3, Table 2] The comparison to training-free efficient-VLM baselines (FastV, SparseVLM, VisionZip) is confounded: VisionThink is trained with RL on 20K (and in variants, 130K) samples, while the baselines are used off-the-shelf. The reported average relative improvement may therefore reflect the RL procedure rather than the adaptive-resolution decision. Please add a control that isolates the contribution of the RL training under matched token budgets, for example by evaluating the full-resolution RL model (VisionThink‡) at 100% token retention and at the same average retention, or by fine-tuning a token-pruning baseline on the same data.
minor comments (5)
  1. [Section 1] The sentence 'achieving up to 100% for DocVQA' is ambiguous; Fig. 4 shows a large speedup on DocVQA, so the text should say 'up to 100% speedup' or a similar concrete claim.
  2. [Table 2 caption] The caption says Down-Sample uses an image 'resolution reduced by half,' while Sec. 1 says 'reducing the image resolution by a factor of four' and Fig. 1 says '1/4 Resolution Image.' Please define whether 1/4 refers to linear dimensions (token count roughly 1/16) or to pixel area (token count roughly 1/4), and use consistent terminology throughout.
  3. [Appendix C.2, Table 9] The MME entries in Table 9 are given as '693/1707' and '444/1354,' which appear to be perception/cognition subtotals, but Tables 1 and 2 report a single MME total (e.g., 2400). Please clarify which MME score is used for the comparisons.
  4. [References] The reference list contains several duplicated entries (e.g., [16]/[17], [19]/[20], [78]/[79], [83]/[84], [89]/[90]); please consolidate them.
  5. [Various] There are minor typos: 'computataion' in Sec. 3.1, 'Mutli-Turn' in Sec. 3.3, and 'Instrcut' in the Table 2 caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the self-referential reward and data-selection design is transparent and the central claims are independently validated by official benchmarks and efficiency measurements.

full rationale

VisionThink's derivation chain is not circular. The central claims are that an LLM-as-Judge reward with a carefully balanced penalty can train a VLM to request higher resolution selectively, and that this preserves accuracy while saving visual tokens. The penalty in Eq. (5) uses the policy's own correct-answer counts Cdirect and Chigh to decide which action to penalize; this is self-referential in flavor, but it is an online estimate of which action is empirically better, not a definition of the target behavior. The final policy is evaluated on held-out benchmarks with official metrics (OCRBench, DocVQA, MME, MMMU, MathVista, etc.), and the efficiency claim is measured by inference time and visual-token retention, both independent of the judge and of the self-generated training labels. Data selection in Sec. 3.5 uses base-model rollouts to label which samples need high resolution; this is self-labeling, but the GRPO-trained model must still generalize and improve accuracy, and the external benchmarks provide an independent check. The main validity risk is the LLM-as-Judge: the only direct reliability evidence is an internal 1,000-case user study with no misclassifications (App. B.1.1), lacking confidence intervals and independent audit, and ChartQA† uses a GPT-4o-Judge protocol similar to the training reward. That is an evaluation-validity concern, not a circular reduction, and it is mitigated by the many non-judge official benchmarks. No load-bearing self-citation or imported uniqueness theorem is present. Therefore the paper does not reduce to its own inputs by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central training pipeline rests on the base model's ability to call tools and on the judge's correctness; both are plausible but not fully validated. The method also depends on several hand-tuned hyperparameters (theta, penalty, data-selection thresholds) that shape the final policy.

free parameters (6)
  • Penalty threshold theta = 0.2
    Controls when direct answers are penalized vs high-res requests in Eq. 5; tuned by ablation (Appendix C.4), and model behavior is sensitive to it.
  • Penalty coefficient = 0.1
    Fixed penalty magnitude in Eq. 5; no ablation reported for this value.
  • Format reward weight = 0.5
    Reward for correct formatting of think, answer, and tool-call tags (Sec. 3.4); chosen by hand.
  • Data selection advantage threshold = 6 of 8 correct
    Samples are labeled high-res-required if high-res correct answers exceed low-res by 6 or more in 8 rollouts (Sec. 3.5).
  • Rollout count for data labeling = 8
    Number of base-model rollouts used to classify each training sample (Sec. 3.5).
  • Resolution reduction factor = 1/4
    The low-resolution input is 1/4 of the original (Sec. 1); chosen based on the observation that most benchmarks tolerate this.
assumptions (5)
  • domain assumption Qwen2.5-VL-7B-Instruct is a strong general-purpose VLM that can follow tool-call prompts and perform OCR at high resolution.
    The whole method builds on this base model's capabilities (Sec. 4.1).
  • domain assumption The LLM-as-Judge assigns correct (1) / incorrect (0) labels that agree with human judgment for general VQA.
    The RL training reward depends on these labels; the paper only reports a 1,000-case user study with no misclassifications (Appendix B.1.1).
  • domain assumption Multi-turn GRPO with token masking provides valid gradients for tool-call tokens (Eq. 3).
    The authors assert the extension works for VLM multi-turn interaction, but provide no formal guarantee or comparison to single-turn RL.
  • domain assumption The official Qwen agent prompt enables zero-shot tool calling before any RL training.
    They choose this prompt after ablating three options (Appendix C.3); it is assumed to be inherited from Qwen's training.
  • domain assumption Training directly from the instruct model without a cold-start phase is stable enough for GRPO.
    They argue cold-start hurts generality (Appendix C.2), and use no cold-start, but this makes training more fragile.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning." pith.science (2026). https://pith.science/paper/QSPMIAV2

@misc{pith2026250713348,
  author       = {Pith},
  title        = {Pith review of: VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QSPMIAV2}},
  note         = {Machine review of arXiv:2507.13348}
}
read the original abstract

Recent advancements in vision-language models (VLMs) have improved performance by increasing the number of visual tokens, which are often significantly longer than text tokens. However, we observe that most real-world scenarios do not require such an extensive number of visual tokens. While the performance drops significantly in a small subset of OCR-related tasks, models still perform accurately in most other general VQA tasks with only 1/4 resolution. Therefore, we propose to dynamically process distinct samples with different resolutions, and present a new paradigm for visual token compression, namely, VisionThink. It starts with a downsampled image and smartly decides whether it is sufficient for problem solving. Otherwise, the model could output a special token to request the higher-resolution image. Compared to existing Efficient VLM methods that compress tokens using fixed pruning ratios or thresholds, VisionThink autonomously decides whether to compress tokens case by case. As a result, it demonstrates strong fine-grained visual understanding capability on OCR-related tasks, and meanwhile saves substantial visual tokens on simpler tasks. We adopt reinforcement learning and propose the LLM-as-Judge strategy to successfully apply RL to general VQA tasks. Moreover, we carefully design a reward function and penalty mechanism to achieve a stable and reasonable image resize call ratio. Extensive experiments demonstrate the superiority, efficiency, and effectiveness of our method. Our code is available at https://github.com/dvlab-research/VisionThink.

Figures

Figures reproduced from arXiv: 2507.13348 by the authors.

Figure 1
Figure 1. Our key observations and VisionThink performance and efficiency. Left: We find that in most general scenarios, even reducing visual tokens by a factor of four results in only minimal performance drop. However, token compression leads to a significant performance drop on strong OCR-related benchmarks. Right: Our VisionThink significantly outperforms previous work in both performance and efficiency. as ChartQA and OCR… view at source ↗
Figure 2
Figure 2. Framework of VisionThink. (a) The left image illustrates VisionThink processing an image with resolution reduced by a factor of four, where the VLM directly provides an answer. (b) The right image shows a case where the model detects insufficient information and requests a high-resolution image to answer the question. high-resolution image is required. So that the model could balance the efficiency and performance. … view at source ↗
Figure 3
Figure 3. (a) Impact of the Penalty Ratio. Applying a penalty to all resize image requests or removing [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Inference Time Cost and Benchmark Performance Comparison for Reasoning Model. Qwen-RL and Qwen-RL (1/4) represent leveraging the LLM-as-Judge on the Qwen2.5-VL-Instruct Model and inference on full resolution image and 1/4 resolution image, respectively. these methods r…
Figure 5
Figure 5. Figure 5: VisionThink smartly determine the high-resolution image ratio. Apply Resize indicates that the model autonomously requests to view the original high-resolution image, while Direct Answer indicates that the model is able to answer the question using only the 1/4-sized i…
Figure 6
Figure 6. Figure 6: An example illustrating the original evaluation method used in ChartQA. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Impact of Prompt Choice. Prompts lead to substantial variation in image resize call ratios, with the Qwen official agent prompt demonstrating the most effective performance. Therefore, it is essential for the base model, when conditioned on our prompt, to show some pre…
Figure 8
Figure 8. Figure 8: Ablation Study on Penalty Ratio Threshold. As the threshold increases, the model progressively favors requesting image resizing instead of providing direct answers. As shown in [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models

    cs.CV 2026-08 conditional novelty 7.0 of 10

    Pretrained vision encoders show spectral response rigidity, and HAFI-VLM injects text-conditioned low/mid/high frequency evidence to improve VLM perception on VQA, text-rich understanding, and hallucination robustness.

  2. Pest-Thinker: Learning to Think and Reason like Entomologists via Reinforcement Learning

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    Pest-Thinker is a reinforcement learning framework that improves MLLMs' expert-level reasoning on pest morphology via synthesized CoT trajectories, GRPO optimization, and an LLM-judged feature reward on new benchmarks...

  3. Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    RL post-training on hallucination-forced multimodal data improves reasoning performance and can outperform standard training.

  4. SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Visual-token attention filtering improves structured pruning of vision-language models, keeping 94% of average benchmark accuracy after removing 20% of parameters.

  5. Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    ViToS uses dual-stream RL with cross-feedback optimization to prune medical image tokens to 77% length while reporting 108.27% and 104.16% relative performance on two 7B VLMs across seven benchmarks.

  6. Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    MAPO improves multimodal chain-of-thought reasoning by requiring explicit textual descriptions of visual tool results and using a novel advantage estimator that combines semantic alignment with task rewards.

  7. Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Saliency-R1 uses a novel saliency map technique and GRPO with human bounding-box overlap as reward to improve VLM reasoning faithfulness and interpretability.

  8. CPPO: Contrastive Perception Policy Optimization for VLM Agents

    cs.CV 2026-01 conditional novelty 6.0 of 10

    CPPO improves VLM agents by adding a contrastive perception loss to GRPO, applied only to entropy-selected image-dependent tokens, beating prior RL finetuning methods on visual reasoning benchmarks.

  9. RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioning

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    RSICCLLM introduces a post-training framework with RSICI dataset, difference-aware supervised fine-tuning, and dual-negative preference optimization that claims to outperform much larger models on remote sensing image...

  10. From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    The survey formalizes MLLM perception as a unified vision-language capability and traces its evolution via a new five-stage taxonomy while outlining future challenges.

  11. Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search

    cs.CV 2025-09 unverdicted novelty 5.0 of 10

    Mini-o3 scales visual search reasoning to tens of interaction turns via a new probe dataset, iterative trajectory collection, and over-turn masking in RL, claiming SOTA performance while training only up to six turns.

Reference graph

Works this paper leans on

90 extracted references · 14 canonical work pages · cited by 11 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024

    2024

  3. [4]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [6]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  5. [7]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682–17690, 2024

    2024

  6. [8]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024

    2024

  7. [9]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv:2311.12793, 2023

    arXiv 2023

  8. [10]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2023

    2023

Show all 90 references
  1. [11]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  2. [12]

    Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024

    2024

  3. [13]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2...

    2023

  4. [14]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint arXiv:2411.14432, 2024

    2024 arXiv

  5. [15]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306.13394, 2023. 11

    2023 arXiv

  6. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  7. [18]

    Zipvl: Efficient large vision-language models with dynamic token sparsification and kv cache compression

    Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipvl: Efficient large vision-language models with dynamic token sparsification and kv cache compression. arXiv preprint arXiv:2410.08584, 2024

    arXiv

  8. [20]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  9. [21]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  10. [22]

    Expedited training of visual conditioned language generation via redundancy reduction

    Yiren Jian, Tingkai Liu, Yunzhe Tao, Chunhui Zhang, Soroush V osoughi, and Hongxia Yang. Expedited training of visual conditioned language generation via redundancy reduction. arXiv preprint arXiv:2310.03291, 2023

    2023 arXiv

  11. [23]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  12. [24]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla...

    2024 arXiv

  13. [25]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  14. [26]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692, 2023

    2023 arXiv

  15. [27]

    Step-dpo: Step-wise preference optimization for long-chain reasoning of llms, 2024

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms, 2024

    2024

  16. [28]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  17. [29]

    Logits-based finetuning

    Jingyao Li, Senqiao Yang, Sitong Wu, Han Shi, Chuanyang Zheng, Hong Xu, and Jiaya Jia. Logits-based finetuning. arXiv preprint arXiv:2505.24461, 2025

    2025 arXiv

  18. [30]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 2023

    2023

  19. [31]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403.18814, 2024. 12

    2024 arXiv

  20. [32]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv:2305.10355, 2023

    2023 arXiv

  21. [33]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv:2310.03744, 2023

    2023 arXiv

  22. [34]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  23. [35]

    Visual instruction tuning.Advances in neural information processing systems, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 2024

    2024

  24. [36]

    Robomamba: Multimodal state space model for efficient robot reasoning and manipulation

    Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation. arXiv preprint arXiv:2406.04339, 2024

    2024 arXiv

  25. [37]

    Ocrbench: On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: On the hidden mystery of ocr in large multimodal models. arXiv:2305.07895, 2023

    2023 arXiv

  26. [38]

    Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement

    Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520, 2025

    2025 arXiv

  27. [39]

    Visual-rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  28. [40]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024

    2024

  29. [41]

    Textcot: Zoom in for enhanced multimodal text-rich image understanding

    Bozhi Luan, Hao Feng, Hong Chen, Yonghui Wang, Wengang Zhou, and Houqiang Li. Textcot: Zoom in for enhanced multimodal text-rich image understanding. arXiv preprint arXiv:2404.09797, 2024

    2024 arXiv

  30. [42]

    Reft: Reasoning with reinforced fine-tuning, 2024

    Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning, 2024

    2024

  31. [43]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  32. [44]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021

    2021

  33. [45]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  34. [46]

    Compositional chain-of- thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14431, 2024

    2024

  35. [47]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennet...

    2024

  36. [48]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024

    2024

  37. [49]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv:2304.03277, 2023

    2023 arXiv

  38. [50]

    Mo- bile edge intelligence for large language models: A contemporary survey

    Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, and Kaibin Huang. Mo- bile edge intelligence for large language models: A contemporary survey. arXiv preprint arXiv:2407.18921, 2024

    2024 arXiv

  39. [51]

    Does your vision-language model get lost in the long video sampling dilemma? arXiv preprint arXiv:2503.12496, 2025

    Tianyuan Qu, Longxiang Tang, Bohao Peng, Senqiao Yang, Bei Yu, and Jiaya Jia. Does your vision-language model get lost in the long video sampling dilemma? arXiv preprint arXiv:2503.12496, 2025

    2025 arXiv

  40. [52]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  41. [53]

    Vlm-r1: A stable and generalizable r1-style large vision-language model

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  42. [54]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024. 14

    2024 arXiv

  43. [55]

    Upop: Unified and progressive pruning for compressing vision-language transformers

    Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Upop: Unified and progressive pruning for compressing vision-language transformers. In International Conference on Machine Learning, pages 31292–31311. PMLR, 2023

    2023

  44. [56]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    2024

  45. [57]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  46. [58]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, ...

    2025

  47. [59]

    Llamav-o1: Rethinking step-by-step visual reasoning in llms

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-o1: Rethinking step-by-step visual reasoning in llms. arXiv preprint arXiv:2501.06186, 2025

    2025 arXiv

  48. [60]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024

    2024 arXiv

  49. [61]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023

    2023 arXiv

  50. [62]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv:2311.03079, 2023

    2023 arXiv

  51. [63]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  52. [64]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in N...

    2022

  53. [65]

    Efficient vision- language models by summarizing visual tokens into compact registers

    Yuxin Wen, Qingqing Cao, Qichen Fu, Sachin Mehta, and Mahyar Najibi. Efficient vision- language models by summarizing visual tokens into compact registers. arXiv preprint arXiv:2410.14072, 2024

    2024 arXiv

  54. [66]

    Mmsearch-r1: Incentivizing lmms to search

    Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, and Ziwei Liu. Mmsearch-r1: Incentivizing lmms to search. arXiv preprint arXiv:2506.20670, 2025

    2025 arXiv

  55. [67]

    xAI. Grok. https://x.ai/, 2023. Large language model

    2023

  56. [68]

    Grok-1.5 vision preview

    X.AI. Grok-1.5 vision preview. https://x.ai/blog/grok-1.5v, 2024. 15

    2024

  57. [69]

    Lillicrap, Kenji Kawaguchi, and Michael Shieh

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P. Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning, 2024

    2024

  58. [70]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247, 2024

    arXiv

  59. [71]

    Llava-o1: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step. arXiv e-prints, pages arXiv–2411, 2024

    2024

  60. [72]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  61. [73]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467, 2024

    2024

  62. [74]

    Lidar-llm: Exploring the potential of large language models for 3d lidar understanding

    Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312.14074, 2023

    2023 arXiv

  63. [75]

    An improved baseline for reasoning segmentation with large language model

    Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240, 2023

    2023 arXiv

  64. [76]

    Unified language-driven zero-shot domain adaptation

    Senqiao Yang, Zhuotao Tian, Li Jiang, and Jiaya Jia. Unified language-driven zero-shot domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23407–23415, June 2024

    2024

  65. [77]

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  66. [79]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv preprint arXiv:2412.18319, 2024

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv preprint arXiv:2412.18319, 2024

    2024 arXiv

  67. [80]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:11809–11822, 2023

    2023

  68. [81]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv:2408.01800, 2024

    2024 arXiv

  69. [82]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. 16

    2025 arXiv

  70. [83]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML, 2024

    2024

  71. [84]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning. PMLR, 2024

    2024

  72. [85]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  73. [86]

    Lmms-eval: Reality check on the evaluation of large multimodal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772, 2024

    2024 arXiv

  74. [87]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320, 2024

    2024 arXiv

  75. [88]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2024

  76. [90]

    Improve vision language model chain-of-thought reasoning

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198, 2024

    2024 arXiv

  77. [91]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024

    arXiv

  78. [92]

    Lyra: An efficient and speech-centric framework for omni-cognition

    Zhisheng Zhong, Chengyao Wang, Yuqi Liu, Senqiao Yang, Longxiang Tang, Yuechen Zhang, Jingyao Li, Tianyuan Qu, Yanwei Li, Yukang Chen, et al. Lyra: An efficient and speech-centric framework for omni-cognition. arXiv preprint arXiv:2412.09501, 2024

    2024 arXiv

  79. [93]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 17 Contents A Related Works 18 A.1 Efficient Vision Language Models . . . . . ....

    2023 arXiv

  80. [94]

    **Question Related to the Image**: Question

  81. [95]

    **Ground Truth Answer**: Ground Truth

  82. [96]

    type": "function

    **Model Predicted Answer**: Prediction Your task is to evaluate the model’s predicted answer against the ground truth answer, based on the context provided by the question related to the image. Consider the following criteria for evaluation: - **Relevance**: Does the predicted...

    1920

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.