Pith. sign in

REVIEW 3 major objections 4 minor 74 references

Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free two-stage prompt loop—localized zoom followed by self-refinement—lets a multimodal large language model recover fine detail lost to downsampling and improve accuracy on high-resolution reasoning benchmarks.

desk verdict A clean training-free two-stage zoom-and-refine recipe that likely helps on high-resolution MLLM benchmarks, but the paper never isolates whether the localized crop or just the second-pass prompt is doing the work. read the letter →

arxiv 2506.01663 v2 pith:HPS7QFEY submitted 2025-06-02 cs.CV

Xuan Yu , Dayan Guan , Yanfeng Gu This is my paper
classification cs.CV
keywords multimodallargelanguagemodelshigh-resolutionimageunderstandingtraining-freeinferencelocalizedzoomself-refinementbounding-boxlocalizationfine-grainedperceptionvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that a multimodal large language model can handle very high-resolution images without any retraining if it is allowed to look twice: once at the whole downsampled image and once at a high-resolution crop of the region it judges most relevant to the question. The proposed procedure, Zoom-Refine, asks the model to give a preliminary answer together with a bounding box, crops that box from the original image, and then feeds the crop back to the model with a prompt to re-check and correct its first answer. On two high-resolution benchmarks, this two-step prompt-only loop improves accuracy over the same model answering in a single pass, with the largest reported gains on diagram/table reasoning and on 8K images. The contribution, read sympathetically, is evidence that much of the fine-detail failure of current MLLMs is an input-formatting problem rather than a missing-capacity problem.

What carries the argument

The carrying mechanism is the two-stage prompt loop: a localized-zoom prompt $P_{\mathrm{loc}}$ that asks the model to answer and to output normalized bounding-box coordinates for the task-relevant region; a deterministic crop operation $I_{\mathrm{crop}} = F_{\mathrm{crop}}(I,B)$ that takes that box from the full-resolution source image; and a self-refinement prompt $P_{\mathrm{refine}}$ that presents the crop as new evidence and asks the model to confirm or revise its initial answer. The crucial design choice is that the crop comes from the original high-resolution image rather than from the downsampled input, so the second pass receives pixel detail the first pass never saw.

What would settle it

Run Zoom-Refine on a high-resolution benchmark with two control conditions: (1) replace the model-predicted box with a random crop of equal area, and (2) replace it with the ground-truth region from human annotation. If random crops match Zoom-Refine's accuracy, localization is not the source of the gain; if oracle crops substantially beat it, then improving box prediction is the highest-leverage improvement. A second check is to count how often the final answer changes from the initial answer and in which direction; if changes are roughly symmetric between correct-to-wrong and wrong-to-correct, the reported accuracy gains would need to be explained by something other than self-correction.

Watch

Extended reading notes

Core claim

The paper's central claim is that the same MLLM that fails on a downsampled high-resolution image already possesses the spatial localization and comparative reasoning needed to recover the lost detail. Zoom-Refine formalizes this as two coupled steps: the model produces an initial answer and a predicted bounding box $B=[x_1,y_1,x_2,y_2]$ from the downsampled image; the box is used to cut a high-resolution crop $I_{\mathrm{crop}}$ from the original image; and the model is then asked to re-evaluate its initial answer with both the crop and the original context in view. In the paper's experiments with 78B-parameter open-weight models, this loop raises weighted-average reasoning accuracy from 49.62 to 53.22 and unweighted-average from 54.31 to 58.70 on the MME-RealWorld benchmark, and raises average accuracy by 5.0 points on the 8K split of HR-Bench. The discovery, if it holds, is that a single fixed prompt can convert single-pass inference into a foveation-and-verification process whose gains come from the model's own output.

Load-bearing premise

The whole method rests on the model's first-pass bounding box landing on the region that actually contains the decisive visual evidence; if the box is wrong, the crop adds nothing and the refinement stage can only re-argue from the same mistaken context.

Editorial extensions

If this is right

  • The paper reports gains across 78B, 14B, 8B, and 2B model scales, so the benefit is not tied to one model size.
  • Because no training is involved, the method can be applied to models whose weights are frozen or inaccessible, requiring only image cropping and text prompting.
  • The reported inference time on the reasoning benchmark is 447 minutes, roughly eight times faster than two compared training-free methods, so the accuracy gain does not require a proportional compute explosion.
  • A single fixed prompt is used for every task and benchmark, so there are no task-specific hyperparameters to tune for deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The localization assumption is the lever most worth testing: an oracle experiment that supplies a human- or ground-truth-annotated region as the crop would separate how much of the gain comes from bounding-box quality and how much from the refinement step itself.
  • The paper leaves iterative zooming unexplored; a natural extension is repeated rounds of crop-and-refine on progressively smaller regions, which could help multi-hop questions but would also compound any localization error.
  • The same loop could be adapted to other fine-grained modalities, such as document screenshots, medical images, or satellite imagery, where the model's first pass can propose a region and the second pass reads it at native resolution.
  • The result hints that some high-resolution understanding failures are actually input-bottleneck failures, implying that test-time compute can substitute for training data in this regime.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Zoom-Refine, a training-free, prompt-only method for high-resolution multimodal understanding. In the Localized Zoom stage, the same MLLM that produces a preliminary answer also predicts a normalized bounding box for the task-relevant image region; a high-resolution crop is extracted from the original image at those coordinates. In the Self-Refinement stage, the MLLM is re-prompted with the original context, the crop, and its initial answer, and is instructed to either confirm or correct the answer. Experiments are reported on MME-RealWorld (reasoning and perception tasks) and HR-Bench (4K and 8K), using InternVL3/2.5 models of several sizes, with comparisons against other training-free methods and timing measurements. The reported results show consistent accuracy gains over the base models (e.g., +3.60 Avg and +4.39 Avg-C for InternVL3-78B on MME-RealWorld reasoning; +5.0 Avg on HR-Bench 8K).

Significance. If the gains are real and causally attributable to the localized zoom, the contribution is practically useful: it improves strong open MLLMs on fine-grained high-resolution benchmarks without retraining, additional parameters, or external expert modules, and it is simple to reproduce from the provided prompts and code. The paper's strengths include consistent improvements across two benchmarks, multiple model families and sizes, a clearly specified two-stage mechanism in Eqs. (1)-(4), a single shared prompt for all experiments, and a public code release. The empirical support is weakened, however, by the absence of variance reporting, statistical tests, and, most importantly, any ablation isolating the localized-crop component from the self-refinement prompt and from arbitrary high-resolution cropping. The paper's main causal claim is therefore not yet fully established.

major comments (3)
  1. [Section 4, Tables 1-3] All accuracy numbers are reported for a single split without variance, repeated runs, or statistical tests. The central claims are differences of roughly 2-5 percentage points (e.g., +3.60 Avg and +4.39 Avg-C for InternVL3-78B in Table 1; +5.0 Avg on HR-Bench 8K in Table 3), and it is not possible to determine whether these are within run-to-run noise. Please report the number of evaluation runs, standard deviations or confidence intervals, and, where practical, a paired significance test over benchmark items.
  2. [Section 3.2, Eqs. (2)-(4); Tables 1-5] There is no component ablation that isolates the localized crop. The reported comparisons are always full Zoom-Refine versus the base model, so the observed gains could come from (a) the second-pass self-refinement prompt alone, which explicitly instructs the model to recheck and correct its previous answer; (b) any high-resolution crop regardless of whether Eq. (2)'s bounding box is task-relevant; or (c) genuinely localized, task-relevant zoom. The self-refinement prompt in Figure 2 even tells the model to ignore the crop if it lacks relevant content, which makes the control conditions especially important. Please add at least: a second-pass refinement with the same prompt but no crop; a crop from a random or fixed center region at the same resolution; and, ideally, a crop from an oracle or ground-truth bounding box. Without such controls, the central claim that localized, task-relevant cropping causes the gains is not supported over simpler explanations.
  3. [Section 3.2, Eq. (2); Section 6] The localization assumption is not empirically tested. Eq. (2) produces a bounding box that is the only link between the initial image and the high-resolution crop, but the paper reports no measurement of box quality (e.g., IoU with a ground-truth region or human evaluation of task relevance) and no analysis of cases where the crop misses the critical evidence. Section 6 concedes localization dependency but does not quantify it, and Section 5, which is titled 'Limitations and Discussion,' does not actually discuss this dependency. Please include a localization-quality analysis and, if feasible, condition final-answer accuracy on whether the predicted box is correct; this would directly test the mechanism claimed in Section 3.2.
minor comments (4)
  1. [Figure 2] The caption states that the system prompt, Localized Zoom prompt, and Self-Refinement prompt are highlighted in orange, blue, and green, respectively, but the printed figure in the manuscript does not clearly label these blocks; consider adding explicit labels (e.g., 'Stage 1' and 'Stage 2') so that the correspondence with Eqs. (2) and (4) is unambiguous.
  2. [Table 1 caption] The sentence 'Avg and Avg-C indicate the weighted and unweighted average accuracy across subtasks in each subtask' is garbled; it should read 'across tasks' or 'across the four subtasks.'
  3. [Tables 1 and 3] Several model names are inconsistently typeset, e.g., 'LLaV A-Next-8B' and 'LLaV A-1.6-34B'; standardize to 'LLaVA' throughout for reproducibility and readability.
  4. [Table 4] The 'Time' comparison lacks the exact evaluation conditions (GPU type, batch size, number of prompts processed, crop resolution, and whether the base-model time includes only one forward pass). These details are needed for the claim that Zoom-Refine offers a favorable performance/cost trade-off relative to DC2 and ZoomEye.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: Zoom-Refine is an empirical, training-free pipeline evaluated on external held-out benchmarks, with no fitted parameter or equation that reduces to the claimed result.

full rationale

Zoom-Refine's claimed gains are empirical measurements against fixed external benchmarks (MME-RealWorld and HR-Bench), not derived quantities. The method is defined by Eq. (2)-(4): the model produces a preliminary answer and bounding box, a crop is extracted from the original image, and the model is prompted to re-evaluate. The final answer Afinal in Eq. (4) is not equal to the preliminary answer Ainit in Eq. (2) by construction; the prompt explicitly instructs the model to ignore the cropped image if it lacks relevant content and to correct or reaffirm its answer based on new evidence. There are no fitted parameters, no calibration on test data, and no result imported from the authors' own prior work. The only self-referential element is that the same MLLM both localizes and later refines, which is an operational property of the method rather than a circular justification. The paper itself acknowledges localization dependency as a limitation in Section 6, but that limitation does not make the benchmark comparison circular. Therefore, the central claim is self-contained and supported by an independent external evaluation protocol.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted and no new entities are introduced. The method rests on three domain assumptions about MLLM localization ability, crop fidelity, and self-refinement reliability, all of which are only indirectly evaluated through final accuracy.

assumptions (3)
  • domain assumption The MLLM can localize task-relevant regions via bounding-box prediction from a downsampled image.
    Introduced in Section 3.2, Eq. (2). If localization fails, the crop extracted in Eq. (3) misses the relevant evidence.
  • domain assumption A high-resolution crop of the predicted region contains the fine details lost in downsampling and is sufficient to resolve the question.
    Assumed in Section 3.2, Eq. (3)-(4). The method presumes the original image retains the needed detail and that the crop captures it.
  • domain assumption The MLLM can perform self-refinement when given new visual evidence, correctly updating its initial answer rather than being anchored to it.
    Assumed in Section 3.2, Self-Refinement and the prompt in Figure 2. Prior work cited by the paper, such as [22] and [34], shows self-correction in LLMs can be unreliable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement." pith.science (2026). https://pith.science/paper/HPS7QFEY

@misc{pith2026250601663,
  author       = {Pith},
  title        = {Pith review of: Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPS7QFEY}},
  note         = {Machine review of arXiv:2506.01663}
}
read the original abstract

Multimodal Large Language Models (MLLM) often struggle to interpret high-resolution images accurately, where fine-grained details are crucial for complex visual understanding. We introduce Zoom-Refine, a novel training-free method that enhances MLLM capabilities to address this issue. Zoom-Refine operates through a synergistic process of \textit{Localized Zoom} and \textit{Self-Refinement}. In the \textit{Localized Zoom} step, Zoom-Refine leverages the MLLM to provide a preliminary response to an input query and identifies the most task-relevant image region by predicting its bounding box coordinates. During the \textit{Self-Refinement} step, Zoom-Refine then integrates fine-grained details from the high-resolution crop (identified by \textit{Localized Zoom}) with its initial reasoning to re-evaluate and refine its preliminary response. Our method harnesses the MLLM's inherent capabilities for spatial localization, contextual reasoning and comparative analysis without requiring additional training or external experts. Comprehensive experiments demonstrate the efficacy of Zoom-Refine on two challenging high-resolution multimodal benchmarks. Code is available at \href{https://github.com/xavier-yu114/Zoom-Refine}{\color{magenta}github.com/xavier-yu114/Zoom-Refine}

Figures

Figures reproduced from arXiv: 2506.01663 by the authors.

Figure 1
Figure 1. The framework of Zoom-Refine. The MLLM initially processes an input [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Prompt engineering of Zoom-Refine. System prompt, "Localized Zoom" prompt, and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Representative examples from the MME-Realworld benchmark show InternVL3-78B with [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

74 extracted references · 39 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint, 2024

    work page 2024

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint, 2023

    work page 2023

  3. [3]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    Dualfocus: Integrat- ing macro and micro perspectives in multi-modal large language models

    Yuhang Cao, Pan Zhang, Xiaoyi Dong, Dahua Lin, and Jiaqi Wang. Dualfocus: Integrat- ing macro and micro perspectives in multi-modal large language models. arXiv preprint arXiv:2402.14767, 2024

    arXiv 2024

  5. [5]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023

    arXiv

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370–387. Springer, 2024

    2024

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  8. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint, 2024

    work page 2024

Show all 74 references
  1. [9]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  2. [10]

    Chain-of-verification reduces hallucination in large language models

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 3563–3578, 2024

    2024

  3. [11]

    Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wa...

    2024

  4. [12]

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd.Advances in Neural Informatio...

    2024

  5. [13]

    Active vision: The psychology of looking and seeing

    John M Findlay and Iain D Gilchrist. Active vision: The psychology of looking and seeing . Oxford University Press, 2003

    2003

  6. [14]

    Small language model can self-correct

    Haixia Han, Jiaqing Liang, Jie Shi, Qianyu He, and Yanghua Xiao. Small language model can self-correct. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18162–18170, 2024

    2024

  7. [15]

    Eye movements in natural behavior

    Mary Hayhoe and Dana Ballard. Eye movements in natural behavior. Trends in cognitive sciences, 9(4):188–194, 2005. 10

    2005

  8. [16]

    Human gaze control during real-world scene perception.Trends in cognitive sciences, 7(11):498–504, 2003

    John M Henderson. Human gaze control during real-world scene perception.Trends in cognitive sciences, 7(11):498–504, 2003

    2003

  9. [17]

    Cogvlm2: Visual language models for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500, 2024

    2024 arXiv

  10. [18]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895, 2024

    2024 arXiv

  11. [19]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  12. [20]

    Ma- tryoshka query transformer for large vision-language models

    Wenbo Hu, Zi-Yi Dou, Liunian Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Ma- tryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37:50168–50188, 2024

    2024

  13. [21]

    Aggregate-and-adapt natural language prompts for downstream generalization of clip

    Chen Huang, Skyler Seto, Samira Abnar, David Grangier, Navdeep Jaitly, and Joshua Susskind. Aggregate-and-adapt natural language prompts for downstream generalization of clip. Advances in Neural Information Processing Systems, 37:81077–81104, 2024

    2024

  14. [22]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2024

    2024

  15. [23]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  16. [24]

    Maven: An effective multi-granularity hybrid visual encoding framework for multimodal large language model

    Chaoya Jiang, Hongrui Jia, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. Maven: An effective multi-granularity hybrid visual encoding framework for multimodal large language model. Advances in Neural Information Processing Systems, 37:10199...

    2024

  17. [25]

    When can LLMs actually correct their own mistakes? a critical survey of self-correction of LLMs

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can LLMs actually correct their own mistakes? a critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics, 12:1417–1440, 2024

    2024

  18. [26]

    Language models can solve computer tasks

    Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36:39648–39677, 2023

    2023

  19. [27]

    Training language models to self-correct via reinforcement learning

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. Trai...

    2025

  20. [28]

    What matters when building vision-language models? Advances in Neural Information Processing Systems, 37:87874–87907, 2024

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37:87874–87907, 2024

    2024

  21. [29]

    Llava-next: Stronger llms supercharge multimodal capabilities in the wild

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild. 2024

    2024

  22. [30]

    Otter: A multi-modal model with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023

    2023 arXiv

  23. [31]

    Dyfo: A training-free dynamic fo- cus visual search for enhancing lmms in fine-grained visual understanding

    Geng Li, Jinglin Xu, Yunzhen Zhao, and Yuxin Peng. Dyfo: A training-free dynamic fo- cus visual search for enhancing lmms in fine-grained visual understanding. arXiv preprint arXiv:2504.14920, 2025. 11

    2025 arXiv

  24. [32]

    Cumo: Scaling multimodal llm with co-upcycled mixture-of-experts

    Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal llm with co-upcycled mixture-of-experts. Advances in Neural Information Processing Systems, 37:131224–131246, 2024

    2024

  25. [33]

    Hrvqa: A visual question answering benchmark for high-resolution aerial images

    Kun Li, George V osselman, and Michael Ying Yang. Hrvqa: A visual question answering benchmark for high-resolution aerial images. ISPRS Journal of Photogrammetry and Remote Sensing, 214:65–81, 2024

    2024

  26. [34]

    When hindsight is not 20/20: Testing limits on reflective thinking in large language models

    Yanhong Li, Chenghao Yang, and Allyson Ettinger. When hindsight is not 20/20: Testing limits on reflective thinking in large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024 , pages ...

    2024

  27. [35]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint, 2024

    2024

  28. [36]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607, 2023

    2023 arXiv

  29. [37]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26689–26699, 2024

    2024

  30. [38]

    Visual anchors are strong information aggregators for multimodal large language model

    Haogeng Liu, Quanzeng You, Xiaotian Han, Yongfei Liu, Huaibo Huang, Ran He, and Hongxia Yang. Visual anchors are strong information aggregators for multimodal large language model. Advances in Neural Information Processing Systems, 37:17696–17718, 2024

    2024

  31. [39]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024

    2024

  32. [40]

    Llava-next: improved reasoning, ocr, and world knowledge (2024)

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: improved reasoning, ocr, and world knowledge (2024). URL https://llava-vl. github. io/blog/2024-01-30-llava-next, 2(5):6, 2024

    2024

  33. [41]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  34. [42]

    Textmonkey: An ocr-free large multimodal model for understanding document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024

    2024 arXiv

  35. [43]

    Deepseek-vl: towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  36. [44]

    Task-to-instance prompt learning for vision-language models at test time

    Zhihe Lu, Jiawang Bai, Xin Li, Zeyu Xiao, and Xinchao Wang. Task-to-instance prompt learning for vision-language models at test time. IEEE Transactions on Image Processing, 2025

    2025

  37. [45]

    Textcot: Zoom in for enhanced multimodal text-rich image understanding

    Bozhi Luan, Hao Feng, Hong Chen, Yonghui Wang, Wengang Zhou, and Houqiang Li. Textcot: Zoom in for enhanced multimodal text-rich image understanding. arXiv preprint arXiv:2404.09797, 2024

    2024 arXiv

  38. [46]

    Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint, 2024

    2024

  39. [47]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems , 36:46534–46594, 2023

    2023

  40. [48]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. Technical report, OpenAI, 2023. 12

    2023

  41. [49]

    Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies

    Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484–...

    2024

  42. [50]

    Orienting of attention

    Michael I Posner. Orienting of attention. Quarterly journal of experimental psychology , 32(1):3–25, 1980

    1980

  43. [51]

    Zoomer: Adaptive image focus optimization for black-box mllm

    Jiaxu Qian, Chendong Wang, Yifan Yang, Chaoyun Zhang, Huiqiang Jiang, Xufang Luo, Yu Kang, Qingwei Lin, Anlan Zhang, Shiqi Jiang, et al. Zoomer: Adaptive image focus optimization for black-box mllm. arXiv preprint arXiv:2505.00742, 2025

    2025

  44. [52]

    Self-reflection in llm agents: Effects on problem-solving performance

    Matthew Renze and Erhan Guven. Self-reflection in llm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682, 2024

    2024 arXiv

  45. [53]

    Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems...

    2024

  46. [54]

    Zoomeye: Enhancing multimodal llms with human-like zooming capabilities through tree-based image exploration

    Haozhan Shen, Kangjia Zhao, Tiancheng Zhao, Ruochen Xu, Zilun Zhang, Mingwei Zhu, and Jianwei Yin. Zoomeye: Enhancing multimodal llms with human-like zooming capabilities through tree-based image exploration. arXiv preprint arXiv:2411.16044, 2024

    2024 arXiv

  47. [55]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  48. [56]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  49. [57]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processi...

    2024

  50. [58]

    LLMs cannot find reasoning errors, but can correct them given the error location

    Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, and Tony Mak. LLMs cannot find reasoning errors, but can correct them given the error location. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 202...

    2024

  51. [59]

    Lever- aging visual tokens for extended text contexts in multi-modal learning

    Alex Jinpeng Wang, Linjie Li, Yiqi Lin, Min Li, Lijuan Wang, and Mike Zheng Shou. Lever- aging visual tokens for extended text contexts in multi-modal learning. Advances in Neural Information Processing Systems, 37:14325–14348, 2024

    2024

  52. [60]

    MME- RealWorld: A Challenging Benchmark towards Real-World Multimodal Understanding

    Hai-Chen Wang, Hong-Yu Yu, Tian-Yi Zhao, Yue-Hen Li, Hao-Ran Li, and Xiang Xue. MME- RealWorld: A Challenging Benchmark towards Real-World Multimodal Understanding. arXiv preprint arXiv:2409.00249 (To Appear ICLR 2025), 2024

    2025 arXiv

  53. [61]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  54. [62]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475–121499, 2024

    2024

  55. [63]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. In Proceedings of the AAAI Conference on Artificial I...

    2025

  56. [64]

    Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. Advances in Neural Information Processing Sy...

    2024

  57. [65]

    V*: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024

    2024

  58. [66]

    Large language models can self-correct with key condition verification

    Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang. Large language models can self-correct with key condition verification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12846–12867, 2024

    2024

  59. [67]

    Graph- based unsupervised disentangled representation learning via multimodal large language models

    Baao Xie, Qiuyu Chen, Yunnan Wang, Zequn Zhang, Xin Jin, and Wenjun Zeng. Graph- based unsupervised disentangled representation learning via multimodal large language models. Advances in Neural Information Processing Systems, 37:103101–103130, 2024

    2024

  60. [68]

    mPLUG-2: A modularized multi-modal foundation model across text, image and video

    Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, and Jingren Zhou. mPLUG-2: A modularized multi-modal foundation model across text, image and video. In ICML, 2023

    2023

  61. [69]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint, 2024

    2024

  62. [70]

    Learning to check: Unleashing potentials for self-correction in large language models

    Che Zhang, Zhenyang Xiao, Chengcheng Han, Yixin Lian, and Yuejian Fang. Learning to check: Unleashing potentials for self-correction in large language models. arXiv preprint arXiv:2402.13035, 2024

    2024 arXiv

  63. [71]

    Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition

    Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112, 2023

    2023 arXiv

  64. [72]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint, 2024

    2024

  65. [73]

    Wings: Learning multimodal llms without text-only forgetting

    Yi-Kai Zhang, Shiyin Lu, Yang Li, Yanqing Ma, Qingguo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Wings: Learning multimodal llms without text-only forgetting. Advances in Neural Information Processing Systems, 37:31828–31853, 2024

    2024

  66. [74]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 14

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.