Pith. sign in

REVIEW 5 major objections 5 minor 83 references

AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AutoV claims that the optimal visual prompt for a vision-language model is instance-dependent, and that a lightweight network trained on loss-based rankings can retrieve it without fine-tuning.

desk verdict New, simple, and mostly solid: AutoV makes visual prompt selection a learned ranking problem and backs it with broad experiments, though a few evaluation gaps need fixing. read the letter →

arxiv 2506.16112 v4 pith:Q755SX4I submitted 2025-06-19 cs.CV

Yuan Zhang , Chun-Kai Fan , Sicheng Yu , Junwen Pan , Tao Huang , Ming Lu , Kuan Cheng , Qi She
show 1 more author
Shanghang Zhang
This is my paper · ORCID
classification cs.CV
keywords visualpromptretrievallargevision-languagemodelsloss-orientedrankingpairwiserewardlossinstance-adaptivepromptingautomaticsupervisionVQAbenchmarksmodeltransferability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that visual prompting has reached a ceiling under fixed designs, and the remaining gains come from retrieval: for each image-question pair, first pick the right visual prompt and then let the LVLM answer. It introduces AutoV, a lightweight ranking network that scores candidate prompts (attention heatmaps, red circles, blur masks) against the query and feeds the top choice into an off-the-shelf LVLM. To avoid human labels, prompt quality is measured automatically by a pretrained LVLM: the lower the language-modeling loss of a prompted answer against ground truth, the better the prompt, and pairwise comparisons of these losses train the ranker. AutoV reports consistent gains across more than a dozen benchmarks and several open and closed LVLMs, including 10.2 points on VizWiz for LLaVA-OneVision and 3.8 points on MMMU for Qwen2.5-VL, with no fine-tuning of the backbone. The transferable claim is that this loss-based ranking generalizes across tasks, model families, and even visual prompt types never seen during training.

What carries the argument

The machinery is a small ranking network built on an unmodified LVLM. For $n$ prompt candidates, the LVLM's frozen visual encoder and projector produce visual tokens $V_i$; the first LLM decoder layer mixes them with query text tokens $T$, two single-FFN mapping modules lower the dimension of both modalities, and a cross-attention scorer outputs a reward scalar $s(VP_i)$ per candidate. The training signal is the pairwise reward loss $\mathcal{L}_r=-\frac{1}{\binom{n}{2}}\mathbb{E}[\log\sigma(s(VP_c)-s(VP_r))]$, which pushes the lower-loss prompt above the higher-loss prompt for every pair. Labels come from a data-generation pipeline in which a frozen LVLM computes the conditional language-modeling loss of each prompted answer against ground truth, ranks the prompts, and filters out low-variance and high-average-loss cases that would reflect language priors; each training sample is a quadruple $\langle$query, candidate prompts, rank, loss$\rangle$. At inference a cheap pre-filter removes the candidate whose visual features are farthest in cosine distance from the others before the ranker outputs its top pick.

What would settle it

A direct test: take a held-out set of image-question pairs with ground-truth answers, compute the language-modeling loss and the downstream answer accuracy for every prompt in the pool, and check whether the lowest-loss prompt is the most accurate one with statistical reliability; if not, or if AutoV's top pick agrees with that oracle at near-chance rates, the central claim is refuted.

Watch

Extended reading notes

Core claim

AutoV's central discovery is that a superior visual prompt can be recognized by the loss it induces in a frozen LVLM: on a given image-question pair, a prompt that lowers the conditional language-modeling loss relative to the ground-truth answer is treated as better, and this relative loss signal is enough to supervise a ranker without human annotation. The ranking network takes each candidate's visual tokens (extracted with the LVLM's own frozen visual encoder and projector), fuses them with query text through the first LLM decoder layer, and scores the pair with cross-attention; training uses a pairwise reward loss over all C(n,2) prompt pairs in an instance. Once trained, AutoV selects the highest-scoring prompt at inference, after removing the cosine-farthest candidate as a pre-filter. The paper's broader claim is that this learned retrieval policy is model-agnostic: it transfers to other open LVLMs and to closed-source API models without retraining, and enlarging the candidate pool with unseen prompt types continues to help.

Load-bearing premise

The method rests on the assumption that a better visual prompt makes the LVLM's language-modeling loss on the ground-truth answer smaller, and that the ranking learned from those losses with one 7B teacher still picks the right prompt for other models and for prompt types the ranker never saw.

Editorial extensions

If this is right

  • Per-instance retrieval beats any single fixed visual prompt: AutoV's pairwise ranking improves average accuracy over the three head-to-head benchmarks by about 2.3 points on LLaVA-7B, versus about 0.6–1.1 points for regression, MoE gating, or the best individual attention prompt.
  • The retrieval policy transfers across model families without retraining: the same ranker trained on LLaVA-OneVision-style data improves InternVL2 and Qwen2.5-VL by average gains of 3.2% and 3.4%, and closed-source models by 6.4 and 6.0 points on average, with per-benchmark gains up to 9.6 points on VizWiz.
  • Bigger prompt pools do not hurt and keep helping: expanding from 4 to 8 candidates, including two prompt types unseen in training, improves all three headline benchmarks and adds a further average gain of about 0.3 points.
  • Pairwise ranking is the right supervision format: under equal data and training budget it outperforms regression, list-wise ranking, and a Gumbel-softmax gating network, because it preserves the relative order of prompt quality without having to predict absolute loss values.
  • The gains are reproducible across seeds: on MMMU the paper reports a mean improvement of +2.06% over five runs with p≤0.05.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the training labels are token-level surprisal against ground truth rather than human-judged answer quality, the ranker can only learn prompt effects that reduce that surprisal; a prompt that improves an answer through longer reasoning without lowering the loss would stay invisible to the training signal, a mismatch the paper does not measure.
  • If the same loss-scoring premise holds for other input dimensions, the AutoV recipe should transfer to selecting video frames, spatial crops, or audio segments as 'prompts', since those are also candidates a frozen LVLM can score by loss; the paper tests static images only.
  • The observed saturation near a pool size of eight suggests a practical ceiling on prompt-type diversity; a natural next experiment is to test AutoV with dozens of prompt families and measure whether the lightweight cross-attention scorer needs more capacity to use them.
  • Because transfer works to closed-source APIs, AutoV could operate as a local pre-filter: run the ranker on the user's image and query, send only the top-ranked prompt overlay to the API, and leave the API itself untouched.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes AutoV, a lightweight ranking network that selects an instance-adaptive visual prompt for an LVLM from a candidate pool. Prompt-level supervision is obtained automatically by evaluating candidate prompts with a pretrained LVLM and using the resulting conditional language-modeling loss to define a preference order; the ranking network is trained with a pairwise reward loss (Eq. 4) and applied at inference to select one prompt per image-query pair. Experiments on fourteen benchmarks and several open- and closed-source LVLMs report consistent gains, e.g., +10.2% on VizWiz for LLaVA-OneVision and +3.8% on MMMU for Qwen2.5-VL.

Significance. The idea of replacing hand-designed visual prompts with a learned, instance-specific retrieval mechanism is timely and potentially useful, and the paper provides a large experimental surface: multiple LVLMs, multiple benchmark families, transfer to closed-source models, an overhead analysis, and ablations on pool size, layer choice, and pre-filtering. The automatic supervision pipeline is attractive because it avoids human annotation of prompt quality. However, the central claim rests on an unvalidated equivalence between lower teacher loss and better prompt quality, and the evaluation is weakened by at least one training/evaluation overlap. If the loss-ranking assumption is made credible and the evaluation cleaned, the contribution would be solid; in its current form the evidence is suggestive but not conclusive.

major comments (5)
  1. [Appendix A.2 / Table 1] The training data include 15K TextVQA samples (Appendix A.2), and Table 1 reports AutoV's performance on TextVQA (column 'VQAText'). The reported gains on TextVQA are therefore not a clean out-of-distribution assessment. Please either remove TextVQA from the training set and re-run the affected rows, or report results on a held-out split of TextVQA that is verifiably disjoint from the training sample.
  2. [Section 3.3] The core supervision assumption—'a superior visual prompt should induce lower conditional language modeling loss'—is never directly validated against actual response quality. The paper does not report any correlation between the teacher's per-prompt loss and correctness or human judgment, nor does it check whether the teacher's preferred prompt is also the best prompt for the target LVLM (e.g., Qwen2.5-VL or InternVL2). Appendix D only addresses language-prior filtering; it does not establish ranking fidelity. Please add experiments that (a) measure agreement between loss ranking and task accuracy on a labeled sample, and (b) quantify how often AutoV's selected prompt matches the empirically best prompt for the deployed model.
  3. [Section 3.4 / Table 5] The pre-filtering step contributes a sizable part of the reported gains: in Table 5, removing pre-filtering lowers AutoV's MMMU score from 38.7 to 38.3 with 6 candidates and from 38.9 to 38.4 with 8 candidates. Because the method always includes pre-filtering, the paper does not isolate the contribution of the learned ranking from the combination of pre-filtering and prompt-pool diversity. A 'random selection + pre-filtering' condition with the same candidate pool and repeated seeds is needed to support the claim that the learned ranking, rather than the pre-filtering heuristic or mere prompt diversity, drives the improvements.
  4. [Section 5.7 / Appendix F] Statistical significance is demonstrated for only one benchmark (MMMU) across five seeds, yet Section 5.7 states that 'similar significance is consistently observed across other benchmarks.' Appendix F provides no such results for the other benchmarks. Either report significance tests for all benchmarks in Table 1 or temper the claim accordingly. In addition, the variance and mean thresholds used to filter (VP, query) pairs in Section 3.3 are never specified; without these values the data-generation pipeline is not reproducible and the sensitivity of the method to these thresholds is unknown.
  5. [Section 4.2 / Appendix C] There is an inconsistency about which model generated the training labels: Section 4.2 says AutoV is 'pre-trained on LLaVA-OneVision' and Table 6 uses that policy, while Appendix C says 'LLaVA-v1.5 7B is consistently adopted as the reference LVLM for generating loss.' Please clarify which model produced the loss labels, which model's first decoder layer is used for the modality-interaction module, and which pair underlies each transfer experiment. This is load-bearing for the transferability claims because the ranking signal is assumed to transfer across teacher-student combinations.
minor comments (5)
  1. [Table 1] In the LLaVA-OneVision block, the Circle result on MMVet is listed as '47.2 + 1.6' even though the baseline is 48.8; the sign of the difference should be negative.
  2. [Abstract / Section 1] The claim that AutoV works 'without additional fine-tuning' of the LVLM is clear, but the phrasing could be read as 'no training at all'; please state explicitly that only the lightweight ranking network is trained.
  3. [Figures 2 and 3] The mathematical notation in Figures 2 and 3 is partially garbled (e.g., subscripts and the loss expression), making the pipeline difficult to follow. Please regenerate the figures with clean, readable equations.
  4. [Section 3.4] The sentence 'we input multiple candidate visual prompts along with the query into the large vision-language model' is imprecise: AutoV encodes candidate prompt images with the visual encoder, but the decoder is only applied to the selected candidate. Clarify the inference flow so that the FLOPs analysis in Table 7 is easier to interpret.
  5. [References] Some benchmark citations are informal (e.g., RealWorldQA is cited as an arXiv technical report and as 'Grok-1.5 vision preview' in reference [67]); please provide canonical dataset references.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the loss-based ranking signal is evaluated against external task accuracy, not against the training loss itself; only minor non-load-bearing self-citations appear.

full rationale

AutoV's derivation chain is: (i) define a candidate pool of visual prompts; (ii) label each (image, query, prompt) triple by the conditional language-modeling loss of a fixed pretrained LVLM (LLaVA-1.5-7B) against the ground-truth answer; (iii) train a lightweight ranking network with pairwise reward loss (Eq. 4) to reproduce that loss ordering; (iv) at inference, retrieve the highest-scoring prompt and evaluate with task accuracy on external benchmarks (Tables 1, 2, and 6). The reported improvements are measured on benchmark accuracy, not on the training loss, and no benchmark constant is fitted anywhere in the pipeline. The central assumption that lower conditional loss implies a better visual prompt (Section 3.3) is an empirical proxy for prompt quality, not a definitional identity, and the paper provides indirect supporting evidence in Appendix D. The few self-citations ([79], [80], [81]) are used as supporting references for layer choice, prompt candidates, and loss-calibration plausibility; they are not the sole justification and are accompanied by ablations (Table 5) or external citations ([21]), so they do not constitute load-bearing circularity. The main residual concern, that the teacher loss ranking may not transfer to target models or to actual answer quality, is an empirical validity question rather than a circularity of the derivation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The ranking network relies on a small set of hand-chosen hyperparameters and domain assumptions. The most consequential is the loss-as-quality proxy: all labels come from a pretrained LVLM's conditional loss, so if that loss does not track answer quality, the supervision is misaligned. The candidate pool, filtering thresholds, and interaction layer are chosen by hand or ablation; none is fitted to the reported benchmark numbers, so the central comparison is not circular.

free parameters (4)
  • Loss variance and mean filtering thresholds = unspecified
    Section 3.3 removes (VP, query) pairs with low loss variance or high average loss; thresholds are not given numerically and directly shape the training distribution.
  • Candidate pool composition = API CLIP layers 15/20/22/23, RedCircle, FGVP (6 prompts; up to 8 in scaling study)
    The prompt candidates were chosen by hand (Section 4.2 footnote and Table 4); retrieval performance depends on this pool.
  • Modality interaction layer index = 0th LLM layer
    Selected via ablation in Table 5 Part 1; the ranking network's input representation changes with this choice.
  • Mapping module output dimension h = not reported
    The vision and text mapping modules use output dimension h, which is not specified in the paper; this affects ranker capacity and FLOPs.
assumptions (4)
  • domain assumption Lower conditional language modeling loss of a pretrained LVLM indicates a better visual prompt.
    Section 3.3: 'a superior visual prompt should induce lower conditional language modeling loss given the visual-question pair'; this is the basis for all labels.
  • domain assumption The first LLM layer provides sufficient modality interaction for ranking.
    Section 3.2 and Table 5 Part 1; adopted based on FastV/LLaVA-Mini observations, with an empirical ablation.
  • domain assumption CLIP features of prompted images contain enough information to rank prompts.
    Section 3.1 encodes prompts with the LVLM's CLIP encoder and ranks from those tokens; no direct verification that the feature space is sufficient.
  • domain assumption Filtered loss labels are not dominated by language priors.
    Section 3.3 and Appendix D use variance filtering and GPT-4o checks to reduce language-prior cases; this assumption is needed for loss to reflect visual grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs." pith.science (2026). https://pith.science/paper/Q755SX4I

@misc{pith2026250616112,
  author       = {Pith},
  title        = {Pith review of: AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q755SX4I}},
  note         = {Machine review of arXiv:2506.16112}
}
abstract

Inspired by text prompts in large language models, visual prompts have been explored to enhance the perceptual capabilities of large vision-language models (LVLMs). However, performance tends to saturate under single visual prompt designs, making further prompt engineering increasingly ineffective. To address this limitation, we shift from prompt engineering to prompt retrieval and propose AutoV, a lightweight framework for instance-adaptive visual prompt identification. Given an input image and a textual query, AutoV automatically locates the most suitable visual prompt from a diverse candidate pool. Training such a retrieval framework requires prompt-level supervision, yet prompt quality is inherently ambiguous and difficult to assess reliably, even for humans. To enable automatic supervision, we evaluate visual prompts using a pre-trained LVLM and label them according to their prediction losses. Using the loss-oriented ranking as a robust training signal, AutoV learns to retrieve the query-aware optimal prompt for each instance without manual annotation. Experiments indicate that AutoV enhances the performance of various LVLMs on image understanding, captioning, grounding, and classification tasks. For example, AutoV improves LLaVA-OV by $\textbf{10.2}\%$ on VizWiz and boosts Qwen2.5-VL by $\textbf{3.8}\%$ on MMMU, respectively.

Figures

Figures reproduced from arXiv: 2506.16112 by the authors.

Figure 1
Figure 1. Motivation of AutoV. (a) Performance Saturation. Existing visual prompts approach benchmark ceilings, limiting further gains from prompt engineering. (b) La￾beling Difficulty and Task Diversity. Optimal prompts vary across tasks, while prompt quality is hard to reliably annotate, motivating automatic supervision. regions. However, existing approaches predominantly rely on fixed, heuristically designed prompts. While… view at source ↗
Figure 2
Figure 2. The illustration of AutoV. It comprises four key components: representa￾tion extraction from candidates, ranking network for reward score, reward-supervised training, and inference. It retrieves the prompt tailored to each query-image pair. concise and explicit, while allowing guidance that reaches fine-grained, pixel￾level details. Early work [52] shows that simply drawing a red circle around a target object steers… view at source ↗
Figure 3
Figure 3. The reward loss of AutoV. As a concrete example, we illustrate the pair￾wise combination process using the visual prompts from [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Retrieval distribution on MMVet. The plots are inde￾pendent accuracy for each prompt. We visualize the prompt distribution of re￾trieval strategies within VP1-VP4 for signif￾icance [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Visualization of retrieval results faced with different queries. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Visual analysis of the composition of training data. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Visualization of a training sample in AutoV. [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 43 canonical work pages

  1. [1]

    arXiv:2303.08774 (2023) 1, 3

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv:2303.08774 (2023) 1, 3

    arXiv 2023

  2. [2]

    Advances in neural information processing systems35, 23716– 23736 (2022) 3

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Men- sch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems35, 23716– 23736 (2022) 3

    2022

  3. [3]

    arXiv:2309.16609 (2023) 1, 8

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv:2309.16609 (2023) 1, 8

    arXiv 2023

  4. [4]

    arXiv:2308.12966 (2023) 1, 3

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-VL: A frontier large vision-language model with versatile abili- ties. arXiv:2308.12966 (2023) 1, 3

    arXiv 2023

  5. [5]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) 1, 3, 8

    arXiv 2025

  6. [6]

    arXiv preprint arXiv:2404.18930 (2024) 3

    Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., Shou, M.Z.: Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930 (2024) 3

    arXiv 2024

  7. [7]

    Advances in Neural Information Processing Systems35, 32897– 32912 (2022) 4

    Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O.K., Aggarwal, K., Som, S., Piao, S., Wei, F.: Vlmo: Unified vision-language pre-training with mixture-of- modality-experts. Advances in Neural Information Processing Systems35, 32897– 32912 (2022) 4

    2022

  8. [8]

    arXiv:2401.02954 (2024) 1, 3

    Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al.: Deepseek LLM: Scaling open-source language models with longtermism. arXiv:2401.02954 (2024) 1, 3

    arXiv 2024

Show all 83 references
  1. [9]

    In: Proceedings of the 23nd annual ACM symposium on User interface software and technology

    Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., White, S., et al.: Vizwiz: nearly real-time answers to visual questions. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology. p...

    2010

  2. [10]

    Advances in neural information processing systems (2020) 1, 3

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems (2020) 1, 3

    2020

  3. [11]

    arXiv preprint arXiv:2403.17297 (2024) 3, 8

    Cai,Z.,Cao,M.,Chen,H.,Chen,K.,Chen,K.,Chen,X.,Chen,X.,Chen,Z.,Chen, Z., Chu, P., et al.: Internlm2 technical report. arXiv preprint arXiv:2403.17297 (2024) 3, 8

    2024 arXiv

  4. [12]

    In: European Conference on Computer Vision (2024) 12

    Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., Chang, B.: An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In: European Conference on Computer Vision (2024) 12

    2024

  5. [13]

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al.: Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems (2024) 8, 24

    2024

  6. [14]

    Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollar, P., Zitnick, C.L.: Microsoft coco captions: Data collection and evaluation server (2015),https:// arxiv.org/abs/1504.0032524

    2015 arXiv

  7. [15]

    arXiv preprint arXiv:2412.05271 (2024) 8 AutoV 17

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024) 8 AutoV 17

    2024 arXiv

  8. [16]

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu,X.,Lu,L.,Li,B.,Luo,P.,Lu,T.,Qiao,Y.,Dai,J.:InternVL:Scalingupvision foundationmodelsandaligningforgenericvisual-linguistictasks.arXiv:2312.14238 (2023) 1, 3, 8

    2023 arXiv

  9. [17]

    In: 2009 IEEE conference on computer vision and pattern recognition (2009) 8

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition (2009) 8

    2009

  10. [18]

    arXiv preprint arXiv:2010.11929 (2020) 3

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020) 3

    2020 arXiv

  11. [19]

    Journal of Machine Learning Research 23(120), 1–39 (2022) 4, 11

    Fedus,W.,Zoph,B.,Shazeer,N.:Switchtransformers:Scalingtotrillionparameter models with simple and efficient sparsity. Journal of Machine Learning Research 23(120), 1–39 (2022) 4, 11

    2022

  12. [20]

    In: European Conference on Computer Vision (2024) 8, 24

    Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N.A., Ma, W.C., Krishna, R.: Blink: Multimodal large language models can see but not per- ceive. In: European Conference on Computer Vision (2024) 8, 24

    2024

  13. [21]

    In: International conference on machine learning

    Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International conference on machine learning. pp. 1321–1330. PMLR (2017) 6

    2017

  14. [22]

    Hsu,H.,Lachenbruch,P.A.:Pairedttest.WileyStatsRef:statisticsreferenceonline (2014) 13

    2014

  15. [23]

    arXiv preprint arXiv:2402.14683 (2024) 3

    Huang, W., Liu, H., Guo, M., Gong, N.Z.: Visual hallucinations of multi-modal large language models. arXiv preprint arXiv:2402.14683 (2024) 3

    2024 arXiv

  16. [24]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. pp. 6700–6709 (2019) 23

    2019

  17. [25]

    arXiv preprint arXiv:2410.21276 (2024) 1, 13

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024) 1, 13

    2024 arXiv

  18. [26]

    arXiv preprint arXiv:2404.04514 (2024) 3, 12

    Jiang, S., Zhang, Y., Zhou, C., Jin, Y., Feng, Y., Wu, J., Liu, Z.: Joint visual and text prompting for improved object-centric perception with multimodal large language models. arXiv preprint arXiv:2404.04514 (2024) 3, 12

    2024 arXiv

  19. [27]

    In: Moschitti, A., Pang, B., Daelemans, W

    Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: ReferItGame: Referring to objects in photographs of natural scenes. In: Moschitti, A., Pang, B., Daelemans, W. (eds.) Proceedings of the 2014 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP). pp. 787–79...

    2014 doi

  20. [28]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023) 22

    2023

  21. [29]

    Advances in neural information processing systems35, 22199–22213 (2022) 1

    Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. Advances in neural information processing systems35, 22199–22213 (2022) 1

    2022

  22. [30]

    International journal of computer vision123, 32–73 (2017) 23 18 Zhang et al

    Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision123, 32–73 (2017) 23...

    2017

  23. [31]

    arXiv preprint arXiv:2408.03326 (2024) 1, 3, 8

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024) 1, 3, 8

    2024 arXiv

  24. [32]

    In: International conference on machine learning (2023) 3

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning (2023) 3

    2023

  25. [33]

    arXiv:2403.18814 (2024) 3

    Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., Jia, J.: Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403.18814 (2024) 3

    2024 arXiv

  26. [34]

    arXiv:2305.10355 (2023) 23

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hallucination in large vision-language models. arXiv:2305.10355 (2023) 23

    2023 arXiv

  27. [35]

    In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13. pp. 740–755...

    2014

  28. [36]

    Lin, W., Wei, X., An, R., Gao, P., Zou, B., Luo, Y., Huang, S., Zhang, S., Li, H.: Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want (2024) 1, 3

    2024

  29. [37]

    arXiv preprint arXiv:2410.05963 (2024) 22

    Lin, Z., Wang, Y., Tang, Z.: Training-free open-ended object detection and seg- mentation via attention as prompts. arXiv preprint arXiv:2410.05963 (2024) 22

    2024 arXiv

  30. [38]

    arXiv:2310.03744 (2023) 1, 3, 8, 22, 24

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. arXiv:2310.03744 (2023) 1, 3, 8, 22, 24

    2023 arXiv

  31. [39]

    Advances in neural information processing systems (2024) 8

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems (2024) 8

    2024

  32. [40]

    arXiv preprint arXiv:2310.02255 (2023) 8, 24

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.W., Galley, M., Gao, J.: Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 (2023) 8, 24

    2023 arXiv

  33. [41]

    In: 2019 international conference on document analysis and recognition (ICDAR)

    Mishra, A., Shekhar, S., Singh, A.K., Chakraborty, A.: Ocr-vqa: Visual question answering by reading text in images. In: 2019 international conference on document analysis and recognition (ICDAR). pp. 947–952. IEEE (2019) 23

    2019

  34. [42]

    arXiv preprint arXiv:2304.07193 (2023) 22

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023) 22

    2023 arXiv

  35. [43]

    Advances in neural information processing sys- tems35, 27730–27744 (2022) 7

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in neural information processing sys- tems35, 27730–27744 (2022) 7

    2022

  36. [44]

    In: Isabelle, P., Charniak, E., Lin, D

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Isabelle, P., Charniak, E., Lin, D. (eds.) Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. pp. 311–318. Association for C...

    2002

  37. [45]

    arXiv:2304.03277 (2023) 1, 3

    Peng, B., Li, C., He, P., Galley, M., Gao, J.: Instruction tuning with gpt-4. arXiv:2304.03277 (2023) 1, 3

    2023 arXiv

  38. [46]

    arXiv preprint arXiv:2410.08876 (2024) 4

    Qi, J., Xu, Z., Shao, R., Chen, Y., Di, J., Cheng, Y., Wang, Q., Huang, L.: Rora-vlm: Robust retrieval-augmented vision language models. arXiv preprint arXiv:2410.08876 (2024) 4

    2024 arXiv

  39. [47]

    arXiv preprint arXiv:2406.05814 (2024) 3 AutoV 19

    Qu, L., Li, H., Wang, T., Wang, W., Li, Y., Nie, L., Chua, T.S.: Unified text-to- image generation and retrieval. arXiv preprint arXiv:2406.05814 (2024) 3 AutoV 19

    2024 arXiv

  40. [48]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 4,...

    2021

  41. [49]

    OpenAI blog (2019) 3

    Radford,A.,Wu,J.,Child,R.,Luan,D.,Amodei,D.,Sutskever,I.,etal.:Language models are unsupervised multitask learners. OpenAI blog (2019) 3

    2019

  42. [50]

    In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining

    Rasley, J., Rajbhandari, S., Ruwase, O., He, Y.: Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 3505–3506 (2020) 25

    2020

  43. [51]

    International Journal of Computer Vision (IJCV)115(3), 211–252 (2015).https://doi.org/10.1007/s11263-015-0816-y 25

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV)115(3), 211–252 (2015).https://do...

    2015 doi

  44. [52]

    In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision

    Shtedritski, A., Rupprecht, C., Vedaldi, A.: What does clip know about a red circle? visual prompt engineering for vlms. In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision. pp. 11987–11997 (2023) 3, 4, 8, 12

    2023

  45. [53]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Singh, A., Natarjan, V., Shah, M., Jiang, Y., Chen, X., Parikh, D., Rohrbach, M.: Towards VQA models that can read. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8317–8326 (2019) 8, 23

    2019

  46. [54]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024

    Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., Wang, J.: Alpha-clip: A CLIP model focusing on wherever you want. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp. 13019–13029 (2...

    2024

  47. [55]

    Team, C.: Cvbench: A benchmark for cross-video multimodal reasoning (2025), https://huggingface.co/datasets/Dongyh35/CVBench24

    2025

  48. [56]

    arXiv:2312.11805 (2023) 1

    Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalk- wyk, J., Dai,A.M.,Hauth, A., et al.:Gemini: a family ofhighlycapable multimodal models. arXiv:2312.11805 (2023) 1

    2023 arXiv

  49. [57]

    arXiv preprint arXiv:2403.05530 (2024) 1, 13

    Team, G., Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vin- cent,D.,Pan,Z.,Wang,S.,etal.:Gemini1.5:Unlockingmultimodalunderstanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024) 1, 13

    2024 arXiv

  50. [58]

    Advances in Neural Information Processing Systems (2024) 8

    Tong, P., Brown, E., Wu, P., Woo, S., IYER, A.J.V., Akula, S.C., Yang, S., Yang, J., Middepogu, M., Wang, Z., et al.: Cambrian-1: A fully open, vision-centric ex- ploration of multimodal llms. Advances in Neural Information Processing Systems (2024) 8

    2024

  51. [59]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 8, 24

    Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., Xie, S.: Eyes wide shut? exploring the visual shortcomings of multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 8, 24

    2024

  52. [60]

    arXiv:2302.13971 (2023) 1, 3

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv:2302.13971 (2023) 1, 3

    2023 arXiv

  53. [61]

    arXiv preprint arXiv:2307.09288 (2023) 8

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023) 8

    2023 arXiv

  54. [62]

    Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: Consensus-based image description evaluation (2015),https://arxiv.org/abs/1411.572624

    2015

  55. [63]

    Advances in neural information processing systems35, 24824–24837 (2022) 1 20 Zhang et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, 24824–24837 (2022) 1 20 Zhang et al

    2022

  56. [64]

    arXiv preprint arXiv:2407.20454 (2024) 3

    Wu, J., Li, X., Yu, T., Wang, Y., Chen, X., Gu, J., Yao, L., Shang, J., McAuley, J.: Commit: Coordinated instruction tuning for multimodal large language models. arXiv preprint arXiv:2407.20454 (2024) 3

    2024 arXiv

  57. [65]

    arXiv preprint arXiv:2409.15310 (2024) 1, 3

    Wu, J., Zhang, Z., Xia, Y., Li, X., Xia, Z., Chang, A., Yu, T., Kim, S., Rossi, R.A., Zhang, R., et al.: Visual prompting in multimodal large language models: A survey. arXiv preprint arXiv:2409.15310 (2024) 1, 3

    2024 arXiv

  58. [66]

    co / datasets / xai - org / RealworldQA[] 8

    xAI: Realworldqa (2022),https : / / huggingface . co / datasets / xai - org / RealworldQA[] 8

    2022

  59. [67]

    X.AI: Grok-1.5 vision preview.https://x.ai/news/grok-1.5v(2024) 24

    2024

  60. [68]

    In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition

    Xiao, J., Yao, A., Li, Y., Chua, T.S.: Can i trust your answer? visually grounded video question answering. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. pp. 13204–13214 (2024) 6

    2024

  61. [69]

    arXiv preprint arXiv:2502.13146 (2025) 4

    Xing, S., Li, P., Wang, Y., Bai, R., Wang, Y., Hu, C.w., Qian, C., Yao, H., Tu, Z.: Re-align:Aligningvisionlanguagemodelsviaretrieval-augmenteddirectpreference optimization. arXiv preprint arXiv:2502.13146 (2025) 4

    2025

  62. [70]

    Advances in Neural Information Processing Systems36, 24993–25006 (2023) 3, 4, 8, 12

    Yang, L., Wang, Y., Li, X., Wang, X., Yang, J.: Fine-grained visual prompting. Advances in Neural Information Processing Systems36, 24993–25006 (2023) 3, 4, 8, 12

    2023

  63. [71]

    In: European conference on computer vision (2016) 8

    Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: European conference on computer vision (2016) 8

    2016

  64. [72]

    In: European Conference on Computer Vision

    Yu, R., Yu, W., Wang, X.: Attention prompting on image for large vision-language models. In: European Conference on Computer Vision. pp. 251–268. Springer (2024) 3, 4, 8, 10, 12, 22, 23

    2024

  65. [73]

    arXiv preprint arXiv:2410.03226 (2024) 6

    Yu, S., Jin, C., Wang, H., Chen, Z., Jin, S., Zuo, Z., Xu, X., Sun, Z., Zhang, B., Wu, J., et al.: Frame-voyager: Learning to query frames for video large language models. arXiv preprint arXiv:2410.03226 (2024) 6

    2024 arXiv

  66. [74]

    arXiv preprint arXiv:2308.02490 (2023) 8, 23

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: Mm- vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023) 8, 23

    2023 arXiv

  67. [75]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 8, 23

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al.: Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  68. [76]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975–11986 (2023) 8, 22

    2023

  69. [77]

    Advances in Neural Information Processing Systems36, 20299–20319 (2023) 1, 3

    Zhang, A., Fei, H., Yao, Y., Ji, W., Li, L., Liu, Z., Chua, T.S.: Vpgtrans: Transfer visual prompt generator across llms. Advances in Neural Information Processing Systems36, 20299–20319 (2023) 1, 3

    2023

  70. [78]

    In: Proceedings of the 27th ACM International Conference on Multimedia

    Zhang, C., Li, W., Ouyang, W., Wang, Q., Kim, W.S., Hong, S.: Referring ex- pression comprehension with semantic visual relationship and word mapping. In: Proceedings of the 27th ACM International Conference on Multimedia. pp. 1258– 1266 (2019) 4

    2019

  71. [79]

    ICLR (2025) 6, 12

    Zhang, S., Fang, Q., Yang, Z., Feng, Y.: Llava-mini: Efficient image and video large multimodal models with one vision token. ICLR (2025) 6, 12

    2025

  72. [80]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 3, 12 AutoV 21

    Zhang, Y., Dong, Y., Zhang, S., Min, T., Su, H., Zhu, J.: Exploring the transfer- ability of visual prompting for multimodal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024) 3, 12 AutoV 21

    2024

  73. [81]

    Advances in Neural Information Processing Systems37, 118632–118653 (2024) 6

    Zhang, Y., Huang, T., Fan, C.K., Dong, H., Li, J., Wang, J., Cheng, K., Zhang, S., Guo, H., et al.: Unveiling the tapestry of consistency in large vision-language models. Advances in Neural Information Processing Systems37, 118632–118653 (2024) 6

    2024

  74. [82]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022) 1

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision- language models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022) 1

    2022

  75. [83]

    CLIP-blind

    Zong, Z., Ma, B., Shen, D., Song, G., Shao, H., Jiang, D., Li, H., Liu, Y.: Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.13046 (2024) 4 22 Zhang et al. Supplementary Material A Details of Training Data A.1 Attention Visual Prompts f...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.