Pith. sign in

REVIEW 3 major objections 4 minor 53 references

Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ViMaR claims that a two-stage search guided by a temporal-difference value model with a margin-based CLIP penalty produces more faithful, more detailed VLM captions at over four times lower inference cost than the VisVM baseline.

desk verdict A genuinely useful two-stage search idea, but the paper's central margin reward equation is written backwards, and the main caption-quality evaluation overlaps the training split. read the letter →

arxiv 2506.15649 v1 pith:CYHI64IT submitted 2025-06-18 cs.CV cs.LG

Ankan Deria , Adinath Madhavrao Dukre , Feilong Tang , Sara Atito , Sudipta Roy , Muhammad Awais , Muhammad Haris Khan , Imran Razzak This is my paper
classification cs.CVcs.LG
keywords vision-languagemodelsinference-timesearchvaluemodeltemporal-differencelearningmargin-basedrewardhallucinationmitigationimagecaptioningself-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that VLM hallucination during descriptive captioning can be reduced without slowing generation by replacing exhaustive stepwise rescoring with a two-stage search: first pick the best whole caption from a sampled batch, then re-generate only the weakly grounded sentences. The decisive addition is a margin-based reward adjustment in value-model training, where candidates whose CLIP similarity falls below a calibrated threshold are supposed to be penalized proportionally to the gap, steering the value model away from low-confidence continuations. If correct, the approach would make inference-time value guidance practical: roughly four times faster than the existing VisVM search while improving CHAIR and MMHal hallucination scores, and it would transfer to a stronger unseen VLM without retraining. The paper also claims that captions produced this way serve as supervision that improves the base model by 15.87% on average across eight visual benchmarks.

What carries the argument

The central object is the value model $V_\rho$, a scalar head attached to the penultimate transformer layer of LLaVA-Next-Mistral-7B and trained by temporal-difference learning on (current sentence, next sentence, image) triplets. Its reward signal is the margin-adjusted CLIP similarity $r_{s_i} = \delta$ if $\delta \geq \tau$ and $\tau - \delta$ otherwise, with $\tau = 0.16$ calibrated to the low end of the CLIP score distribution; the paper's stated intent is that weakly grounded, low-CLIP candidates receive a penalty proportional to how far they fall below the margin. The search machinery is the two-stage procedure: a holistic best-of pass over $N \times K$ temperature-sampled full captions, followed by targeted resampling and rescoring of only under-grounded segments, which removes the per-step full-candidate rescoring that makes VisVM slow.

What would settle it

Take a held-out COCO set with ground-truth object annotations, compute CLIP-ViT similarity for each candidate sentence, and check whether sentences with delta below tau are more likely to contain hallucinated objects; if the correlation is absent or reversed, the margin mechanism cannot be the cause of the reported hallucination gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that ViMaR, a two-stage inference-time decoding framework built on a temporal-difference value model, generates captions that are more reliable, factually accurate, detailed, and explanatory than prior value-guided search while running substantially faster. Trained on sentence-level transitions from COCO and LLaVA-150K data with CLIP-ViT similarity as the process reward, the value model scores whole captions and candidate sentences; at inference, Stage 1 samples paragraph candidates over multiple temperatures, scores them holistically, and selects the highest-value caption, and Stage 2 resamples only segments with weak grounding. The paper reports hallucination improvements over VisVM-guided search (CHAIR_S 23.1 vs 26.2, MMHal rate 0.35 vs 0.39) at 108 seconds per sample versus 462 seconds, and it demonstrates that a value model trained only on LLaVA-Mistral-7B can guide a stronger unseen model, LLaVA-OneVision-Qwen2-7B, to consistent gains. It further claims that ViMaR-generated captions, when used as supervised fine-tuning data, lift the base LLaVA-Next-7B model by an average of 15.87% across eight visual comprehension and hallucination benchmarks.

Load-bearing premise

The load-bearing premise is that a sentence's CLIP-ViT similarity to the image is a valid and monotonic proxy for visual grounding, so steering the value model away from low-similarity continuations suppresses hallucinations rather than punishing descriptive detail or rewarding generic text.

Editorial extensions

If this is right

  • ViMaR-guided decoding cuts hallucination on the COCO Val2017 evaluation: CHAIR_S drops to 23.1 from VisVM's 26.2 and the MMHal rate to 0.35 from 0.39, while average per-sample time falls from 462 seconds to 108 seconds.
  • A value model trained solely on LLaVA-Mistral-7B data can be attached to a stronger unseen VLM, LLaVA-OneVision-Qwen2-7B, and still improve every reported benchmark.
  • Captions generated by ViMaR can be used as supervised fine-tuning data, yielding a 15.87% average improvement over the base LLaVA-Next-7B across eight visual comprehension and hallucination benchmarks.
  • A single holistic best-of pass plus targeted refinement replaces stepwise rescoring of every candidate, so the quality gains do not require the O(N x S) inference overhead of VisVM-style search.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate the margin penalty from the two-stage search, so the reported gains should be read as the joint effect of the value model, holistic reranking, and targeted refinement, not as proof that the margin term alone is responsible.
  • Equation (1) as printed assigns a positive reward tau - delta to low-confidence sentences, which is the opposite of a penalty; if the implementation matches the printed formula, the value model would be trained to prefer low-CLIP candidates and the stated mechanism would need reinterpretation.
  • The threshold tau = 0.16 is calibrated to the CLIP score distribution of COCO and LLaVA-150K; applying ViMaR to datasets with different CLIP score scales would likely require recalibrating tau, so cross-model generalization should be tested for sensitivity to this choice.
  • The self-training result suggests a closed loop in which ViMaR-generated captions improve the base VLM, plausibly enabling another round of value-model training and search on the improved model; testing whether multiple rounds compound would clarify the scalability claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes ViMaR, a two-stage inference-time value-guided decoding strategy for vision-language model captioning. In the first stage, a temporal-difference value model scores N×K full-caption candidates generated at different temperatures and selects the highest-value caption; in the second stage, only under-grounded or missing segments are resampled and rescored, which the authors claim yields over 4× speedup over the VisVM baseline. The value model is trained on a CLIP-similarity reward with a margin-based adjustment intended to penalize low-confidence continuations. The paper reports reduced hallucination (CHAIR, MMHal) and improved caption quality by human and GPT-4o judgments, plus self-training gains on eight visual-comprehension benchmarks and cross-model generalization to LLaVA-OneVision-Qwen2-7B.

Significance. If the mechanism worked as described, the paper would make a useful contribution: it offers an explicit two-stage inference procedure, reports human and GPT-4o preference studies, evaluates hallucination with standard metrics, demonstrates cross-model transfer of the value model, and shows that ViMaR-generated captions can be used for self-training. These are concrete, falsifiable empirical claims. However, the central novelty of the paper, the margin-based reward adjustment, is mis-specified as printed in Eq. (1), and this error propagates through the TD training objective and both stages of inference. Because the paper does not release code or checkpoints, the reader cannot determine whether the implemented reward matches the intended negative penalty or the printed positive one. The significance of the empirical results is therefore conditional on a corrected and verifiable specification.

major comments (3)
  1. [Section 3.1, Eq. (1); Appendix D] The reward defined in Eq. (1) is r_si = δ when δ ≥ τ, and r_si = τ − δ otherwise. For any δ < τ, this second branch is positive and grows as the CLIP similarity δ decreases; for example, with τ = 0.16, δ = 0.10 gives +0.06 while δ = 0.14 gives +0.02. This is the opposite of the 'negative penalty proportional to the margin gap' described in Section 3.1 and Appendix D. Since r_si enters the TD target in Eq. (2), the value model Vρ is trained to assign higher long-term value to low-confidence, weakly grounded states, and both stages of Algorithm 1 select candidates by argmax Vρ. As printed, the proposed margin mechanism would prefer hallucinated or under-grounded continuations, contradicting the paper's central claim. Please correct Eq. (1) (the intended second branch appears to be δ − τ), update all corresponding text in Appendix D, retrain and rerun the experiments, and release code so the implemented sign can be verified.
  2. [Section 3.2, Algorithm 1] Stage 2 is not formally specified. The text says to 'identify under-grounded or missing visual regions' and to refine 'segments that lack sufficient visual grounding', but no concrete criterion is given for deciding which segment yi is under-grounded, how many refinement rounds are performed, or when the loop in Algorithm 1 terminates ('while Generation is not Done', 'until all salient content is addressed'). The efficiency claim of 108 s per sample and the comparison with VisVM depend critically on this stopping rule. Please provide a precise, reproducible definition of the under-grounding test and the termination condition, or report the empirical distribution of refinement rounds.
  3. [Table 2, cross-model generalization] The LLaVA-OneVision-Qwen row reports ViMaR hallucination results (CHAIR_s 15.3, CHAIR_i 3.0, MMHal 3.96, MMHal rate 0.34) but the corresponding 'Original' row has dashes for all hallucination metrics. As a result, the claim that ViMaR reduces hallucinations in the stronger unseen model is not supported by the table. Please provide the baseline hallucination measurements for LLaVA-OneVision-Qwen under greedy decoding or the same inference scheme, or restrict the cross-model claim to the visual-comprehension benchmarks where baselines are present.
minor comments (4)
  1. [Section 4.1, Table 1 caption and text] The text twice writes 'CLP-PRM' instead of 'CLIP-PRM'.
  2. [Section 4.1, paragraph on fairness] The sentence 'In all experiments, we kept the total decode calls identical to ensure a fair comparison of quality versus compute' is unclear: ViMaR uses fewer value-model scoring calls by design, so 'total decode calls' cannot be identical across methods. Please clarify which quantity was controlled (e.g., number of sampled sentences, number of CLIP evaluations, or wall-clock budget).
  3. [Section 5 and Figure 2] The text describes local and global value scores (e.g., 2.285, 2.342, 2.254) but does not state the units or the discount factor γ used to compute them; adding this detail would make the case study reproducible.
  4. [Appendix C] The training details say the model was fine-tuned 'on the LLAVA dataset using the provided train and test splits', but the main text says training samples come from COCO 2017 with LLaVA-150K prompts. Please state which dataset and splits were actually used for the value-model training and for self-training.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ViMaR is an empirical systems paper whose central claims are evaluated against external hallucination and comprehension benchmarks, not against its own training reward.

full rationale

ViMaR's derivation chain is a standard TD value-model training loop: a CLIP-similarity reward (Section 3.1, Eq. 1) is used to fit V_rho via Eq. (2), and at inference the model selects and refines candidates by maximizing V_rho (Algorithm 1). This is not circular because the paper's headline claims are measured against external benchmarks: lower CHAIR/MMHal hallucination rates, higher MM-Vet/MMBench/MMMU/MathVista/CVBench scores, and 4x speedup (Tables 1 and 2; Sections 4.1-4.3). These evaluation signals are not the training reward. No fitted parameter is renamed as a prediction: the margin threshold tau is calibrated from the CLIP-score distribution in Appendix D as a design choice, and the reported gains are empirical rather than derived from that calibration. No load-bearing self-citation appears: the comparison method VisVM [44] is prior work by a different author group, and the LLaVA checkpoints are external base models. The sign inconsistency in Eq. (1), where tau-delta is positive when delta < tau and therefore is not the 'negative penalty' described in the text, is a correctness/specification concern rather than a circularity: it does not make any claimed result equal to its inputs by construction. The self-training experiment in Section 4.2 is also evaluated on external benchmarks, so the 'self-improvement' claim is not definitionally forced. Accordingly, no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on: one fitted threshold tau, an unreported gamma, and manually chosen inference budgets; a domain assumption that CLIP similarity measures factual grounding; standard TD assumptions; and an unstated resolution of the Eq. (1) sign mismatch. No code or data is provided to test these assumptions independently.

free parameters (4)
  • margin threshold tau = 0.16
    Chosen from the empirical CLIP-similarity distribution on the training set (approximate 17th percentile), Appendix D; defines the margin penalty in Eq. (1).
  • discount factor gamma = not reported
    Required by the TD objective in Eq. (2), but its value is never given in the paper; presumably inherited from VisVM without ablation.
  • number of temperatures N and candidates K = N=5, K=6
    Hand-selected inference budget; used in both stages of Algorithm 1 and compared across baselines in Section 4.1.
  • fine-tuning hyperparameters = LR 1e-6, 4 epochs, batch 16, fp16
    Used for the self-training experiment in Section 4.2; no sensitivity analysis is provided.
assumptions (4)
  • domain assumption CLIP-ViT similarity is a valid proxy for visual grounding and hallucination tendency
    The entire training reward and inference scoring rely on delta from CLIP-ViT (Section 3.1); no evidence is given that CLIP score correlates with CHAIR or MMHal.
  • standard math Temporal-difference learning converges to the long-term value of sentence-level reward
    Equations (1)-(2) apply TD updates on a non-tabular, function-approximated setting; convergence is assumed, standard practice.
  • domain assumption Sentence-level rewards are additive over the caption MDP
    The MDP formulation and TD target in Eq. (2) treat the discounted sum of sentence CLIP rewards as the caption quality; this ignores interactions between sentences beyond the immediate next-state value.
  • ad hoc to paper The printed Eq. (1) matches the implemented reward function
    The prose describes a negative penalty for delta < tau, but Eq. (1) computes tau - delta, which is positive; the paper gives no indication of a sign correction, so the claim depends on an unstated fix.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning." pith.science (2026). https://pith.science/paper/CYHI64IT

@misc{pith2026250615649,
  author       = {Pith},
  title        = {Pith review of: Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYHI64IT}},
  note         = {Machine review of arXiv:2506.15649}
}
abstract

Despite significant advances in inference-time search for vision-language models (VLMs), existing approaches remain both computationally expensive and prone to unpenalized, low-confidence generations which often lead to persistent hallucinations. We introduce \textbf{Value-guided Inference with Margin-based Reward (ViMaR)}, a two-stage inference framework that improves both efficiency and output fidelity by combining a temporal-difference value model with a margin-aware reward adjustment. In the first stage, we perform a single pass to identify the highest-value caption among diverse candidates. In the second stage, we selectively refine only those segments that were overlooked or exhibit weak visual grounding, thereby eliminating frequently rewarded evaluations. A calibrated margin-based penalty discourages low-confidence continuations while preserving descriptive richness. Extensive experiments across multiple VLM architectures demonstrate that ViMaR generates captions that are significantly more reliable, factually accurate, detailed, and explanatory, while achieving over 4$\times$ speedup compared to existing value-guided methods. Specifically, we show that ViMaR trained solely on LLaVA Mistral-7B, \textit{generalizes effectively to guide decoding in a stronger unseen model}. To further validate this, we adapt the ViMaR to steer generation in LLaVA-OneVision-Qwen2-7B, leading to consistent improvements in caption quality and demonstrating robust cross-model guidance. This cross-model generalization highlights ViMaR's flexibility and modularity, positioning it as a scalable and transferable inference-time decoding strategy. Furthermore, when ViMaR-generated captions are used for self-training, the underlying models achieve substantial gains across a broad suite of visual comprehension benchmarks, underscoring the potential of fast, accurate, and self-improving VLM pipelines.

Figures

Figures reproduced from arXiv: 2506.15649 by the authors.

Figure 1
Figure 1. Comparison of image-description quality across search strategies for LLaVA-Next-7B. (a) [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Qualitative comparison of decoding strategies. The top section shows how our value [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison example 1 15 [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison example 2 16 [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison example 3 17 [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Case study 1: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Case study 2: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Case study 3: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Case study 4: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Case study 5: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Case study 6: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Case study 7: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Case study 8: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Case study 9: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Case study 10: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: Case study 11: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: Case study 12: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: Case study 13: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: Case study 14: ViMaR and baseline captions. [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    arXiv 2023

  3. [3]

    Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930, 2024

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930, 2024

    arXiv 2024

  4. [4]

    Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-time

    Uri Berger, Omri Abend, Lea Frermann, and Gabriel Stanovsky. Improving image captioning by mimicking human reformulation feedback at inference-time.arXiv preprint arXiv:2501.04513, 2025

    work page Pith review arXiv 2025

  5. [5]

    Improving image generation with better captions.Computer Science

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  6. [6]

    Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726, 2024

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726, 2024

    arXiv 2024

  7. [7]

    Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    arXiv 2024

  8. [8]

    Transfer q-star: Principled decoding for llm alignment.Advances in Neural Information Processing Systems, 37:101725–101761, 2024

    Souradip Chakraborty, Soumya Suvra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Amrit Singh Bedi, and Furong Huang. Transfer q-star: Principled decoding for llm alignment.Advances in Neural Information Processing Systems, 37:101725–101761, 2024

    work page 2024

Show all 53 references
  1. [9]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision, pages 370–387. Springer, 2024

    2024

  2. [10]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    2024 arXiv

  3. [11]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer visi...

    2024

  4. [12]

    Mit- igating hallucination in visual language models with visual supervision.arXiv preprint arXiv:2311.16479, 2023

    Zhiyang Chen, Yousong Zhu, Yufei Zhan, Zhaowen Li, Chaoyang Zhao, Jinqiao Wang, and Ming Tang. Mit- igating hallucination in visual language models with visual supervision.arXiv preprint arXiv:2311.16479, 2023

    2023 arXiv

  5. [13]

    Partially non-autoregressive image captioning

    Zhengcong Fei. Partially non-autoregressive image captioning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1309–1316, 2021. 10

    2021

  6. [14]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProcee...

    2024

  7. [15]

    From images to textual prompts: Zero-shot visual question answering with frozen large language models

    Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. InProceedings of the IEEE/CVF conference on computer vision and pattern re...

    2023

  8. [16]

    A hierarchical approach for generating descriptive image paragraphs

    Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 317–325, 2017

    2017

  9. [17]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  10. [18]

    Coco-cn for cross-lingual image tagging, captioning, and retrieval.IEEE Transactions on Multimedia, 21(9):2347–2360, 2019

    Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, and Jieping Xu. Coco-cn for cross-lingual image tagging, captioning, and retrieval.IEEE Transactions on Multimedia, 21(9):2347–2360, 2019

    2019

  11. [19]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  12. [20]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  13. [21]

    Mitigating hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023

    2023 arXiv

  14. [22]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  15. [23]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  16. [24]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  17. [25]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    arXiv

  18. [26]

    Hallucination detection and hallucination mitigation: An investigation.arXiv preprint arXiv:2401.08358, 2024

    Junliang Luo, Tianyu Li, Di Wu, Michael Jenkin, Steve Liu, and Gregory Dudek. Hallucination detection and hallucination mitigation: An investigation.arXiv preprint arXiv:2401.08358, 2024

    2024 arXiv

  19. [27]

    Training for diversity in image paragraph captioning

    Luke Melas-Kyriazi, Alexander M Rush, and George Han. Training for diversity in image paragraph captioning. Inproceedings of the 2018 conference on empirical methods in natural language processing, pages 757–761, 2018

    2018

  20. [28]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/ learning-to-reason-with-llms/, 2024. Accessed: 2025-04-30

    2024

  21. [29]

    Self-critical sequence training for image captioning

    Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7008–7024, 2017

    2017

  22. [30]

    Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    2018 arXiv

  23. [31]

    Mastering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016. 11

    2016

  24. [32]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  25. [33]

    The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism.arXiv preprint arXiv:2407.10457, 2024

    Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism.arXiv preprint arXiv:2407.10457, 2024

    2024 arXiv

  26. [34]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  27. [35]

    Learning to predict by the methods of temporal differences.Machine learning, 3:9–44, 1988

    Richard S Sutton. Learning to predict by the methods of temporal differences.Machine learning, 3:9–44, 1988

    1988

  28. [36]

    Toward self-improvement of llms via imagination, searching, and criticizing.Advances in Neural Information Processing Systems, 37:52723–52748, 2024

    Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. Toward self-improvement of llms via imagination, searching, and criticizing.Advances in Neural Information Processing Systems, 37:52723–52748, 2024

    2024

  29. [37]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processin...

    2024

  30. [38]

    Solving math word problems with process-and outcome-based feedback

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022

    2022 arXiv

  31. [39]

    Faithfulness-aware decoding strategies for abstractive summarization.arXiv preprint arXiv:2303.03278, 2023

    David Wan, Mengwen Liu, Kathleen McKeown, Markus Dreyer, and Mohit Bansal. Faithfulness-aware decoding strategies for abstractive summarization.arXiv preprint arXiv:2303.03278, 2023

    2023 arXiv

  32. [40]

    Litesearch: Efficacious tree search for llm.arXiv preprint arXiv:2407.00320, 2024

    Ante Wang, Linfeng Song, Ye Tian, Baolin Peng, Dian Yu, Haitao Mi, Jinsong Su, and Dong Yu. Litesearch: Efficacious tree search for llm.arXiv preprint arXiv:2407.00320, 2024

    2024 arXiv

  33. [41]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  34. [42]

    Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Systems, 37:121475–121499, 2024

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Systems, 37:121475–121499, 2024

    2024

  35. [43]

    Mitigating hallucinations in large vision- language models with instruction contrastive decoding.arXiv preprint arXiv:2403.18715, 2024

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision- language models with instruction contrastive decoding.arXiv preprint arXiv:2403.18715, 2024

    2024 arXiv

  36. [44]

    Scaling inference-time search with vision value model for improved visual comprehension.arXiv preprint arXiv:2412.03704, 2024

    Xiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu, Yuancheng Xu, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Scaling inference-time search with vision value model for improved visual comprehension.arXiv preprint arXiv:2412.03704, 2024

    2024 arXiv

  37. [45]

    Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences.arXiv preprint arXiv:2401.10529, 2024

    Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences.arXiv preprint arXiv:2401.10529, 2024

    2024 arXiv

  38. [46]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  39. [47]

    Llava-critic: Learning to evaluate multimodal models.arXiv preprint arXiv:2410.02712, 2024

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models.arXiv preprint arXiv:2410.02712, 2024

    2024 arXiv

  40. [48]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    2024 arXiv

  41. [49]

    Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    2022 arXiv

  42. [50]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023. 12

    2023 arXiv

  43. [51]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  44. [52]

    Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    2024

  45. [53]

    Calibrated self-rewarding vision language models.arXiv preprint arXiv:2405.14622, 2024

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models.arXiv preprint arXiv:2405.14622, 2024. A Human Evaluation This section details the human evaluati...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.