Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LACING claims that language bias in large vision-language models can be reduced by a dual-attention mechanism plus a learned soft-image prompt, cutting object hallucination rates by about 40% and raising LLaVA-Bench scores by 12–16 points…

desk verdict A promising new combination of dual-attention and a trained soft null prompt for LVLM hallucination, but the printed attention mask is wrong and the hyperparameters are tuned on the test benchmarks, so the headline numbers cannot be taken at face value. read the letter →

arxiv 2411.14279 v2 pith:OLN65HND submitted 2024-11-21 cs.CV cs.CL

Haozhe Zhao , Shuzheng Si , Liang Chen , Yichi Zhang , Maosong Sun , Mingjia Zhang , Baobao Chang This is my paper · ORCID
classification cs.CVcs.CL
keywords largevision-languagemodelslanguagebiashallucinationreductionmultimodaldual-attentionsoft-imageguidancecontrastivedecodingvisualcomprehension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LACING argues that large vision-language models hallucinate and under-read images largely because their text-pretrained backbone dominates the smaller multimodal training stage, and because autoregressive text decoding leans on adjacent words more than on distant visual tokens. To counter the first cause, it inserts a multimodal dual-attention mechanism that computes separate attention weights for visual and text tokens and fuses them, so visual tokens get real attention in every layer while text keeps its causal mask. To counter the second cause, it trains a learnable soft visual prompt that, at inference, replaces the image to form a multimodal-null input; the model's output on this null input is treated as the text-only prior and contrasted with the normal output in a Bayesian decoding recipe. On LLaVA-1.5 backbones, the method reports a 40.36% relative drop in response-level object hallucination and a 12.11-point absolute gain on LLaVA-Bench for the 7B model, with analogous gains for 13B. If the result holds, it matters because it promises hallucination reduction and better visual comprehension without additional data, extra training stages, or RLHF-style feedback.

What carries the argument

Two mechanisms carry the argument. Multimodal Dual-Attention (MDA) computes two masked attention maps in parallel—a visual map from $M_I$ and a text map from $M_T$—and adds them before the value projection, creating a dedicated visual attention channel that survives into deeper layers. Soft-Image Guidance (SIG) introduces a learnable soft visual prompt $\epsilon$ of the same shape as the visual embedding; during training it replaces the real image with probability $\theta$, and at inference the logits from this multimodal-null input are combined with normal logits via $\ell_g = \ell_u + (\ell_c-\ell_u)\lambda$ to up-weight visual evidence. These are the mechanisms whose removal (ablation) or replacement (FastV pruning) is used to attribute the benchmark gains.

What would settle it

Replace the trained soft visual prompt with a randomly initialized embedding of the same shape at inference and re-run Object-HalBench and LLaVA-Bench; the paper's mechanism predicts the learned prompt is doing the work, so the hallucination reduction should largely disappear, whereas if random embeddings preserve the gains, the reported improvement comes from the contrastive decoding formula itself rather than from Soft-Image Guidance.

Watch

Extended reading notes

Core claim

The paper's central claim is that language bias in LVLMs has two separable causes and that each has a targeted fix: MDA counters training-scale dominance by computing separate attention weights for visual and text tokens and fusing them as $A=(W_I+W_T)V$, so visual tokens receive meaningful attention in every layer while text tokens keep causal attention; SIG counters short-term text dependency by training a learnable soft visual prompt $\epsilon$ that replaces the image with probability $\theta$, then at inference decoding from $\ell_g = \ell_u + (\ell_c-\ell_u)\lambda$, the pragmatic form of the Bayes-derived adjustment $\log \tilde{p}(y_t|v) \propto \lambda(\log p(y_t|v)-\log p(y_t))+\log p(y_t)$. On the LLaVA-1.5 backbone this combination is reported to reduce response-level object hallucination by 40.36% (7B) and 42.18% (13B) and to raise LLaVA-Bench by 12.11 and 16.28 absolute points, all without additional training data or RLHF-style feedback.

Load-bearing premise

The load-bearing premise is that the text-attention mask written in Eq. (1) is a typo: as printed it lets each text token attend to tokens that come after it, which would leak future text during training and invalidate the causal-attention claim on which the whole method rests; the experiments must implicitly use the reversed inequality with attention only to earlier tokens.

Editorial extensions

If this is right

  • Because LACING reuses the LLaVA-1.5 training recipe and adds only about 0.03% parameters, the two mechanisms could be folded into any existing visual-instruction-tuning pipeline without new data or an RLHF stage, making hallucination reduction a byproduct of standard alignment.
  • If the reported Object-HalBench and LLaVA-Bench gains are real, a large share of LVLM hallucination is caused by the text-pretrained backbone's dominance and by adjacent-token decoding bias rather than by a lack of vision capability, which reframes hallucination as a distributional artifact rather than an architectural limit.
  • The SIG contrastive formula only needs the null-input logits, so it could be applied at decoding time to already-trained models that never saw the soft prompt, although the paper's consistency argument predicts it will work better when the prompt was trained in.
  • The FastV-pruning result—a large performance drop when deep visual tokens are pruned after MDA training—implies that after such training, deep visual tokens carry information the model actually uses, in contrast to the standard picture where most deep-layer visual tokens are redundant.
  • The reported gains under both greedy and nucleus sampling, unlike the comparison methods VCD and VDD, suggest the method would also work in deterministic deployment settings where sampling noise cannot mask the bias.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct check the paper does not report is whether the soft prompt transfers across model sizes or model families: transferring the 7B prompt to the 13B model would test whether the prompt encodes generic 'ignore the image' statistics or model-specific text priors.
  • The decoding formula is essentially an interpolation between the null-input distribution and the normal distribution; an untested extension would be to apply the same contrast with an image caption or a text description as the null input, which would measure how much of the gain is specific to the learned prompt rather than to any stable reference distribution.
  • Because several headline benchmarks are scored by GPT-4, part of the reported improvement may reflect what GPT-4 prefers in an answer; the paper's 60-item human evaluation is reassuring but small, so a larger human study would be needed to confirm the gains are content-level rather than stylistic.
  • The method's assumption about where visual tokens sit in the input sequence suggests it may generalize naturally to interleaved image-text inputs, but the paper only demonstrates it on the standard image-first setup.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LACING, a framework to reduce language bias in large vision-language models. It introduces Multimodal Dual-Attention (MDA), which computes separate attention weights for visual and text tokens and combines them, and Soft-Image Guidance (SIG), which trains a learnable soft visual prompt and uses it at inference to contrast the output distributions of original and multimodal-null inputs. The method is evaluated on LLaVA-1.5 7B/13B across object hallucination, MMHal-Bench, LLaVA-Bench, MMBench, TextVQA, and MM-VET, reporting large gains such as a 40.36% relative reduction in Object HalBench response-level hallucination and a 12.11% absolute gain on LLaVA-Bench for the 7B model.

Significance. If the reported results are trustworthy, the framework is significant because it addresses language bias from both training and inference perspectives with only a small number of additional parameters and no extra training data. The paper also contains a self-contained Bayesian derivation for the contrastive decoding update and a useful attention-allocation analysis. However, the current manuscript does not support the reported numbers: the attention-mask equations are internally inconsistent with the claimed causal mechanism, and the main hyperparameters are selected on the same benchmarks used for the final results. These issues must be resolved before the significance of the contribution can be assessed.

major comments (4)
  1. [§3.1, Eq. (1)] The text-attention mask is defined as M_T[i,j] = 1 when s_j is a text token and i <= j, which is anti-causal: a query at position i may attend to text positions j > i. This contradicts the paper's repeated statement that MDA retains causal attention for text inputs. If the formula is interpreted literally, an autoregressive token at step i has no earlier text keys available and can attend only to itself in the text branch, which is incompatible with coherent generation and with the reported benchmark results. The experiments must therefore be using a different mask, presumably j <= i, but this correction is never stated.
  2. [§3.1, Eq. (2)] Equation (2) defines W_I and W_T as softmax(QK^T/sqrt(d_k) * M) with M in {0,1}. A binary mask multiplied elementwise before the softmax does not actually mask logits: masked positions contribute exp(0)=1 to both numerator and denominator. If the multiplication is instead applied after softmax, the attention rows are no longer normalized. The paper does not specify the actual implementation, such as additive negative-infinity masking, so the architecture whose results are reported cannot be recovered from the equations as written.
  3. [§4.1 and Appendix D] The scaling parameter lambda and replacement probability theta are said to be determined on the validation set (§4.1), but the reported sweeps in Figures 5, 6, and 7 and in Table 8 are performed on the same benchmarks whose final scores appear in the main tables: LLaVA-Bench, MMHal-Bench, and MMBench. The headline improvements are therefore partly fitted to the evaluation benchmarks. An unbiased assessment requires either a separate validation split or a single pre-specified configuration.
  4. [§3.2, Eq. (10)] The contrastive decoding formula rests on identifying the soft-prompt logits l_u with the unconditional log-probability log p(y_t). The paper does not provide any analysis showing that the distribution produced by the learned soft visual prompt matches the text-only unconditional distribution. Without such evidence, the Bayes-derived adjustment in Eq. (9) is an asserted approximation, and the effectiveness of SIG depends on an unvalidated quantity.
minor comments (5)
  1. [Abstract and §3.2] The abstract uses the acronym IFG, while the method is consistently named Soft-Image Guidance (SIG) in the body; please use one acronym throughout.
  2. [§4.3 and References] Section 4.3 refers to 'Table 2' for the main comparison of baselines, but the corresponding table is labeled Table 1; the text also cites 'VDD [2]' where the correct reference is [66].
  3. [Table 3] The MDA + FastV row reports scores around 10.5 on LLaVA-Bench with deltas of about −50 to −72; these values look implausible and should be checked for unit errors or mislabeling.
  4. [Figures 5–7] The x-axis label 'N/A' is unclear; it should be replaced with the actual baseline configuration, for example lambda = 1.
  5. [Appendix E] The human evaluation reports only majority voting without inter-annotator agreement; please report agreement statistics or at least the per-annotator breakdown.

Circularity Check

1 steps flagged · score 6.0 of 10

LACING's headline benchmark gains are partly selected maxima: λ and θ are tuned on the same benchmarks and the optimal values are then reported as the method's performance; the MDA component retains independent support.

  1. fitted input called prediction [Section 4.1 Implementation Details; Appendix B.3 and D.1/D.2; Eq. (10)]
    "The scaling parameter λ and replacement probability θ are determined based on the model’s performance on the validation set. ... The optimal performance of our method under various λ values is reported in the experiments."

    Eq. (10), ℓg = ℓu + (ℓc − ℓu) × λ, is a one-parameter interpolation: λ = 1 returns the baseline logits ℓc and λ = 0 returns the soft-prompt logits ℓu. The headline benchmark improvements (e.g., 40.36% Obj HalBench reduction and 12.11% LLaVA-Bench gain for the 7B model) are obtained after selecting λ and θ to maximize scores on those same benchmarks, as shown in Figures 5–7 and Table 8. The paper explicitly states that the reported numbers are the optimal performance over the searched λ values, so the reported 'prediction' is the maximum of a fit to the target benchmark rather than an out-of-sample evaluation of a fixed method. The improvement is therefore partly manufactured by the hyperparameter search instead of being derived from the debiasing mechanism itself.

full rationale

The only substantive circularity is the hyperparameter selection for SIG. The decoding rule is a linear interpolation whose strength λ is chosen per benchmark on the validation set, and the appendix confirms that the reported numbers are the optimal ones found during the search; consequently the headline gains are partly fitted quantities. This does not make the whole paper circular: the MDA component is supported by the FastV pruning experiment (Table 3), where pruning visual tokens in deeper layers causes a large performance drop, an independent and falsifiable diagnostic even though FastV is the authors' prior work; the Bayes derivation in Eqs. (4)–(9) is standard algebra; and the soft-prompt training is a well-defined auxiliary objective. The anti-causal-looking mask in Eq. (1) with i ≤ j and the multiplicative mask in Eq. (2) are serious correctness or specification issues, but they are not circularity because they do not make the derivation equivalent to its inputs. The score of 6 reflects that one of the two headline empirical claims—the benchmark deltas—is partially a fitted maximum rather than a genuine prediction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The central claims rest on two tuned hyperparameters (lambda, theta) and on several unstated assumptions: that text attention is actually causal in MDA, that the soft-prompt logits equal the unconditional distribution, and that raw logits can be subtracted without normalization. The paper also introduces a learnable soft visual prompt, a new trainable component with no external falsifiable handle.

free parameters (2)
  • lambda (scaling parameter) = 1.5 to 2.0 (optimal), tested 1.25-2.50
    Controls strength of contrastive decoding in Eq. 10; tuned directly on MMBench, LLaVA-Bench and MMHal-Bench (Figs. 5-7), so reported gains are partially fitted to the test benchmarks.
  • theta (replacement probability) = 10%
    Probability of replacing visual input with soft prompt during training; selected from {5,10,15,20} on MMBench and LLaVA-Bench (Table 8).
assumptions (3)
  • domain assumption Text attention must remain causal for autoregressive generation (j <= i in M_T)
    Eq. (1) prints i <= j, which is anti-causal; the paper claims causal text attention is retained, so the plot must rely on the corrected inequality, an unstated fix.
  • ad hoc to paper The soft-prompt logits l_u approximate the unconditional log-probability log p(y_t) in Eq. (9)
    Section 3.2 uses l_u as p(y_t) without justifying that a trained placeholder embedding yields the true unconditional distribution.
  • domain assumption The partition function of l_u and l_c is shared, so subtracting raw logits in Eq. (10) is valid
    Standard contrastive decoding requires normalized probabilities; the paper omits normalization.
invented entities (1)
  • Learnable soft visual prompt epsilon
    purpose: Placeholder to replace visual tokens so the model produces an 'unconditional' distribution p(y_t) while preserving input length and modality pattern.
    Introduced in Sec. 3.2 and trained with the LVLM; no external falsifiable prediction beyond the benchmark gains it is tuned for.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance." pith.science (2026). https://pith.science/paper/OLN65HND

@misc{pith2026241114279,
  author       = {Pith},
  title        = {Pith review of: Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OLN65HND}},
  note         = {Machine review of arXiv:2411.14279}
}
read the original abstract

Large vision-language models (LVLMs) have achieved impressive results in various vision-language tasks. However, despite showing promising performance, LVLMs suffer from hallucinations caused by language bias, leading to diminished focus on images and ineffective visual comprehension. We identify two primary reasons for this bias: 1. Different scales of training data between the pretraining stage of LLM and multimodal alignment stage. 2. The learned inference bias due to short-term dependency of text data. Therefore, we propose LACING, a systemic framework designed to address the language bias of LVLMs with muLtimodal duAl-attention meChanIsm (MDA) aNd soft-image Guidance (IFG). Specifically, MDA introduces a parallel dual-attention mechanism that enhances the integration of visual inputs across the model. IFG introduces a learnable soft visual prompt during training and inference to replace visual inputs, designed to compel LVLMs to prioritize text inputs. Then, IFG further proposes a novel decoding strategy using the soft visual prompt to mitigate the model's over-reliance on adjacent text inputs. Comprehensive experiments demonstrate that our method effectively debiases LVLMs from their language bias, enhancing visual comprehension and reducing hallucinations without requiring additional training resources or data. The code and model are available at [lacing-lvlm.github.io](https://lacing-lvlm.github.io).

Figures

Figures reproduced from arXiv: 2411.14279 by the authors.

Figure 1
Figure 1. Overview of LACING, a systemic framework, which consists of Multimodal Dual Attention (bottom) and Soft-Image Guid￾ance (above) to mitigate language bias of LVLMs. MDA proposes a parallel dual-attention mechanism that constructs two separate attention for visual and text inputs during both training and inference. SIG implements a learnable soft visual prompt during training to replace visual inputs. This soft prompt… view at source ↗
Figure 3
Figure 3. Comparison of attention allocation between a standard [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Attention allocation of LVLMs to visual and text to [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Model performance on LLaVABench across various [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Model performance on MMHal-Bench across various [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 8
Figure 8. Figure 8: Human evaluation on LLaVABench. following principles to make the decision: Principles of Human Evaluation for LLaVABench You are asked to evaluate the responses generated by different models. Your evaluation should adhere to the following principles: 1.Correctness: Ass…
Figure 9
Figure 9. Figure 9: Comparison of LACING with other baselines on a sam [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Comparison of LACING with other baselines on a sam [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Comparison of LACING with other baselines on a sample from LLaVABench. LACING effectively recognizes the individual [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Comparison of LACING with other baselines on a sample from LLaVABench. LACING is the only model capable of success [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Comparison of Attention Maps across the 1st to 16th Layer in LLaVA and LACING. [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Comparison of Attention Maps across the 17th to 32nd Layer in LLaVA and LACING. [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplification

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Adaptively scaling visual attention boosting per token, guided by a combined entropy-and-visual-grounding risk score, reduces hallucinations in LVLMs more than fixed boosting.

  2. Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation

    cs.CV 2025-10 conditional novelty 6.0 of 10

    Tracking positive shifts in visual attention over information-rich query words yields a saliency map that, when used to boost visual and query attention during decoding, reduces object hallucination on CHAIR, POPE, an...

  3. Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Modality bias, an imbalanced attention to text or image during hallucinated outputs, is shown to be mitigated by a training-free attention intervention plus contrastive decoding.

  4. SafeCoT: Improving VLM Safety with Minimal Reasoning

    cs.AI 2025-06 conditional novelty 5.0 of 10

    Training vision-language models to emit a short rule-based reasoning chain before refusing improves the safety-usefulness balance, with reported gains even at 100 training samples.

  5. MLLMs are Deeply Affected by Modality Bias

    cs.AI 2025-05 conditional novelty 4.0 of 10

    A position paper with a case study showing that multimodal LLMs rely on language priors and underuse visual input, together with a research roadmap and calls for balanced training.

Reference graph

Works this paper leans on

72 extracted references · 51 canonical work pages · cited by 5 Pith papers

  1. [2]

    Tran, and Mostafa De- hghani

    Ibrahim Alabdulmohsin, Vinh Q. Tran, and Mostafa De- hghani. Fractal patterns may illuminate the success of next- token prediction, 2024. 1, 7

    work page 2024

  2. [66]

    Debi- asing multimodal large language models, 2024

    Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Debi- asing multimodal large language models, 2024. 1, 2, 3, 4, 5, 6, 7, 14

    work page 2024

  3. [1]

    Lawrence Zitnick, Dhruv Batra, and Devi Parikh

    Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering, 2016. 6

    work page 2016

  4. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023. 1, 6, 12, 13

    work page 2023

  5. [4]

    Attend first, consolidate later: On the importance of attention in different llm layers,

    Amit Ben-Artzy and Roy Schwartz. Attend first, consolidate later: On the importance of attention in different llm layers,

    work page

  6. [5]

    Are we on the right way for evaluating large vision-language models?, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models?, 2024. 1, 3, 12

    work page 2024

  7. [6]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024. 1, 2, 3, 4, 5, 8, 12

    work page 2024

  8. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023. 1

    arXiv 2023

Show all 72 references
  1. [8]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 12

    2023

  2. [9]

    Frustratingly short attention spans in neural language modeling, 2017

    Michał Daniluk, Tim Rockt ¨aschel, Johannes Welbl, and Se- bastian Riedel. Frustratingly short attention spans in neural language modeling, 2017. 1

    2017

  3. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  4. [11]

    Mini-internvl: A flexible-transfer pocket multimodal model with 5% parameters and 90% perfor- mance

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: A flexible-transfer pocket multimodal model with 5% parameters and 90% perfor- mance. arXiv preprint arXiv:2410.16261, 2024. 3

    arXiv

  5. [12]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  6. [13]

    Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models. In P...

    2024

  7. [14]

    The curious case of neural text degeneration, 2020

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2020. 7

    2020

  8. [15]

    A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions, 2023

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions, 2023. 3

    2023

  9. [16]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation, 2024

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation, 2024. 6, 14

    2024

  10. [17]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 12

    2019

  11. [18]

    Exploring concept depth: How large language mod- els acquire knowledge at different layers?, 2024

    Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. Exploring concept depth: How large language mod- els acquire knowledge at different layers?, 2024. 8

    2024

  12. [19]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 con- ference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 12

    2014

  13. [20]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of compute...

    2017

  14. [21]

    A survey of hallucination in large visual language models, 2024

    Wei Lan, Wenyi Chen, Qingfeng Chen, Shirui Pan, Huiyu Zhou, and Yi Pan. A survey of hallucination in large visual language models, 2024. 1, 3

    2024

  15. [22]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding, 2023

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding, 2023. 2, 3, 5, 6, 7, 14

    2023

  16. [23]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 3

    2024 arXiv

  17. [24]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day, 2023

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day, 2023. 1

    2023

  18. [25]

    Evaluating object hallucination in large vision- language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 292–305, Singapore, 2023. Association for ...

    2023

  19. [26]

    Mini-gemini: Mining the potential of multi-modality vision language models, 2024

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models, 2024. 1 9

    2024

  20. [27]

    Lawrence Zitnick, and Piotr Doll ´ar

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll ´ar. Microsoft coco: Common objects in context, 2015. 6, 13

    2015

  21. [28]

    Mitigating hallucination in large multi-modal models via robust instruction tuning, 2024

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning, 2024. 3, 6, 13, 14

    2024

  22. [29]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 1, 3, 6, 12, 13, 14

    2023

  23. [30]

    Improved baselines with visual instruction tuning, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. 3, 4, 6, 7, 12, 13, 14, 15

    2024

  24. [31]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 1

    2024

  25. [32]

    A survey on hallucination in large vision-language models,

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models,

  26. [33]

    Mmbench: Is your multi-modal model an all-around player?, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. 6, 13, 15

    2024

  27. [34]

    Muffin: Cu- rating multi-faceted instructions for improving instruction- following, 2024

    Renze Lou, Kai Zhang, Jian Xie, Yuxuan Sun, Janice Ahn, Hanzi Xu, Yu Su, and Wenpeng Yin. Muffin: Cu- rating multi-faceted instructions for improving instruction- following, 2024. 6, 13, 14

    2024

  28. [35]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 12

    2016

  29. [36]

    Gpt-driver: Learning to drive with gpt, 2023

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt, 2023. 1

    2023

  30. [37]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 12

    2019

  31. [38]

    Mm1: Methods, analysis & insights from multimodal llm pre- training, 2024

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu H`e, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang...

    2024

  32. [39]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–

    2019

  33. [40]

    Introducing chatgpt

    OpenAI. Introducing chatgpt. 2022. 1

    2022

  34. [41]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023. 1, 3

    2023

  35. [42]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1, 3

    2023

  36. [43]

    hello-gpt-4o, 2024

    OpenAI. hello-gpt-4o, 2024. 1, 3

    2024

  37. [44]

    Strengthening multi- modal large language model with bootstrapped preference optimization, 2024

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Run- tao Liu, Rui Pan, and Tong Zhang. Strengthening multi- modal large language model with bootstrapped preference optimization, 2024. 1, 2, 3, 6, 7, 14

    2024

  38. [45]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  39. [46]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 14

    2024

  40. [47]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Confer- ence for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 12

    2020

  41. [48]

    Object hallucination in image cap- tioning, 2019

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning, 2019. 3, 6, 13

    2019

  42. [49]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. In European Conference on Computer Vision , pages 146–162. Springer, 2022. 12

    2022

  43. [50]

    https://sharegpt.com/, 2023

    ShareGPT. https://sharegpt.com/, 2023. 12

    2023

  44. [51]

    Textcaps: a dataset for image caption- ing with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image caption- ing with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 742–758. Springer,

    2020

  45. [52]

    Drivelm: Driving with graph visual question answering, 2024

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering, 2024. 1

    2024

  46. [53]

    Towards vqa models that can read, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019. 6, 13

    2019

  47. [54]

    Aligning large multimodal models with factually augmented rlhf, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf, 2023. 3, 6, 13, 14

    2023

  48. [55]

    Gemini: A family of highly capable multi- modal models

    Gemini Team. Gemini: A family of highly capable multi- modal models. arXiv preprint arXiv: 2312.11805 , 2023. 1, 3

    2023 arXiv

  49. [56]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023. 1 10

    2023 arXiv

  50. [57]

    Llama 2: Open foundation and fine-tuned chat models.ArXiv preprint, abs/2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.ArXiv preprint, abs/2307.09288, 2023. 1

    2023 arXiv

  51. [58]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 3

    2024 arXiv

  52. [59]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding, 2024

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Bie- mann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding, 2024. 3

    2024

  53. [60]

    Wong, Zhenguo Li, and Hengshuang Zhao

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K. Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Let- ters, 9(10):8186–8193, 2024. 1

    2024

  54. [61]

    Recurformer: Not all transformer heads need self-attention, 2024

    Ruiqing Yan, Linghan Zheng, Xingbo Du, Han Zou, Yufeng Guo, and Jianfei Yang. Recurformer: Not all transformer heads need self-attention, 2024. 1

    2024

  55. [62]

    Gpt4tools: Teaching large language model to use tools via self-instruction

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36, 2024. 1

    2024

  56. [63]

    Rlhf-v: Towards trust- worthy mllms via behavior alignment from fine-grained cor- rectional human feedback, 2024

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trust- worthy mllms via behavior alignment from fine-grained cor- rectional human feedback, 2024. 3, 6, 7, 13, 14

    2024

  57. [64]

    Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023. 6, 13, 14

    2023

  58. [65]

    Less is more: Mitigat- ing multimodal hallucination from an eos decision perspec- tive, 2024

    Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigat- ing multimodal hallucination from an eos decision perspec- tive, 2024. 6, 14

    2024

  59. [67]

    Mmicl: Empowering vision-language model with multi-modal in-context learning, 2024

    Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning, 2024. 1, 3, 12

    2024

  60. [68]

    Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization, 2024

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization, 2024. 6, 14

    2024

  61. [69]

    Aligning modalities in vision large lan- guage models via preference fine-tuning, 2024

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large lan- guage models via preference fine-tuning, 2024. 6, 14

    2024

  62. [70]

    Analyzing and mitigating object hallucination in large vision-language models, 2024

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models, 2024. 3

    2024

  63. [71]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023. 3

    2023

  64. [72]

    The Lion King

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision- language models via image-biased decoding, 2024. 3 11 Appendix This appendix is organized as follows. • In Section A, we discuss the limitations for LACING. • ...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.