Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

This paper claims that LVLM hallucination stems from an attention imbalance biased toward text, and that enriching every text-token embedding with an average-pooled visual summary (VisAlign) rebalances attention and reduces hallucinations a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:32 UTC pith:5B7LCE2L

load-bearing objection A plausible, parameter-light fusion tweak whose benchmark gains are real-but-unproven; the paper needs error bars, training details, and one equation fix before I'd buy the headline. the 4 major comments →

arxiv 2511.05017 v2 pith:5B7LCE2L submitted 2025-11-07 cs.CV cs.CL

Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings

Aakriti Agrawal , Gouthaman KV , Rohith Aralikatti , Gauri Jagatap , Jiaxin Yuan , Sarvesh Baskar , Vijay Kamarshi , Andrea Fanelli
show 1 more author
Furong Huang
This is my paper
classification cs.CV cs.CL
keywords hallucination mitigationlarge vision-language modelsmodality imbalanceattention distributiontextual embedding refinementvisual groundingmultimodal reasoningVisAlign
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that when visual embeddings are merely appended to an LLM's text input, the language backbone keeps favoring textual tokens, so outputs stay fluent but drift from the image. It proposes VisAlign, which averages the projected visual tokens and concatenates that summary into every text-token embedding before a linear projection returns them to the original dimension. Trained on the same recipe as the baseline, VisAlign shifts attention toward visual tokens and yields consistent benchmark gains, including +9.33% on a fine-grained visual-pair benchmark and +2.99% on an object-presence test. The point is that hallucination can be mitigated at the input-representation level, without architectural changes or extra supervision.

Core claim

The core discovery is that the conventional practice of appending visual embeddings to the text sequence leaves the pre-trained LLM's language bias intact: attention concentrates on text tokens at both ends of the sequence and under-uses the visual tokens in between. VisAlign fixes this by injecting visual context directly into each text-token embedding before the LLM processes the sequence. Concretely, it average-pools the projected visual tokens into one vector, concatenates that vector to every text embedding (doubling the feature dimension), and applies a linear projection back to the LLM's embedding size. The resulting visually grounded text tokens, followed by the original visual token

What carries the argument

The central mechanism is a fusion block: \hat V = mean-pool of the projected visual tokens; TV = [T \parallel \hat V \otimes \mathbf{1}_{N_t}] concatenation along the feature dimension; then \hat T = TV W_d, a learned linear projection back to the LLM embedding dimension. The sequence fed to the LLM becomes [\hat T_{1:k}, \hat V, \hat T_{k+1:N_t}]. This operation ensures every text position already carries visual context before self-attention, which is what rebalances the attention distribution and reduces reliance on language priors.

Load-bearing premise

The evaluation assumes that running the same datasets and training strategy as the baseline, with the VisAlign block added, isolates the effect of the method; no hyperparameters, learning-rate schedules, data splits, or random seeds are reported, so training noise or undocumented recipe differences could explain part or all of the score gaps.

What would settle it

Retrain the baseline and the VisAlign-augmented model with identical hyperparameters, data order, and multiple seeds; if the reported benchmark gaps shrink to within seed variance, the central claim that VisAlign reduces hallucinations is not supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Hallucination can be reduced at the input-representation stage: no new architecture, no external supervision, and no decoding-time correction are required.
  • Averaged visual context fused into every text token produces more balanced attention maps, with sharper attention bands on visual tokens across the first six layers.
  • Gains appear on benchmarks that specifically punish language-prior reliance: +9.33 points on fine-grained visual-pair questions, +2.99 points accuracy and +1.18 F1 on object-presence probing, up to +3.4 points on object-existence verification, and about +3 points on the hard split of a visual-illusion/conflict suite.
  • VisAlign complements an inference-time contrastive decoding method: combining them raises object-presence accuracy from 54.1 to 58.8 and F1 from 68.45 to 70.04.
  • The method transfers to a second image-language baseline, improving object-presence accuracy by 2 points, while introducing modest drops in some world-knowledge-heavy subcategories but improving grounding-heavy ones.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The attention-balance diagnosis suggests a cheap diagnostic: measuring the text-versus-visual attention ratio on a few layers could predict hallucination propensity without running full benchmarks.
  • Average pooling discards spatial and temporal structure; a learned pooling or per-token gating might yield larger gains while keeping the same core idea.
  • Because the injection happens before the LLM, VisAlign could combine with visual-token pruning to cut compute without losing grounding.
  • The trade-offs on world-knowledge-heavy tasks suggest the fusion strength could be conditioned on task type to avoid suppressing useful linguistic priors.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VisAlign, a training-time method for reducing hallucinations in Large Vision-Language Models (LVLMs). The key idea is to replace each textual token embedding with a visually informed embedding obtained by concatenating the original text token with an average-pooled visual embedding and then projecting the result back to the LLM embedding dimension. The method is applied to Video-LLaVA and evaluated on MMVP-MLLM, POPE, MERLIN, Mementos, and HallusionBench, with reported gains such as +9.33% on MMVP-MLLM, +2.99% on POPE-AOKVQA, and +3% on the HallusionBench hard split. The authors also provide attention-map visualizations suggesting a more balanced attention distribution after training with VisAlign.

Significance. If the reported gains are robust, VisAlign is a meaningful contribution: it is a simple, architecture-agnostic training-time intervention that targets modality imbalance at the input-representation level, rather than using post-hoc decoding corrections or hallucination-supervised fine-tuning. The paper includes a diagnostic attention analysis, evaluations on several external hallucination benchmarks, and an appendix showing compatibility with inference-time methods such as VCD and portability to LLaVA-1.5. However, the strength of the central claim rests on the reliability of the benchmark deltas, and the current manuscript does not provide error bars, significance tests, full training details, or a consistent treatment of regressions. The method is plausible but not yet rigorously established.

major comments (4)
  1. [§6, Tables 1–4] The paper claims 'statistically significant gains' in Sec. 1 and 'consistent improvements across all benchmarks' in Sec. 6, but no error bars, confidence intervals, significance tests, or random seeds are reported. This is load-bearing because the central result is a set of score deltas (e.g., +9.33% MMVP-MLLM, +2.99% POPE) that may be within training or evaluation noise. The concern is sharpened by unreported regressions: POPE recall drops from 99.6 to 98.33 in Table 1 and HallusionBench easy-split average drops from 43.97 to 41.1 in Table 4, and Mementos daily-life action accuracy drops in Table 3. A paired evaluation with repeated runs and standard deviations is needed before the deltas can be attributed to VisAlign.
  2. [§5, Eq. (4)] Equation (4) is not dimensionally consistent. \hat{V} is defined as 1×d_t in Eq. (2), yet \hat{X} is asserted to be in R^{(N_t+N_v)×d_t} with \hat{V} inserted into the concatenation. The RHS has N_t + 1 tokens, not N_t+N_v, unless the visual token sequence V (or V_proj) is meant. If the original visual tokens are discarded, this contradicts Fig. 2 and the description in §5; if they are retained, the equation needs to show V, not \hat{V}. The exact tensor shapes and the implementation of the concatenation block must be specified for the method to be reproducible.
  3. [§5, Training Stages] The statement 'We use the same datasets and training strategy as used in the baseline VideoLLaVA' is not sufficient to isolate the effect of VisAlign. No learning rate, batch size, number of epochs, warmup schedule, data splits, initialization, or random seed is reported. Changes in any of these can easily produce differences of the size reported in Tables 1–4. Please provide full training details and, ideally, code/checkpoints; otherwise the attribution of the observed gains to the fusion module is not established.
  4. [§6, Mementos and HallusionBench] The narrative that VisAlign 'consistently surpasses' or 'consistently improves' the baseline is not supported by Tables 3–4. In Mementos, action accuracy in the Daily Life domain drops from 13.50 to 12.31, and Comics action F1 drops from 8.08 to 7.10. In HallusionBench, the easy-split average falls from 43.97 to 41.1, with category-level drops in Figure, Illusion, Chart, Table, and others. These regressions should be reported in the abstract/conclusion and discussed; a selective summary is misleading.
minor comments (6)
  1. [Abstract] The phrase 'statistically significant gains' is not backed by any statistical test; please remove or substantiate with standard errors/p-values.
  2. [§4] The text says 'As defined in Eq.(1), the input sequence X follows a fixed order', but Eq. (1) defines only V_proj; X is never defined in the paper. Please add the definition of X and its token ordering.
  3. [§6, Table 2] Typos: 'benchamark' and 'Merlin' vs. 'MERLIN'/'MERLIM' in text and table; check the reference for Villa et al. 2023, which is cited as MERLIM but referred to as MERLIN.
  4. [§5] Equation (2) is not formatted cleanly (the summation index/limit is confusing). Please rewrite using standard summation notation.
  5. [General] There are inconsistent spellings of 'Video-LLaVA'/'Video-LLava' and 'HallusionBench' labels; a final proofread would help.
  6. [Appendix A, Table 6] The MME table shows several categories with drops (e.g., Position 135→125, Celebrity 103.74→78.24? Please check) and the caption lacks model names for each column. Please add clear column headers and a sensitivity discussion.

Circularity Check

0 steps flagged

No significant circularity: VisAlign's benchmark claims rest on external evaluations, not on a derivation chain that reduces to its inputs.

full rationale

The paper does not present a formal derivation chain of the kind that would support a circularity finding. Its central claim is empirical: adding a fusion module (average-pooled visual embedding concatenated to each textual embedding, then linearly projected) to Video-LLaVA yields lower hallucination scores on external benchmarks. The method's equations (2)-(4) define a transformation, but the reported improvements on MMVP, POPE, Merlin, Mementos, and HallusionBench are measured against externally published benchmarks rather than being implied by the construction. The attention-distribution analysis in Section 4 is a diagnostic motivation, not a proof that the benchmark deltas must occur. There is no fitted parameter that is later renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no self-citation chain that forces the conclusion. The HallusionBench citation includes a co-author, but the benchmark is a published external evaluation suite with many authors; its use does not make the empirical results circular. Eq. (4) appears dimensionally inconsistent or typographically sloppy, but dimensional inconsistency is a reproducibility/correctness concern, not a circularity concern. The lack of hyperparameters, seeds, and significance tests could undermine attribution of the gains, but that too is a reporting/correctness issue rather than an instance of the paper reducing a prediction to its own inputs by construction. Honest non-finding: no specific circular step can be quoted, so the score is 0.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The paper contributes a learned fusion pipeline, not a derivation. The central empirical claim rests on domain assumptions about what causes hallucinations and how to measure them, plus a new projection matrix that is trained but not specified.

free parameters (1)
  • W_d projection matrix (VisAlign fusion layer) = learned (not reported)
    Introduced in Eq. (3), W_d maps concatenated text+visual embeddings back to d_t dimensions. Its values are fit during pretraining and finetuning; no trained values or initialization are given.
axioms (5)
  • domain assumption Hallucinations in LVLMs stem substantially from over-reliance on textual priors and under-utilization of visual cues.
    Central hypothesis in Abstract/Sec. 1; Sec. 4 supports it with qualitative attention maps only.
  • domain assumption A more balanced attention distribution (more attention to visual tokens) reduces hallucinations.
    Stated as the 'underlying hypothesis' in Sec. 5; not independently validated.
  • domain assumption Average pooling of all projected visual embeddings preserves enough visual information for grounding.
    Eq. (2) collapses N_v visual tokens to one vector; Sec. 8 admits more sophisticated fusion could improve results.
  • domain assumption Benchmark scores on POPE, MMVP-MLLM, Merlin, and HallusionBench are valid proxies for hallucination reduction.
    The paper uses these benchmarks as ground truth for hallucination without discussing their limitations or mutual disagreements.
  • domain assumption Using the same datasets and training strategy as the baseline isolates the effect of VisAlign.
    Sec. 5 states the recipe is identical but provides no hyperparameters, seeds, or data splits, so unobserved differences could confound results.

pith-pipeline@v1.3.0-alltime-deepseek · 15329 in / 14404 out tokens · 119875 ms · 2026-08-03T23:32:13.193428+00:00 · methodology

0 comments
read the original abstract

Hallucinations in Large Vision-Language Models (LVLMs) remain a persistent challenge, often stemming from inadequate integration of visual information during multimodal reasoning. A key cause is the model's over-reliance on textual priors and underutilization of visual cues, leading to outputs that are linguistically fluent but visually inaccurate. For example, given an image of an empty kitchen countertop, an LVLM might hallucinate a "bowl of fruit" or "cup of coffee", relying on language associations rather than visual evidence. Most LVLMs incorporate visual features by appending them to the input stream of a pre-trained LLM and training on large-scale vision-language datasets. Our systematic analysis reveals that this strategy often leads to over-dependence on textual information due to the inherent bias of LLMs towards language-dominant representations. This imbalance skews attention towards the text over visual content, weakening the model's ability to ground outputs in visual inputs. To address this, we propose a simple yet effective visual feature incorporation method that encourages the model to learn visually-informed textual embeddings distinct from those of the base LLM and promotes a more balanced attention distribution. Experimental results across multiple hallucination benchmarks demonstrate that our method significantly reduces hallucinations and fosters more balanced multimodal reasoning. Notably, our approach achieves substantial gains, including +9.33% on MMVP-MLLM, +2.99% on POPE-AOKVQA, up to +3.4% on Merlin, and +3% on the hard-data split of HallusionBench.

Figures

Figures reproduced from arXiv: 2511.05017 by Aakriti Agrawal, Andrea Fanelli, Furong Huang, Gauri Jagatap, Gouthaman KV, Jiaxin Yuan, Rohith Aralikatti, Sarvesh Baskar, Vijay Kamarshi.

Figure 2
Figure 2. Figure 2: Top: Architecture of typical LVLMs like Video-LLaVA, which fuse language and vision embeddings by simple concatenation. Bottom: Our modified architecture with a concatenation block that appends the averaged vision embedding to each token embedding, followed by a projection layer. This encourages the model to learn visually informed textual embeddings and better attend to visual input during training. the e… view at source ↗
Figure 3
Figure 3. Figure 3: Attention score distributions across the first six attention layers of the baseline Video-LLaVA model (top row) and the VisAlign-enhanced model (bottom row). Video-LLaVA concatenates tokens in a fixed order: 35 initial text tokens, followed by 256 visual embeddings, and then the remaining text tokens. In each map, the x-axis denotes attended tokens (keys), and the y-axis denotes attending tokens (queries).… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results from the MMVP-MLLM Benchmark: Below each image, the baseline model’s response is shown first, followed by the response from the model trained with VisAlign. work (Villa et al., 2025), we focus on the most challenging setting: Adversarial SEEM from A-OKVQA, which applies SEEM-based object detection to A-OKVQA images. This subset probes whether models falsely affirm the presence of common… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative examples from POPE A-OKVQA, HallusionBench, MMVP, and Mementos benchmarks illustrating various hallucination types. Input prompts are shown in orange, baseline Video-LLaVA outputs in yellow, and VisAlign￾enhanced outputs in green. VisAlign consistently improves performance across object, action, attribute, and relation hallucinations. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results on the Mementos benchmark (Wang et al., 2024a). Text highlighted in red indicates hallucinated content, while text in blue shows the corresponding corrections. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Improving Vision-language Models with Perception-centric Process Reward Models

    cs.CV 2026-04 unverdicted novelty 7.0

    Perceval is a perception-centric PRM that detects token-level perceptual errors in VLMs, supporting token-advantage RL training and iterative test-time scaling for improved reasoning.

  2. Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth

    cs.CV 2026-05 conditional novelty 6.0

    Constraining visual token budgets during SFT and RL forces VLMs to learn functional active perception, yielding ~5% relative gains and strong transfer to unconstrained evaluation.

  3. Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth

    cs.CV 2026-05 unverdicted novelty 5.0

    Constraining visual token budget per observation during VLM training forces genuine active perception and delivers 5% average relative improvement without auxiliary losses or architecture changes.

Reference graph

Works this paper leans on

20 extracted references · 12 linked inside Pith · cited by 2 Pith papers

  1. [4]

    Visual description grounding reduces hallucinations and boosts reasoning in lvlms.arXiv preprint arXiv:2405.15683,

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, Oriol Nieto, Zeyu Jin, and Dinesh Manocha. Visual description grounding reduces hallucinations and boosts reasoning in lvlms.arXiv preprint arXiv:2405.15683,

    Pith/arXiv arXiv

  2. [5]

    Efficientmultimodal learning from data-centric perspective.arXiv preprint arXiv:2402.11530,

    MuyangHe,YexinLiu,BoyaWu, JianhaoYuan,YuezeWang,TiejunHuang, andBoZhao. Efficientmultimodal learning from data-centric perspective.arXiv preprint arXiv:2402.11530,

    Pith/arXiv arXiv

  3. [7]

    Faith: Faithful and informative textual hallucination detection in image captioning

    Long Jing, Zhe Wang, Yichen Zhang, Dacheng Tao, and Mingli Song. Faith: Faithful and informative textual hallucination detection in image captioning. InProceedings of the 2023 Conference on Computer Vision and Pattern Recognition, pages 3456–3465. IEEE,

    2023

  4. [8]

    URL https://openaccess.thecvf.com/content/CVPR2023/html/Jing_FAITH_Faithful_ and_Informative_Textual_Hallucination_Detection_in_Image_Captioning_CVPR_2023_ paper.html

    doi: 10.1109/CVPR.2023.00345. URL https://openaccess.thecvf.com/content/CVPR2023/html/Jing_FAITH_Faithful_ and_Informative_Textual_Hallucination_Detection_in_Image_Captioning_CVPR_2023_ paper.html. 13 Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu...

    arXiv 2023

  5. [9]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023a. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large ...

    Pith/arXiv arXiv

  6. [10]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee

    URLhttps://arxiv.org/abs/ 2411.02571. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024a. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuni...

    Pith/arXiv arXiv

  7. [11]

    URL https: //openaccess.thecvf.com/content/ICCV2023/html/Lovenia_NOPE_Evaluating_and_ Explaining_Negative_Object_Presence_in_Image_Captioning_ICCV_2023_paper.html

    doi: 10.1109/ICCV.2023.00123. URL https: //openaccess.thecvf.com/content/ICCV2023/html/Lovenia_NOPE_Evaluating_and_ Explaining_Negative_Object_Presence_in_Image_Captioning_ICCV_2023_paper.html. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv...

    arXiv 2023

  8. [12]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024),

    2024

  9. [13]

    Aloha: Assessing language-only hallucinations in image captioning

    Nathan Petryk, Shikhar Sharma, Ali Furkan Biten, Lluis Gomez, Dimosthenis Karatzas, C V Jawahar, and Minesh Mathew. Aloha: Assessing language-only hallucinations in image captioning. InPro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6789–

    2023

  10. [14]

    URL https://aclanthology.org/2023.emnlp-main.567/

    doi: 10.18653/v1/2023.emnlp-main.567. URL https://aclanthology.org/2023.emnlp-main.567/. Hao Sun et al. Mmhal-bench: Multimodal hallucination benchmark for vision-language dialogue.arXiv preprint arXiv:2312.00704,

    Pith/arXiv arXiv 2023

  11. [15]

    Behind the magic, merlim: Multi-modal evaluation benchmark for large image-language models.arXiv preprint arXiv:2312.02219,

    Andrés Villa, Juan Carlos León Alcázar, Alvaro Soto, and Bernard Ghanem. Behind the magic, merlim: Multi-modal evaluation benchmark for large image-language models.arXiv preprint arXiv:2312.02219,

    Pith/arXiv arXiv

  12. [16]

    Eagle: Enhanced visual grounding minimizes hallucinations in instructional multimodal models.arXiv preprint arXiv:2501.02699,

    Andrés Villa, Juan León Alcázar, Motasem Alfarra, Vladimir Araujo, Alvaro Soto, and Bernard Ghanem. Eagle: Enhanced visual grounding minimizes hallucinations in instructional multimodal models.arXiv preprint arXiv:2501.02699,

    Pith/arXiv arXiv

  13. [17]

    Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences.arXiv preprint arXiv:2401.10529, 2024a

    Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences.arXiv preprint arXiv:2401.10529, 2024a. Yixin Wang, Yuxiang Liu, Chunyuan Chen, Zhe Wang, Shuchang Yan, et al. Mitiga...

    Pith/arXiv arXiv

  14. [18]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment.arXiv preprint arXiv:2310.01852, 2023a

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment.arXiv preprint arXiv:2310.01852, 2023a. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vi...

    Pith/arXiv arXiv 2024

  15. [19]

    We evaluate the individual and combined effects of Visual Contrastive Decoding (VCD), and VisAlign

    54.5 52.38 99.39 68.6 + VisAlign 57.09 53.9 98.33 69.63 + VisAlign + VCD 58.8 55.0396.3370.04 Table 5:Comparison of baseline Video-LLava with different combination of hallucination mitigation approaches on POPE-AOKVQA. We evaluate the individual and combined effects of Visual Contrastive Decoding (VCD), and VisAlign. The combination of both yields the bes...

    arXiv

  16. [20]

    170 105.00 76.66 125 100.00 100.88 155.75 154.5 99.25 77.5 +VisAlign 190 131.6653.33148.33103.06 78.24 151 125 94 87.5 Table 6:Comparison of baseline Video-LLava with different combination of hallucination mitigation approaches on MME. Effect of VisAlign on generic LVLM benchmarks:In the main paper, we comprehensively evaluated 16 Towards Mitigating Hallu...

    arXiv 2023

  17. [2022]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 3,

    Fu Chaoyou, Chen Peixian, Shen Yunhang, Qin Yulei, Zhang Mengdan, Lin Xu, Yang Jinrui, Zheng Xiawu, Li Ke, Sun Xing, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 3,

    Pith/arXiv arXiv

  18. [2023]

    Pali: A jointly-scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794,

    Pith/arXiv arXiv

  19. [2024]

    Mitigating object hallucinations in large vision-language models via attention calibration

    Jing Huo et al. Mitigating object hallucinations in large vision-language models via attention calibration. arXiv preprint arXiv:2502.01969,

    arXiv

  20. [2025]

    Paint: Paying attention to informed tokens to mitigate hallucination in large vision-language model.arXiv preprint arXiv:2501.12206,

    Kazi Hasan Ibn Arif, Sajib Acharjee Dip, Khizar Hussain, Lang Zhang, and Chris Thomas. Paint: Paying attention to informed tokens to mitigate hallucination in large vision-language model.arXiv preprint arXiv:2501.12206,

    Pith/arXiv arXiv