Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models

Fuzheng Zhang; Junfei Wu; Qiang Liu; Tieniu Tan; Xinlong Chen; Yuanxing Zhang

REVIEW 4 major objections 5 minor 1 cited by

Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mixture of Decoding reduces vision-language hallucinations by comparing full-image outputs with outputs from the model's attended image tokens and switching between complementary and contrastive decoding at each step.

desk verdict A well-ablated decoding heuristic that reduces LVLM hallucination; the attention-correctness interpretation is not supported, and the SOTA claim is undercut by DeGF. read the letter →

arxiv 2505.17061 v3 pith:QF5VABJK submitted 2025-05-17 cs.CL cs.AIcs.CV

Xinlong Chen , Yuanxing Zhang , Qiang Liu , Junfei Wu , Fuzheng Zhang , Tieniu Tan This is my paper · ORCID

classification cs.CLcs.AIcs.CV

keywords hallucinationmitigationlargevision-languagemodelsmixtureofdecodingcontrastiveattentionmaskingJensen-Shannondivergencetraining-freeobject

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models routinely produce fluent descriptions that mention objects or attributes absent from the image. The paper claims that this failure can be caught during decoding by comparing the next-token distribution predicted from the full image with the distribution predicted from only the image tokens the model's attention concentrates on. When the two agree, Mixture of Decoding (MoD) treats the attention as correct and adds the two logit streams to amplify what the model is looking at; when they diverge past a fixed threshold, MoD treats the attention as misleading and subtracts the attended-token logits from the original ones. The switch is training-free, token-by-token, and uses one shared set of hyperparameters across models and benchmarks. On POPE, MME, CHAIR, and AMBER, MoD is reported to outperform VCD, M3ID, and AvisC on LLaVA-1.5, Qwen-VL, and LLaVA-NEXT, which matters because it suggests hallucination can be reduced without retraining or extra data.

What carries the argument

The load-bearing mechanism is the attention-masked image $v_{\text{att}}$: the original image token sequence with every token outside the top $\lambda=0.2$ fraction of average attention (computed over all layers and heads from the final input token's attention to image tokens) zeroed out. The model is run twice—once on the full image tokens $v$ and once on $v_{\text{att}}$—and the Jensen-Shannon divergence $d(v,v_{\text{att}})$ between the two next-token distributions is the switch that selects between the complementary and contrastive branches of Eq. (6). Figure 1 supplies the paper's evidence that the switch is meaningful: non-hallucinated POPE outputs cluster at low divergence, and CHAIRi correlates with divergence at Pearson $r=0.85$.

What would settle it

Build a set of images with a visually salient decoy that strongly attracts the model's attention while a different object is the correct answer, and check whether hallucinated tokens still occur with $d(v,v_{\text{att}})\le\gamma$; if many do, the consistency criterion is not separating correct from incorrect attention. A quantitative form is to measure, on a large corpus with human-verified object presence, whether the $\gamma=0.05$ threshold separates hallucinated from non-hallucinated generations better than a language-prior-only baseline.

Watch

Extended reading notes

Core claim

MoD's central claim is that a single divergence number—the Jensen-Shannon divergence between $p_\theta(y_t\mid v,x,y_{<t})$ and $p_\theta(y_t\mid v_{\text{att}},x,y_{<t})$, where $v_{\text{att}}$ is the image with all but the top $\lambda$ attended tokens zeroed out—tells whether the model's visual attention is reliable at the current step. If $d(v,v_{\text{att}})\le\gamma$, the outputs are consistent, the attention is judged correct, and MoD complements the logits (adding $\alpha_1$ times the attended-token logits to the full-image logits). If $d(v,v_{\text{att}})>\gamma$, the attention is judged wrong, and MoD contrasts (weighting the full-image logits by $1+\alpha_2$ and subtracting $\alpha_2$ times the attended-token logits). This per-token switch is applied without training and with shared hyperparameters ($\lambda=0.2$, $\alpha_1=4$, $\alpha_2=1$, $\gamma=0.05$) across three LVLM architectures, and the paper reports consistent gains over VCD, M3ID, and AvisC on object, attribute, and relation hallucination benchmarks.

Load-bearing premise

The load-bearing premise is that agreement between the full-image output and the attended-image output is a reliable sign the model is attending correctly, so a fixed threshold can tell good from bad attention rather than both outputs merely reflecting the same language prior.

Editorial extensions

If this is right

MoD is a training-free, model-agnostic intervention: the same hyperparameters are reported to work on three different LVLM architectures and four benchmarks, so it can be layered onto already deployed models without retraining.
Because the consistency test is applied per generated token, the method can react within a single response, suppressing a hallucinated object as it is about to be generated and returning to complementary decoding when attention is back on track.
The reported gains are largest on false-positive and object-existence errors (POPE precision, MME existence, CHAIRi), suggesting the divergence switch specifically reduces the yes-saying and object-insertion tendencies of LVLMs.
At the same two-forward-pass cost as VCD, M3ID, and AvisC, the adaptive mixture beats either branch used alone, as shown in the ablation where MoD outperforms both pure complementary and pure contrastive decoding on MME.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One extension the authors do not test: replacing the hard zero-mask with soft attention-weighted pooling could preserve spatial layout and recover some of the position-sensitive MME scores that MoD currently sacrifices.
Since the divergence is computed from two forward passes at every step, caching the attended-token pass or sharing cached attention states could cut MoD's doubling of latency, a limitation the authors acknowledge.
The same consistency criterion could serve as a standalone zero-shot hallucination detector for LVLMs, flagging tokens where the full-image and attended-image distributions diverge, without needing the complementary or contrastive branches.
A testable refinement is to make the threshold or the attended-token proportion adaptive to model or input difficulty; the paper shows stability over a range but does not explore per-image thresholds.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: worth a serious referee, but read the adaptive-switch story as a heuristic, not as proof about attention correctness. MoD is a token-level switch: mask low-attention image tokens, run a second forward pass, and either add the masked logits to the original (low JS) or subtract them (high JS). The experiments are wide — three models, four benchmarks, consistent wins over VCD/M3ID/AvisC — and the ablations cover all four hyperparameters. Code is released. That is real work, and the method will likely become a useful baseline.

The genuinely new bit is replacing DeGF's generated image with an attention-masked copy of the same image and adding a complementary branch when the two passes agree. That is a legitimate combination of known pieces, and the paper is honest about the lineage. It is not a big conceptual leap.

The weakest point is the load-bearing interpretation. Section 3.3 claims that low JS between the original and masked outputs indicates the model's attention is right. That does not follow. If attention is wrong but strongly concentrated on the wrong tokens, masking low-attention tokens changes little, JS stays low, and MoD runs the complementary branch, amplifying hallucinated logits exactly when contrastive suppression is needed. The paper's own MME position result shows the mirror image: zeroing low-attention tokens can remove decisive evidence, inflate JS, and send you to contrastive when attention was actually useful. Figure 1 shows a correlation between JS and hallucination, not between JS and attention correctness. The mechanism as described is underspecified; the numbers support a generic partial-image complement/contrast heuristic.

Two more issues. Everything is single-seed with no error bars, which matters for a sampling-based decoder at temperature 1. And the closest adaptive baseline, DeGF, appears only in Appendix B.5, where it slightly beats MoD on MME (643.3 vs 638.3). The paper admits this and notes DeGF costs extra image generation — a fair trade-off — but the abstract's 'significantly outperforms existing decoding methods' is too strong.

Verdict: conditional accept with revisions. The core result, that MoD reduces hallucination across models and benchmarks, is credible and reproducible in principle. The attention-correctness interpretation should be softened or tested directly (e.g., with misdirection cases or interventions). Add multi-seed results and move DeGF into the main table. A serious editor should send this to review, not desk-reject it.

Referee Report

4 major / 5 minor

Summary. The paper proposes Mixture of Decoding (MoD), a training-free adaptive decoding strategy for mitigating hallucinations in large vision-language models. MoD computes the Jensen-Shannon divergence between the output distribution from the original image tokens and the output distribution from a masked input retaining only the top-λ attended image tokens. If the divergence is below a threshold γ, MoD treats the model's attention as correct and combines the two logit distributions additively; otherwise it treats the attention as erroneous and applies contrastive decoding. The method is evaluated on POPE, MME, CHAIR, and AMBER across LLaVA-1.5, Qwen-VL, and LLaVA-NEXT, and is compared with sampling, VCD, M3ID, and AvisC. The paper reports consistent gains and argues that MoD is robust to hyperparameter choices.

Significance. If the central mechanism is validated, MoD is a useful contribution: it is training-free, model-agnostic, and the reported gains are consistent across four benchmarks and three LVLMs. The paper also provides ablations on the key hyperparameters (λ, α1, α2, γ, β) and releases code, which strengthens reproducibility. The main conceptual novelty is the adaptive switch between complementary and contrastive decoding based on a consistency criterion. However, the validity of this switch rests on the claim that consistency between original and attended-only outputs indicates correct attention. This claim is not directly tested, and the paper's SOTA claim is qualified by the DeGF comparison in Appendix B.5. These issues are substantial but addressable, so the work is promising rather than settled.

major comments (4)

[§3.3, Eq. (5)–(6)] The load-bearing premise that low JS divergence between pθ(yt|v) and pθ(yt|vatt) indicates correct attention is not secure. If the model's attention is wrong but strongly committed to the top-λ tokens, then vatt keeps those tokens and pθ(yt|vatt) will closely mirror pθ(yt|v), yielding a low JS value and routing to the complementary branch—exactly when contrastive suppression is needed. Conversely, zeroing low-attention tokens can remove decisive visual evidence when attention to it was low, inflating JS and routing to contrastive even when attention is correct. Figure 1 shows that JS is correlated with hallucination, but it does not show that this correlation is driven by attention correctness rather than by sensitivity to any token removal. The authors should provide a control experiment (e.g., random token masking at the same retention rate) and, ideally, an evaluation of the gate's precision/recall against an independent measure of attention correctness.
[§4.1, 'Implementation Details'] All results are reported for a single fixed random seed (seed 42) under temperature=1 and top-p=1. Since the method involves stochastic decoding, the absence of multiple seeds or confidence intervals makes it impossible to assess whether small reported gains (e.g., 86.0 vs. 85.5 for Qwen-VL on POPE random) are statistically meaningful. The authors should rerun the main comparisons over at least three seeds and report means and variances, or otherwise justify why single-seed results are representative.
[Appendix B.5, Table 9] The abstract claims MoD 'significantly outperforms existing decoding methods,' but the DeGF comparison in Table 9 shows DeGF achieving a higher total MME score (643.3 vs. 638.3) on LLaVA-1.5. This directly qualifies the SOTA claim. DeGF should either be included in the main experimental tables and discussed in the abstract, or the claims should be restricted to the specific baselines evaluated in the main text. The paper should also report comparisons with DeGF on additional benchmarks and models before making an unqualified superiority claim.
[§3.2, Eq. (2)–(3)] The attention aggregation in Eq. (2) averages attention weights uniformly over all layers and heads, and Eq. (3) then selects the top λ proportion of image tokens. This ignores the known heterogeneity of attention heads: some heads may encode visual grounding while others encode positional or textual information. The paper does not justify why uniform averaging is appropriate, nor does it test whether the conclusions change when using a different layer/head subset. The consistency criterion in §3.3 could behave differently under alternative aggregations, so an ablation on the attention aggregation would strengthen the attention-correctness story.

minor comments (5)

[Throughout] The model name is typeset inconsistently as 'LLaV A' (with a space) in many places; it should be 'LLaVA' throughout.
[§4.3, Figure 4] The text states that γ is varied from 0.02 to 0.08, while the figure caption says the range is 0.01 to 0.10; these should be reconciled.
[Eq. (6)] The notation pθ(yt|v,vatt,x,y<t) overloads the probability symbol; it should clearly denote the modified MoD distribution, not an ordinary conditional distribution given both image representations.
[Appendix B.5] The comparison with DeGF is reported only on MME and only for LLaVA-1.5; this is too narrow to support the general statement that MoD is preferable in terms of accuracy/cost trade-off.
[Fig. 1] The right panel reports a Pearson correlation of 0.85 for CHAIRi but does not specify over how many samples or which model; adding this information would improve interpretability.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor interpretative circularity: 'attention correctness' is operationalized by the same JS divergence used to choose the decoding branch; the benchmark results remain external and independent.

self definitional [Section 3.3, Eq. 5-6]
"If the output based onvatt aligns with the original output derived fromv, it indicates that the model’s attention over image tokens is right, as illustrated in the upper part of Fig. 2. Conversely, if the outputs are inconsistent, it suggests that the model’s attention over image tokens is wrong, as depicted in the lower part of Fig. 2."

The only operational content assigned to 'attention right/wrong' is the JS divergence d(v,vatt) in Eq. 5 between pθ(yt|v) and pθ(yt|vatt). The same d(v,vatt)≤γ then selects complementary versus contrastive decoding in Eq. 6. So the claim that MoD 'evaluates the correctness of the model's attention' is true by construction rather than by an independent ground-truth measurement of attention quality. This does not make the external benchmark results circular, but the paper's central interpretative claim reduces to its own decision variable.

full rationale

MoD's adaptive switch has a mild self-referential element: attention is labeled 'right' or 'wrong' by the same JS divergence d(v,vatt) that chooses the complementary or contrastive branch, so 'evaluating attention correctness' is partly definitional. There is also a same-group citation (Liu et al., 2025) behind the consistency-correctness premise, though the idea is also credited to the external SelfCheckGPT line. However, the central reported predictions—POPE, MME, CHAIR, and AMBER scores—are not fitted to those targets or renamed from the criterion; they are external evaluations with fixed hyperparameters and robustness ablations. The concern that low JS can arise from wrong-but-dominant attention reproducing the hallucinated distribution, or that masking can remove decisive evidence (as the paper itself concedes for the MME 'position' subset), is a correctness-risk argument about whether consistency tracks genuine attention quality, not a circular derivation. Overall, the circularity is interpretative and minor, and the benchmark claims remain self-contained.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method contributes an adaptive decoding rule, but its core correctness signal depends on several unproven modeling premises: attention as a faithful relevance map, consistency as a proxy for attention correctness, and zeroing-out as a safe counterfactual. The numerical hyperparameters are fixed from ablations on the MME benchmark, which is also used as a headline result. No new entities are introduced.

free parameters (5)

lambda (attended token proportion) = 0.2
Proportion of highest-attention image tokens retained when masking low-attention tokens in Eq. 3. Chosen on MME and held fixed across tasks and models.
alpha1 (complementary coefficient) = 4.0
Weight on attended-token logits in the complementary branch of Eq. 6. Tuned on MME in Appendix B.4.
alpha2 (contrastive coefficient) = 1.0
Weight on attended-token logits in the contrastive branch of Eq. 6. Tuned on MME in Appendix B.4.
gamma (JS divergence threshold) = 0.05
Threshold deciding whether to complement or contrast in Eq. 6. Tuned on MME and swept in Fig. 4.
beta (adaptive plausibility constraint) = 0.5
Truncation strength in Eq. 8. Set across all models and tasks after ablation on MME in Appendix B.4.

assumptions (3)

domain assumption The average attention of the final input token over image tokens, aggregated across all layers and heads, identifies the image tokens that matter for the current generation.
Section 3.2 states the final token "encapsulates the model's comprehensive understanding" and uses Eq. 2 to derive attended tokens; this is a modeling assumption not proven.
domain assumption Low JS divergence between p(yt|v) and p(yt|vatt) indicates correct attention, and high divergence indicates wrong attention.
Section 3.3 builds the entire adaptive switch on this dichotomy, supported by Fig. 1 but not formally derived.
domain assumption Zeroing out low-attention image tokens yields a valid counterfactual without corrupting the model's visual processing.
Section 3.2 masking is used to obtain vatt; the Limitations section notes that zeroing may reduce position sensitivity, implying the counterfactual is imperfect.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/QF5VABJK

@misc{pith2026250517061,
  author       = {Pith},
  title        = {Pith review of: Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QF5VABJK}},
  note         = {Machine review of arXiv:2505.17061}
}

read the original abstract

Large Vision-Language Models (LVLMs) have exhibited impressive capabilities across various visual tasks, yet they remain hindered by the persistent challenge of hallucinations. To address this critical issue, we propose Mixture of Decoding (MoD), a novel approach for hallucination mitigation that dynamically adapts decoding strategies by evaluating the correctness of the model's attention on image tokens. Specifically, MoD measures the consistency between outputs generated from the original image tokens and those derived from the model's attended image tokens, to distinguish the correctness aforementioned. If the outputs are consistent, indicating correct attention, MoD employs a complementary strategy to amplify critical information. Conversely, if the outputs are inconsistent, suggesting erroneous attention, MoD utilizes a contrastive strategy to suppress misleading information. Extensive experiments demonstrate that MoD significantly outperforms existing decoding methods across multiple mainstream benchmarks, effectively mitigating hallucinations in LVLMs. The code is available at https://github.com/xlchen0205/MoD.

Figures

Figures reproduced from arXiv: 2505.17061 by the authors.

**Figure 1.** The JS divergence excels at distinguishing hallucinated from non-hallucinated outputs, as evidenced in both discriminative tasks (POPE, left) and generative tasks (CHAIR, right). In the left figure, nonhallucinated outputs are primarily concentrated in regions with lower JS divergence values, while hallucinated outputs exhibit a pronounced long-tail distribution. In the right figure, there is a significant posit… view at source ↗

**Figure 2.** Overview of our proposed MoD. MoD involves three key steps: (1) extracting the model’s attended image tokens while masking the others; (2) generating vanilla output logits from both original image tokens and masked image tokens; and (3) computing the JS divergence between the two logit distributions to assess the correctness of the model’s attention. Based on this evaluation, MoD adaptively adopts either complementa… view at source ↗

**Figure 3.** Case study of generative tasks using QwenVL. We compare responses generated by sampling, VCD, M3ID, AvisC, and our proposed MoD. Hallucinated content is highlighted in red, while more detailed and accurate content is marked in green. the masking strategy, such as incorporating pooling mechanisms to allow the model to retain partial awareness of low-attention image tokens. We leave this exploration for future wor… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Ablation study on the consistency threshold γ. We vary γ from 0.01 to 0.10 with an increment of 0.01, and compare the results with those obtained using individual methods on MME benchmark. In MoD, γ is consistently set to 0.05 across all experiments. We provide a detai…

**Figure 5.** Figure 5: Ablation study of the decoding coefficient α1 and α2 on MME benchmark. We set α1 = 4.0 and α2 = 1.0 for all three evaluated LVLMs. For reference, the optimal performances achieved by the baseline methods in our experiments were 596.7, 593.3, and 613.3 for each respect…

**Figure 6.** Figure 6: Case study of generative tasks using QwenVL. We compare responses generated by sampling, VCD, M3ID, AvisC, and our proposed MoD. Hallucinated content is highlighted in red, while more detailed and accurate content is marked in green. in [PITH_FULL_IMAGE:figures/full…

**Figure 7.** Figure 7: Case study of generative tasks using LLaVA-1.5. We compare responses generated by sampling, VCD, M3ID, AvisC, and our proposed MoD. Hallucinated content is highlighted in red, while more detailed and accurate content is marked in green [PITH_FULL_IMAGE:figures/full_fi…

**Figure 8.** Figure 8: Case study of generative tasks using LLaVA-NEXT. We compare responses generated by sampling, VCD, M3ID, AvisC, and our proposed MoD. Hallucinated content is highlighted in red, while more detailed and accurate content is marked in green [PITH_FULL_IMAGE:figures/full_f…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models
cs.AI 2026-07 conditional novelty 6.0 of 10

Monitoring attention entropy and image-output correlation during decoding, then applying targeted contrastive corrections, reduces hallucination in multimodal LLMs without retraining.

Reference graph

Works this paper leans on

43 extracted references · 3 canonical work pages · cited by 1 Pith paper

[1]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023 a . Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023
[2]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023 b . Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023
[3]

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930

arXiv 2024
[4]

Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. 2024 a . Automated evaluation of large vision-language models on self-driving corner cases. arXiv preprint arXiv:2404.10595

arXiv 2024
[5]

Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David F Fouhey, and Joyce Chai. 2024 b . Multi-object hallucination in vision-language models. arXiv preprint arXiv:2407.06192

arXiv 2024
[6]

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883

arXiv 2023
[7]

Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495

arXiv 2023
[8]

Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14303--14312

2024

Show all 43 references

[9]

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394

2023 arXiv
[10]

Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. 2023. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. arXiv preprint arXiv:2310.05694

2023 arXiv
[11]

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Info...

2023
[12]

Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. 2024. Self-introspective decoding: Alleviating hallucinations for large vision-language models. arXiv preprint arXiv:2408.02032

2024 arXiv
[13]

Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2024. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

2024
[14]

Taehyeon Kim, Joonkee Kim, Gihun Lee, and Se-Young Yun. 2024. Instructive decoding: Instruction-tuned large language models are self-refiner from noisy instructions. In The Twelfth International Conference on Learning Representations

2024
[15]

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

2024
[16]

Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36

2024
[17]

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097

2022 arXiv
[18]

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

2023 arXiv
[19]

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

2024
[20]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

2024
[21]

Qiang Liu, Xinlong Chen, Yue Ding, Shizhen Xu, Shu Wu, and Liang Wang. 2025. Attention-guided self-reflection for zero-shot hallucination detection in large language models. arXiv preprint arXiv:2501.09997

2025 arXiv
[22]

Junyu Luo, Cao Xiao, and Fenglong Ma. 2023. Zero-resource hallucination prevention for large language models. arXiv preprint arXiv:2309.02654

2023 arXiv
[23]

Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896

2023 arXiv
[24]

Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922

2023 arXiv
[25]

Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156

2018 arXiv
[26]

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525

2023 arXiv
[27]

Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. 2024. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289

2024 arXiv
[28]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[29]

Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987

2023 arXiv
[30]

Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023 a . Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397

2023 arXiv
[31]

Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. 2023 b . Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07257

2023 arXiv
[32]

Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. 2024. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715

2024 arXiv
[33]

Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. 2024. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint arXiv:2405.17820

2024 arXiv
[34]

Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. 2024. Logical closed loop: Uncovering object hallucinations in large vision-language models. arXiv preprint arXiv:2402.11622

2024 arXiv
[35]

Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2024. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105

2024
[36]

Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. 2024. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

2024
[37]

Hongbang Yuan, Pengfei Cao, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, and Jun Zhao. 2024. Whispers that shake foundations: Analyzing and mitigating false premise hallucinations in large language models. arXiv preprint arXiv:2402.19103

2024 arXiv
[38]

Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis-Philippe Morency, Katia Sycara, and Yaqi Xie. 2025. Self-correcting decoding with generative feedback for mitigating hallucinations in large vision-language models. arXiv pre...

2025 arXiv
[39]

Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411

2024 arXiv
[40]

Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754

2023 arXiv
[41]

Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. 2024. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402.18476

2024 arXiv
[42]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[43]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023 a . Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023

[2] [2]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023 b . Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023

[3] [3]

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930

arXiv 2024

[4] [4]

Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. 2024 a . Automated evaluation of large vision-language models on self-driving corner cases. arXiv preprint arXiv:2404.10595

arXiv 2024

[5] [5]

Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David F Fouhey, and Joyce Chai. 2024 b . Multi-object hallucination in vision-language models. arXiv preprint arXiv:2407.06192

arXiv 2024

[6] [6]

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883

arXiv 2023

[7] [7]

Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495

arXiv 2023

[8] [8]

Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14303--14312

2024