Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation

Bang Yang; Hongxiang Li; Jinghan Ru; Lexiang Tang; Lu Ma; Xianwei Zhuang; Yuexian Zou; Zhiyuan Hu

REVIEW 3 major objections 5 minor 1 cited by

Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VisFlow curbs visual hallucinations in LVLMs by reweighting attention at inference, lifting LLaVA-1.5 POPE F1 from 82.81 to 89.55.

desk verdict VisFlow's empirical gains are real, but the mechanism is under-specified and the TAI rule is internally contradictory; worth a serious referee but not acceptance as-is. read the letter →

arxiv 2506.12609 v2 pith:SAZZ3V7R submitted 2025-06-14 cs.CV

Lexiang Tang , Xianwei Zhuang , Bang Yang , Zhiyuan Hu , Hongxiang Li , Lu Ma , Jinghan Ru , Yuexian Zou This is my paper · ORCID

classification cs.CV

keywords visualhallucinationlargevision-languagemodelsattentioninterventiondecoding-timemitigationtraining-freetoken-levelhead-level

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VisFlow is a training-free, inference-time framework that reduces visual hallucinations in large vision-language models (LVLMs) by editing attention inside the decoder. Its central claim is that only a few visual tokens and attention heads are load-bearing for grounding: most visual attention goes to uninformative or positionally biased tokens, while certain heads over-focus on system prompts and the previously generated token. Token-level Attention Intervention (TAI) strengthens attention to reception-score-selected salient visual tokens and damps sink tokens, correcting a bias the paper attributes to Rotary Position Embedding. Head-level Attention Intervention (HAI) suppresses attention to system prompts and preceding text inside the heads that dominate those regions. On POPE, LLaVA-1.5 F1 rises from 82.81 (DoLa) or 81.54 (greedy) to 89.55; on CHAIR, sentence-level hallucination drops from 6.8 to 3.8, at near-greedy speed.

What carries the argument

The mechanism is a dual-level attention reweighting applied during decoding, each step followed by row renormalization so the edited weights remain a valid distribution. Token-level Attention Intervention (TAI) computes R(j), the total attention each visual token receives from other visual tokens across heads; tokens above a threshold τ are treated as salient and amplified by k>1, while near-maximum tokens are treated as sinks and attenuated by δ<1. Head-level Attention Intervention (HAI) classifies each head by its total attention to visual, system, and text tokens, then suppresses attention to system prompts by (1−αsys) and to previously generated text by (1−αtxt) only in the corresponding dominant head types. The paper's claim is that these two edits jointly correct a positional bias (RoPE) and a linguistic-prior bias, without retraining or extra forward passes.

What would settle it

A controlled run of VisFlow that applies the same k-scaling to randomly selected visual tokens, or the same α-suppression to randomly selected text/system heads, with all other settings identical: if hallucination scores fall as much as with the reception-score-selected tokens and heads, the paper's specific token/head identification is not the active ingredient. A second test shuffles the order of visual tokens before decoding; if the gains survive, the RoPE-bias correction is not the operative mechanism.

Watch

Extended reading notes

Core claim

The paper's discovery is that visual hallucination in LVLMs is driven by two correctable attention patterns, both readable from the model's own attention statistics. Token-level: due to Rotary Position Embedding, visual tokens nearer to the text receive disproportionate attention, while a few 'sink' tokens absorb attention without contributing semantic content, and a small set of 'salient' tokens carry the grounding information. Head-level: a sparse set of heads are visual-sensitive, while numerous system-dominant and text-dominant heads over-allocate attention to system prompts and previously generated tokens. Using an intra-visual reception score R(j) to pick out sink and salient tokens, and per-head visual/system/text attention statistics to pick out heads to suppress, VisFlow rescales the chosen attention weights (k for salient tokens, δ for sink tokens, 1−α for text and system tokens) and renormalizes each row. The edited attention yields more visually faithful outputs across POPE, CHAIR, and a GPT-4.1-aided evaluation, with head identities fixed once at prefill so decoding stays near greedy speed.

Load-bearing premise

The method works only if changing post-softmax attention weights—rather than something merely correlated with them—is what improves the outputs; the head and token selections are inferred from correlations between attention patterns and hallucination scores, not from a causal proof that those specific edits are the mechanism.

Editorial extensions

If this is right

Deployed LVLMs can be made more visually faithful without retraining or extra decoding passes; the only costs are a prefill-time head classification and per-token attention scaling.
The largest gains appear on the adversarial POPE subset, indicating the intervention is most effective precisely where language priors are misleading.
Masking visual-sensitive heads sharply increases hallucination while masking text heads in shallow layers reduces it, so the head taxonomy is itself an interpretability result independent of the mitigation.
The hyperparameters are benchmark-dependent (k=20 and δ=0.2 for POPE; k=10 and δ=0.4 for CHAIR) and TAI is omitted for compressed-token models, so VisFlow is a recipe to be tuned per model rather than a single fixed setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A stronger reading of the reception score is that the model's own intra-visual attention is a general-purpose relevance map; that map could be exported as an interpretability tool or as a filter for curating instruction-tuning data.
A test the paper does not run: apply the same k-scaling and α-suppression to randomly selected tokens and heads. If hallucination scores still drop, the active ingredient would be gross attention rebalancing rather than the specific salience and head taxonomy.
HAI and contrastive decoding target complementary priors—HAI suppresses attention to prompt text directly, while VCD subtracts a linguistically biased distribution—so a hybrid that stacks them may reduce hallucinations further than either alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

VisFlow is worth reading as an empirical possibility, but its core mechanistic claim is not yet supported. The paper reports striking gains on POPE and CHAIR—e.g., LLaVA-1.5 POPE F1 from ~82 to ~89.5, CHAIRs from 6.8 to 3.8—with a training-free attention intervention combining token-level and head-level edits. That is a real result if it holds up. The paper also does useful analysis: it distinguishes visual sink tokens from salient tokens and shows layer-wise attention patterns across models, which is more than most methods papers bother with.

The problems are in the attribution. The paper claims the specific saliency-based token and head selection reduces hallucinations, but there is no matched control isolating that selection. Table 4 masks heads rather than applying the paper's own scaling-and-renormalization, and Figure 9b compares against VAR's redistribution rule, not against a random or counter-salient token set. So the gains could come from a generic "attend more to visual, less to text" intervention, with the saliency machinery adding tuning knobs.

That suspicion is sharpened by the hyperparameters: k, delta, alpha_sys, alpha_txt, lambdas, and layer ranges are set separately per model and per benchmark. k=20, delta=20 on POPE; k=10, delta=0.4 on CHAIR. Figure 9a shows k=20 is chosen because it peaks on POPE. There are five random runs but no error bars or significance tests. This is tuning-to-benchmark risk, not fatal, but it means the numbers overstate generality.

Then there's the internal contradiction in TAI. Equation (6) defines δ as a decay factor with δ<1 to suppress sink tokens; the introduction says TAI amplifies attention to the sink token; the experiments set δ=20 for POPE. Those cannot all be true. Also, the sink and salient sets from Equation (5) can overlap, so a token might be multiplied by both k and δ. As written, the TAI rule is ambiguous and the causal attribution untestable.

On balance, this paper deserves a serious referee, but not acceptance as-is. The empirical direction is plausible, and the analysis is a genuine start. The method needs a contradiction-free specification and controls that isolate the proposed selection mechanisms—random token selection, uniform visual boosts—with fixed hyperparameters across benchmarks. Without that, it's a promising empirical note, not a demonstrated mechanism.

Recommendation: send to peer review, but expect major revision and re-analysis.

Referee Report

3 major / 5 minor

Summary. The paper proposes VisFlow, a training-free, inference-time framework that mitigates visual hallucinations in LVLMs by intervening directly in the attention mechanism. Token-level Attention Intervention (TAI) scales attention toward visual tokens identified as salient and away from visual sink tokens, while Head-level Attention Intervention (HAI) downscales attention from heads that over-attend to system prompts and previous text tokens. The method is evaluated on POPE, CHAIR, and a GPT-4.1-aided benchmark across LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2, reporting consistent improvements over greedy decoding and several decoding-time baselines, with only modest computational overhead. The paper also provides ablations on the two intervention components and on head masking.

Significance. If the causal story holds, VisFlow would be a useful and inexpensive addition to the hallucination-mitigation toolbox: it is training-free, requires no extra forward passes or external tools, and the reported gains on POPE and CHAIR are substantial, especially for LLaVA-1.5 (e.g., POPE Adversarial F1 from 78.36 with DoLa to 84.35, and CHAIR_s from 6.8 to 3.8). The paper also explicitly tests some functional claims about attention heads (Table 4), which is a strength. However, the central causal claim—that the specific saliency-based token/head selection, rather than a generic reweighting toward visual tokens, drives the improvement—is not yet established, and the method's hyperparameters are tuned per model and per benchmark. These concerns are substantial enough that the manuscript needs revision before the claims can be accepted.

major comments (3)

[Eq. (6) and §Experiments (Implementation)] The TAI rule is internally inconsistent as written. Eq. (6) states that δ < 1 is a decay factor for sink tokens, but the Introduction says 'TAI amplifies attention to the sink token on RoPE' and the experimental setup reports δ = 20 for POPE. With δ = 20, sink tokens are amplified rather than suppressed. Furthermore, I_salient and I_sink are both defined through Eq. (5) with different τ values, and the sets are not stated to be disjoint, so a visual token can be simultaneously multiplied by both k and δ. This ambiguity means the actual intervention implemented in the experiments is not uniquely specified, and the causal attribution of the results to the TAI rule cannot be tested as written.
[§Experiments (Implementation) and Fig. 9a] The hyperparameters are tuned separately for each model and each benchmark, and at least one of them is selected based on the test benchmark itself. The text reports k = 20 for POPE and k = 10, δ = 0.4 for CHAIR, with αtxt, αsys, λvis, λsys, λtxt, τ, and layer ranges also varied. Fig. 9a shows that k = 20 was chosen because it peaks on the POPE adversarial subset, which means the reported improvements partly reflect fitting to the evaluation data. No error bars or significance tests are reported despite the claims that results are averaged over five runs. The authors should report variance and either evaluate on held-out benchmarks or demonstrate that a single hyperparameter setting works across benchmarks.
[§More Analysis (Table 4, Fig. 9b)] The paper does not provide matched controls that isolate the causal role of the proposed saliency-based selection. Table 4 masks heads by zeroing them out, while HAI applies a (1−α) scaling followed by renormalization; these are different interventions, so the masking results do not directly validate the HAI rule. Fig. 9b compares TAI against V AR's redistribution rule, but it does not compare against an equally sized random token set or an inverse-saliency token set under the same renormalization and the same scaling factor k. Without such controls, the gains could be produced by a generic 'attend more to visual tokens and suppress text' intervention, leaving the claimed saliency-based mechanism unsupported.

minor comments (5)

[Eq. (1)] The saliency definition uses a task loss L(x) that is never specified, and it is unclear how gradients with respect to attention weights are obtained during decoding; please define L(x) and the exact computation procedure.
[§Experiments (Implementation)] The text states αtxt = 1 for LLaVA; since Eq. (12) multiplies attention by (1−αtxt), this value zeroes out attention to text tokens rather than suppressing it. Please clarify whether this is intended and how the model retains language coherence under such a strong intervention.
[§More Analysis, Table 4 paragraph] The sentence 'Table 4 and Figures 10a–10b further show POPE-based interventions' is misleading because Table 4 reports CHAIR results; only Figure 10 reports POPE-based interventions.
[Figure 1 caption] There is a grammatical error in the caption: 'We allocates stronger attention' should be 'We allocate stronger attention.'
[References] The GPT-4.1 evaluation is attributed to 'Chiang, Li et al. 2023b' in the references, but that entry appears to be the Vicuna technical report; the GPT-4.1 model should be cited from its original source.

Circularity Check

1 steps flagged · score 4.0 of 10

The POPE headline result is partly fitted to POPE: k=20 is selected by peaking F1 on POPE and then reported on POPE; other results remain independent, so circularity is partial.

fitted input called prediction [Experiments, Implementation and Experimental Setup; Figure 9a; Table 1]
"For the POPE dataset, we use k = 20 and δ = 20. ... As shown in Figure 9a, F1 Score peaks at k = 20, while both smaller ( k = 1) and larger ( k = 30) values degrade performance. Figure 9: Ablation study of TAI on POPE (adversarial). (a) Evaluates performance sensitivity to the enhancement factor k."

The POPE row of Table 1 is presented as evidence that VisFlow mitigates hallucination, but k=20 is chosen because it maximizes F1 on POPE (adversarial), the same benchmark and even the same setting reported in Table 1. The intervention is therefore tuned to the test metric before the metric is quoted as a result; the POPE gain is not an independent prediction of the method. This does not force the CHAIR results, since k=10 is used there, but it makes the main POPE comparison partly a selection artifact.

full rationale

No formal derivation in the paper reduces to its inputs by definition. The token/head selection rules (Eqs. 4-13) are explicit attention-based heuristics, not identities, and the ablations (Tables 3-4) provide some independent evidence that token and head selection matters. The main circularity is the fitted-input pattern: k=20 was selected by peaking F1 on POPE and then used in the reported POPE results; the same is true for the per-benchmark δ values. Because CHAIR uses a separately chosen k=10 and the method is evaluated on external benchmarks without reusing a fitted parameter as a prediction, the score is 4 rather than 6+. Separately, a correctness risk that is not circularity: Eq. (6) defines δ<1 as a sink-token decay, while the experiments state δ=20 for POPE, and I_sink can overlap I_salient under Eq. (5); this ambiguity makes the tested TAI rule under-specified, but it does not by itself make the argument circular.

Assumptions & free parameters 9 free parameters · 4 assumptions · 1 invented entities

Free parameters dominate the ledger: k, delta, alpha_txt, alpha_sys, lambda_vis, lambda_sys, lambda_txt, tau, and layer ranges are hand-selected per model and benchmark. The axioms are domain assumptions about attention causing hallucination and about the semantic meaning of reception-score-selected tokens. The only invented conceptual entity is the visual sink/salient token class, with no independent evidence outside the paper's own benchmark tuning.

free parameters (9)

k (salient token scaling factor) = 20 (POPE), 10 (CHAIR)
Scaling factor for salient visual token attention in Eq. (6); chosen per benchmark, with ablation showing peak at k=20 on POPE.
delta (sink token factor) = 20 (POPE), 0.4 (CHAIR)
Decay factor for sink tokens in Eq. (6), defined as delta<1 yet set to 20 on POPE; per-benchmark and internally inconsistent.
alpha_txt (text-head suppression coefficient) = 1.0 (LLaVA-1.5), 0.6 (MiniGPT-4/mPLUG-Owl2)
Suppression coefficient for text-dominant heads in Eq. (12); tuned per model.
alpha_sys (system-head suppression coefficient) = 0.6 (LLaVA-1.5), 0.4 (MiniGPT-4/mPLUG-Owl2)
Suppression coefficient for system-dominant heads in Eq. (12); tuned per model.
lambda_vis (visual head sensitivity threshold) = 1
Threshold for visual-sensitive head selection in Eq. (9).
lambda_sys (system head threshold) = 0.8
Threshold for system-dominant head selection in Eq. (11).
lambda_txt (text head threshold) = 0.3
Threshold for text-dominant head selection in Eq. (11).
tau (token selection threshold) = not reported (examples 1/20 and 1/2 in text)
Fraction of the maximum reception score used to select salient vs sink tokens in Eq. (5); the actual value used in experiments is not given.
layer application ranges = TAI: layer>=2; HAI text heads: layers 0-7; HAI system heads: all layers
Hand-selected design choices that affect which layers receive modified attention; not derived from analysis.

assumptions (4)

domain assumption Visual hallucination is causally driven by insufficient visual attention and over-reliance on language priors, so editing attention directly at inference changes hallucination behavior.
Introduced in the Introduction and Fig. 2; the paper provides correlational evidence but no causal proof.
domain assumption Visual tokens with high intra-visual reception score R(j) are the semantically salient tokens for grounding, while low-score sink tokens are unimportant.
Used to build TAI in Eqs. (4)-(6); validated only through visualization and ablation on the same benchmarks used for tuning.
domain assumption Renormalizing scaled attention weights preserves model coherence for scaling factors as large as k=20 and for suppression coefficients up to alpha_txt=1.
The method relies on this to keep generation fluent; no theoretical bound or stability analysis is given.
domain assumption RoPE creates a positional bias where visual tokens closer to text receive too much attention, and this bias contributes to hallucination.
Stated in the Motivation and used to justify TAI; supported by attention maps, not by a controlled experiment isolating RoPE.

invented entities (1)

Visual sink token and visual salient token classes
purpose: Token-level categories TAI uses to decide which visual tokens to amplify (salient) or decay (sink) in Eq. (6).
These are internal analysis categories introduced to justify attention scaling. They are validated through visualizations and ablations on the same benchmarks used for hyperparameter selection, not through an external falsifiable prediction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation." pith.science (2026). https://pith.science/paper/SAZZ3V7R

@misc{pith2026250612609,
  author       = {Pith},
  title        = {Pith review of: Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SAZZ3V7R}},
  note         = {Machine review of arXiv:2506.12609}
}

read the original abstract

Large vision-language models (LVLMs) have demonstrated impressive capabilities across diverse multimodal tasks, yet they remain highly susceptible to visual hallucinations (VH), often producing confident but inaccurate descriptions of visual content. Building on the insight that not all tokens and attention heads contribute equally to VH mitigation, we introduce VisFlow, a lightweight and training-free framework that alleviates hallucinations by directly modulating attention patterns during inference. To address two primary challenges of VH, namely insufficient visual attention and the dominance of language priors, we identify three problematic attention behaviors in LVLMs: (1) disproportionate allocation of attention to uninformative or trailing visual tokens, (2) over-dependence on the previously generated token, and (3) excessive fixation on system prompts that hinders multimodal integration. To overcome these issues, VisFlow introduces a dual-level Attention Intervention, consisting of Token-level Attention Intervention (TAI), which reinforces attention to salient visual regions, and Head-level Attention Intervention (HAI), which suppresses undue focus on system prompts and adjacent text tokens. Together, these interventions strengthen visual alignment while reducing linguistic bias. Extensive experiments across diverse models and benchmarks demonstrate that VisFlow effectively mitigates hallucinations with minimal computational overhead.

Figures

Figures reproduced from arXiv: 2506.12609 by the authors.

**Figure 1.** Illustration of token-level attention distribution in VisFlow. We allocates stronger attention to visual salient token [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Overview of visual hallucination causes and our solution. (a) Linguistic Over-reliance: hallucinations caused by [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Layer-wise token attention and information flow [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (14 more)

**Figure 4.** Figure 4: Visualization of intra-visual reception score [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Visualization of visual attention in LLaVA-1.5-7B. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Layer-wise heatmaps of attention weights across all heads, showing preference for visual, text, and system tokens. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: Attention maps from representative heads across [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

**Figure 8.** Figure 8: Comparison on the MSCOCO dataset using GPT [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

**Figure 10.** Figure 10: Ablation study of HAI on POPE. (a) Evaluates [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]

**Figure 11.** Figure 11: Comparison of our method with different base [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]

**Figure 14.** Figure 14: Layer-wise attention weights to visual tokens in [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]

**Figure 12.** Figure 12: Layer-wise attention weights to visual tokens in [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]

**Figure 13.** Figure 13: Layer-wise attention weights to visual tokens in [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]

**Figure 15.** Figure 15: Layer-wise heatmaps of attention weights across all heads in MiniGPT-4 [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]

**Figure 16.** Figure 16: Comparison of visual attention distributions be [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]

**Figure 17.** Figure 17: Visualization of visual attention in LLaVA-1.5- [PITH_FULL_IMAGE:figures/full_fig_p013_17.png]

**Figure 18.** Figure 18: More cases on the CHAIR benchmark with Lmax = 128. Hallucinated content is highlighted in red [PITH_FULL_IMAGE:figures/full_fig_p014_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs
cs.CV 2026-07 conditional novelty 6.0 of 10

C-PTQ weights quantization error by per-channel Fisher information of the task loss, improving low-bit accuracy of multimodal LLMs by small margins over existing channel-wise scaling methods.

Reference graph

Works this paper leans on

57 extracted references · 12 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

An, W.; Tian, F.; Leng, S.; Nie, J.; Lin, H.; Wang, Q.; Chen, P.; Zhang, X.; and Lu, S. 2025. Mitigating object hallucinations in large vision-language models with assembly of global and local attention. In Proceedings of the Computer Vision and Pattern Recognition Conference, 29915--29926

2025
[4]

An, W.; Tian, F.; Leng, S.; Nie, J.; Lin, H.; Wang, Q.; Dai, G.; Chen, P.; and Lu, S. 2024. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint arXiv:2406.12718

arXiv 2024
[5]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 1(2): 3

arXiv 2023
[6]

Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

arXiv 2025
[7]

Chen, C.; Liu, M.; Jing, C.; Zhou, Y.; Rao, F.; Chen, H.; Zhang, B.; and Shen, C. 2025. PerturboLLaVA: Reducing multimodal hallucinations with perturbative visual training. arXiv preprint arXiv:2503.06486

arXiv 2025
[8]

Chen, J.; Zhu, D.; Shen, X.; Li, X.; Liu, Z.; Zhang, P.; Krishnamoorthi, R.; Chandra, V.; Xiong, Y.; and Elhoseiny, M. 2023 a . Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478

arXiv 2023

Show all 57 references

[9]

Chen, K.; Zhang, Z.; Zeng, W.; Zhang, R.; Zhu, F.; and Zhao, R. 2023 b . Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic. ArXiv, abs/2306.15195

2023 arXiv
[10]

Chen, Z.; Wang, W.; Cao, Y.; Liu, Y.; Gao, Z.; Cui, E.; Zhu, J.; Ye, S.; Tian, H.; Liu, Z.; et al. 2024 a . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

2024 arXiv
[11]

Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and patte...

2024
[12]

Chen, Z.; Zhao, Z.; Luo, H.; Yao, H.; Li, B.; and Zhou, J. 2024 c . Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425

2024 arXiv
[13]

Chiang, W.-L.; Li, Z.; et al. 2023 a . Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\ https://vicuna.lmsys.org. Accessed: April 14, 2023

2023
[14]

Chiang, W.-L.; Li, Z.; et al. 2023 b . Vicuna : An open-source chatbot impressing GPT-4 with 90\ https://vicuna.lmsys.org. Accessed: 2023-04-14

2023
[15]

Chuang, Y.-S.; Xie, Y.; Luo, H.; Kim, Y.; Glass, J.; and He, P. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883

2023 arXiv
[16]

Darcet, T.; Oquab, M.; Mairal, J.; and Bojanowski, P. 2023. Vision transformers need registers. arXiv preprint arXiv:2309.16588

2023 arXiv
[17]

Fan, S.; Xie, L.; Shen, C.; Teng, G.; Yuan, X.; Zhang, X.; Huang, C.; Wang, W.; He, X.; and Ye, J. 2025. Improving complex reasoning with dynamic prompt corruption: A soft prompt optimization approach. arXiv preprint arXiv:2503.13208

2025 arXiv
[18]

Gunjal, A.; Yin, J.; and Bas, E. 2024. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 18135--18143

2024
[19]

Huang, Q.; Dong, X.; Zhang, P.; Wang, B.; He, C.; Wang, J.; Lin, D.; Zhang, W.; and Yu, N. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vis...

2024
[20]

Huo, F.; Xu, W.; Zhang, Z.; Wang, H.; Chen, Z.; and Zhao, P. 2024. Self-introspective decoding: Alleviating hallucinations for large vision-language models. arXiv preprint arXiv:2408.02032

2024 arXiv
[21]

Jiang, C.; Xu, H.; Dong, M.; Chen, J.; Ye, W.; Yan, M.; Ye, Q.; Zhang, J.; Huang, F.; and Zhang, S. 2024. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 270...

2024
[22]

Kang, S.; Kim, J.; Kim, J.; and Hwang, S. J. 2025. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.03321

2025 arXiv
[23]

Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13872--13882

2024
[24]

Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023 a . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

2023
[25]

X.; and Wen, J.-R

Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, W. X.; and Wen, J.-R. 2023 b . Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

2023 arXiv
[26]

Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 1...

2014
[27]

Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2024 a . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26296--26306

2024
[28]

Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. Advances in neural information processing systems, 36: 34892--34916

2023
[29]

Liu, H.; Xue, W.; Chen, Y.; Chen, D.; Zhao, X.; Wang, K.; Hou, L.; Li, R.; and Peng, W. 2024 b . A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253

2024 arXiv
[30]

Liu, S.; Ye, H.; Xing, L.; and Zou, J. 2024 c . Reducing hallucinations in vision-language models via latent space steering. arXiv preprint arXiv:2410.15778

2024 arXiv
[31]

Liu, S.; Zheng, K.; and Chen, W. 2024. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In European Conference on Computer Vision, 125--140. Springer

2024
[32]

Manakul, P.; Liusie, A.; and Gales, M. J. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896

2023 arXiv
[33]

Michel, P.; Levy, O.; and Neubig, G. 2019. Are sixteen heads really better than one? Advances in neural information processing systems, 32

2019
[34]

Park, Y.; Lee, D.; Choe, J.; and Chang, B. 2025. Convis: Contrastive decoding with hallucination visualization for mitigating hallucinations in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 6434--6442

2025
[35]

A.; Burns, K.; Darrell, T.; and Saenko, K

Rohrbach, A.; Hendricks, L. A.; Burns, K.; Darrell, T.; and Saenko, K. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156

2018 arXiv
[36]

Ru, J.; Xie, Y.; Zhuang, X.; Yin, Y.; and Zou, Y. 2025. Do we really have to filter out random noise in pre-training data for language models? arXiv preprint arXiv:2502.06604

2025 arXiv
[37]

\"O .; and Pfister, T

Sarkar, P.; Ebrahimi, S.; Etemad, A.; Beirami, A.; Ar k, S. \"O .; and Pfister, T. 2024. Data-augmented phrase-level alignment for mitigating object hallucination. arXiv preprint arXiv:2405.18654

2024 arXiv
[38]

Simonyan, K.; Vedaldi, A.; and Zisserman, A. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034

2013 arXiv
[39]

Su, J.; Ahmed, M.; Lu, Y.; Pan, S.; Bo, W.; and Liu, Y. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063

2024
[40]

Sun, Z.; Zang, X.; Zheng, K.; Song, Y.; Xu, J.; Zhang, X.; Yu, W.; and Li, H. 2024. Redeep: Detecting hallucination in retrieval-augmented generation via mechanistic interpretability. arXiv preprint arXiv:2410.11414

2024
[41]

Tang, F.; Huang, Z.; Liu, C.; Sun, Q.; Yang, H.; and Lim, S.-N. 2025. Intervening anchor token: Decoding strategy in alleviating hallucinations for MLLMs. In The Thirteenth International Conference on Learning Representations

2025
[42]

Team, Q. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

2024 arXiv
[43]

Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[44]

Wang, L.; Li, L.; Dai, D.; Chen, D.; Zhou, H.; Meng, F.; Zhou, J.; and Sun, X. 2023. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint arXiv:2305.14160

2023 arXiv
[45]

Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. 2024 a . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

2024 arXiv
[46]

Wang, X.; Pan, J.; Ding, L.; and Biemann, C. 2024 b . Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715

2024 arXiv
[47]

Xing, Y.; Li, Y.; Laptev, I.; and Lu, S. 2024. Mitigating object hallucination via concentric causal attention. Advances in Neural Information Processing Systems, 37: 92012--92035

2024
[48]

Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025 a . Qwen3 technical report. arXiv preprint arXiv:2505.09388

2025 arXiv
[49]

Yang, T.; Li, Z.; Cao, J.; and Xu, C. 2025 b . Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention. In International Conference on Learning Representations (ICLR). Poster

2025
[50]

Yang, Z.; Luo, X.; Han, D.; Xu, Y.; and Li, D. 2025 c . Mitigating hallucinations in large vision-language models via dpo: On-policy data hold the key. In Proceedings of the Computer Vision and Pattern Recognition Conference, 10610--10620

2025
[51]

Ye, Q.; Xu, H.; Ye, J.; Yan, M.; Hu, A.; Liu, H.; Qian, Q.; Zhang, J.; and Huang, F. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, 13040--13051

2024
[52]

Yin, H.; Si, G.; and Wang, Z. 2025. ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large Language Models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 14625--14634

2025
[53]

Yin, S.; Fu, C.; Zhao, S.; Xu, T.; Wang, H.; Sui, D.; Shen, Y.; Li, K.; Sun, X.; and Chen, E. 2024. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12): 220105

2024
[54]

Zhang, Z.; Yadav, S.; Han, F.; and Shutova, E. 2024. Cross-modal Information Flow in Multimodal Large Language Models. arXiv preprint arXiv:2411.18620

2024 arXiv
[55]

Zhuang, X.; Xie, Y.; Deng, Y.; Liang, L.; Ru, J.; Yin, Y.; and Zou, Y. 2025 a . VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model. arXiv preprint arXiv:2501.12327

2025 arXiv
[56]

Zhuang, X.; Zhu, Z.; Xie, Y.; Liang, L.; and Zou, Y. 2025 b . VASparse: Towards Efficient Visual Hallucination Mitigation for Large Vision-Language Model via Visual-Aware Sparsification. arXiv preprint arXiv:2501.06553

2025 arXiv
[57]

Zou, X.; Wang, Y.; Yan, Y.; Lyu, Y.; Zheng, K.; Huang, S.; Chen, J.; Jiang, P.; Liu, J.; Tang, C.; et al. 2024. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. arXiv preprint arXiv:2410.03577

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

An, W.; Tian, F.; Leng, S.; Nie, J.; Lin, H.; Wang, Q.; Chen, P.; Zhang, X.; and Lu, S. 2025. Mitigating object hallucinations in large vision-language models with assembly of global and local attention. In Proceedings of the Computer Vision and Pattern Recognition Conference, 29915--29926

2025

[4] [4]

An, W.; Tian, F.; Leng, S.; Nie, J.; Lin, H.; Wang, Q.; Dai, G.; Chen, P.; and Lu, S. 2024. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint arXiv:2406.12718

arXiv 2024

[5] [5]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 1(2): 3

arXiv 2023

[6] [6]

Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

arXiv 2025

[7] [7]

Chen, C.; Liu, M.; Jing, C.; Zhou, Y.; Rao, F.; Chen, H.; Zhang, B.; and Shen, C. 2025. PerturboLLaVA: Reducing multimodal hallucinations with perturbative visual training. arXiv preprint arXiv:2503.06486

arXiv 2025

[8] [8]

Chen, J.; Zhu, D.; Shen, X.; Li, X.; Liu, Z.; Zhang, P.; Krishnamoorthi, R.; Chandra, V.; Xiong, Y.; and Elhoseiny, M. 2023 a . Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478

arXiv 2023