Pith. sign in

REVIEW 4 major objections 6 minor 42 references

Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Object hallucinations in multimodal LLMs are reduced by extracting the logit difference at the intermediate layer where image-aware and text-only distributions diverge most and adding it to the final logits.

desk verdict Useful contrastive-decoding variant with clear writing, but the text-only subtraction at its core is confounded by input-structure mismatch and the empirical validation lacks significance testing. read the letter →

arxiv 2507.15652 v1 pith:UAP2TLLQ submitted 2025-07-21 cs.CV

Haoran Zhou , Zihan Zhang , Hao Chen This is my paper
classification cs.CV
keywords multimodallargelanguagemodelsobjecthallucinationdecodingstrategyintermediatelayersJensen-Shannondivergencevisualfactualknowledgetraining-freecontrastive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that in a multimodal large language model, the intermediate layer where the next-token probability distribution diverges most between the original image-text input and a text-only input is the layer richest in visual factual knowledge. It introduces EVA (Decoding by Extracting Visual Facts), a training-free decoding rule that selects that layer, subtracts the text-only logits from the image-text logits to isolate visual facts, and adds a scaled version of that difference to the final-layer logits. On POPE, MME, and CHAIR benchmarks across InstructBLIP, MiniGPT-4, LLaVA-1.5, and Qwen-VL under greedy, beam, and nucleus decoding, the method reports consistently lower hallucination rates than vanilla decoding and than DoLa, DeCo, OPERA, and VCD. If correct, EVA offers a general plug-in correction for object hallucination that needs no retraining and no image perturbation.

What carries the argument

Dynamic Target Layer Selection: early-exit heads at candidate intermediate layers produce next-token distributions from both the original multimodal input and a pure-text input; the paper computes the Jensen-Shannon divergence between these two distributions over a top-p-truncated candidate vocabulary and selects the layer M with the maximum divergence. Visual Fact Extraction and Logit Correction: the visual fact signal is logits_v = $\varphi$(h^M) - $\varphi$(hat h^M), the difference between image-aware and text-only logits at the selected layer, and the final logits become $\varphi$(h^N) + $\alpha$ times max_prob times ($\varphi$(h^M) + max_JSD times logits_v). The coefficients max_prob and max_JSD provide dynamic soft modulation, so the injected visual knowledge is scaled by how confident and how visually divergent the selected layer is, while $\alpha$ controls the overall injection strength.

What would settle it

Measure, on a large set of image-caption pairs, the correlation between per-layer JS divergence (image-aware vs text-only) and per-layer probability of the ground-truth next token; if the layer of maximum JSD is not also a layer where the ground-truth token probability is at or near its peak, the selection hypothesis fails. Alternatively, run EVA on images whose correct object is strongly predictable from the caption prefix alone and check whether accuracy on those tokens drops relative to vanilla decoding.

Watch

Extended reading notes

Core claim

EVA's central claim is hierarchical: visual facts are not lost in a multimodal LLM but are suppressed by language-prior distributions in deeper layers, and at some intermediate layer the gap between the original distribution and the pure-text prior distribution is widest. The paper identifies that layer by Jensen-Shannon divergence and treats the difference between the original layer logits and the text-only layer logits as the visual factual knowledge to inject. The final prediction is corrected as logits = final_logits + alpha times max_prob times (target_layer_logits + max_JSD times logits_v), where logits_v equals target_layer_logits minus prior_logits, and max_prob and max_JSD are dynamic weights taken from the selected layer. The paper reports that this correction consistently improves F1 on POPE, total scores on the MME hallucination subset, and CHAIR scores across four 7B multimodal LLMs and three decoding strategies, with the largest POPE gain reaching 21.9 points under nucleus sampling.

Load-bearing premise

The correction assumes that running the model on text-only input at the selected layer produces a clean language prior, so that subtracting it from the image-aware logits leaves exactly the visual facts; if that prior already contains or supports the correct object, the subtraction can erase real visual evidence.

Editorial extensions

If this is right

  • EVA can be applied on top of any autoregressive multimodal LLM without retraining or additional data, because it only rewrites the next-token logits during generation.
  • It composes with greedy search, nucleus sampling, and beam search, so existing deployment pipelines do not need to change their sampling logic.
  • Because the selected layer is chosen per token, the correction adapts to each generation step rather than applying one fixed layer globally.
  • Reductions in hallucination appear in both probing-style VQA tasks (POPE and the MME hallucination subset) and open-ended captioning (CHAIR), indicating the effect is not limited to a single task format.
  • The largest reported improvement, 21.9 F1 points on POPE under nucleus sampling for MiniGPT-4, suggests the method is especially helpful when stochastic sampling introduces more hallucination risk.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper does not run is to compare the argmax-JSD layer with the layer where the ground-truth next token has peak probability; if these consistently disagree, the selection rule may be picking prior-suppression artifacts rather than visual facts.
  • Because the same contrast is computed over the whole token distribution, EVA could plausibly be extended beyond object hallucination to attribute, relation, or counting errors without changing its formula.
  • The method's practical cost is the two forward passes per step and the alpha tuning range from 0.6 to 5; a per-token adaptive schedule for alpha could remove the grid search and make the method more deployment-ready.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes EVA, a training-free decoding method for mitigating object hallucination in multimodal large language models (MLLMs). It selects an intermediate layer by maximizing the JS divergence between the probability distributions obtained from the original multimodal input and a pure-text input, treats the difference of logits at that layer as "visual factual knowledge," and adds a scaled version to the final-layer logits. The method is evaluated on POPE, MME, and CHAIR across InstructBLIP, MiniGPT-4, LLaVA-1.5, and Qwen-VL under greedy, beam, and nucleus decoding, reporting consistent hallucination reductions, along with ablations and case studies.

Significance. If the central assumption holds, EVA offers an inexpensive, model-agnostic decoding plug-in that improves hallucination metrics across several MLLMs and decoding strategies. The paper includes a reasonable set of baselines and three standard benchmarks, and its ablations support the importance of the dynamic modulation coefficients. The main weakness is that the validity of the text-only contrast as a source of visual facts is not established: the subtraction in Eq. (9) is confounded by positional and attention-structure differences, and the empirical evaluation lacks statistical rigor. Nevertheless, the method is simple, well-described, and likely reproducible, which makes the claims worth investigating further.

major comments (4)
  1. [Section 3.3.2, Eq. (9)] The text-only forward pass removes the image-token sequence, which in standard MLLMs is prepended to the text. This shifts the positional IDs of every text token and removes image-related attention context from all earlier layers. Consequently, logits_v = phi(h^M_{T-1}) - phi(hat h^M_{T-1}) does not isolate visual factual knowledge; it also captures positional-embedding differences and altered attention traces. Since Eq. (8) directly scales logits_v into the final prediction, this confound propagates into all reported gains. Please validate the clean-prior assumption, for example by comparing against a position-preserving text-only condition where image tokens are replaced by neutral embeddings, or by showing that the gains are robust when the positional mismatch is explicitly controlled.
  2. [Section 4.1 and Tables 1–3] The hyperparameter alpha is described as "tuned within the range of 0.6 to 5," but the paper does not state whether tuning was performed on the evaluation benchmarks themselves, on a held-out validation set, or separately per model and decoding strategy. This creates a risk that the reported improvements reflect selection on the test set. In addition, none of the tables report variance or significance tests; several improvements are small (e.g., Qwen-VL POPE gains of 0.9–1.1 points in Table 1), so the claim that EVA "significantly outperforms" baselines is not supported. Please report standard deviations or confidence intervals, or at least multiple runs with different random seeds for stochastic decoding.
  3. [Section 3.2 and Table 4] The key hypothesis—that the maximum-JSD layer contains the richest visual factual knowledge—is supported only by the anecdotal curves in Figure 2 and by the token-counting experiment in Table 4. Table 4 reports raw counts without statistical testing and uses the heuristic activation criterion in Eq. (11), whose relation to final object-level accuracy is not established. A more direct evaluation would measure the precision and recall of the extracted logits_v against ground-truth object annotations, or ablate the layer-selection rule against fixed layers, to confirm that the maximum-JSD choice is genuinely better than alternatives.
  4. [Section 4.3, Table 3 and footnote] The CHAIR improvements for Qwen-VL are very large (e.g., CHAIRS from 46.0 to 6.8 under greedy), and the footnote states that Qwen-VL may generate relatively short sentences. If EVA systematically shortens captions, CHAIRS and CHAIRI can drop mechanically because fewer objects are mentioned. Please report average caption length for each method, and consider evaluating CHAIR at matched length or reporting object recall alongside hallucination rates to disentangle hallucination reduction from output-length effects.
minor comments (6)
  1. [Throughout] The method name is formatted inconsistently as "EV A" (with a space) in many places, including tables and captions; please use consistent formatting, e.g., "EVA".
  2. [Figure 1] The equation text inside Figure 1 is garbled (e.g., "1() N Thφ −" and "max_ probs"), making the proposed formula hard to read; please fix the rendering.
  3. [Section 3.3.1, Eq. (2)] The candidate-token definition is unclear: the set Vp is not previously defined, and the summation and inequality notation should be made precise (e.g., define Vp as the top-p probability mass set and state how the condition is evaluated).
  4. [Section 4.4, Eq. (11) and Table 4] The condition p(xa|x<s) - p(xh|x<s) > 0 in Eq. (11) compares probabilities of different tokens; please clarify the intended comparison, for instance whether xa is compared with the highest-probability hallucinated token within the candidate set.
  5. [Figure 4 and Appendix B] The "Ours" blocks are duplicated verbatim for the same prompt in several examples (e.g., the first and second case in Figure 4, and multiple figures in Appendix B); please deduplicate.
  6. [Appendix captions] The captions for Figures 5–8 contain the typo "decoing methods"; this should be "decoding methods".

Circularity Check

3 steps flagged · score 4.0 of 10

EVA's "visual factual knowledge" is the contrast signal it injects (Eq. 9), and the alpha gains are tuned on the same benchmarks; the JSD-based layer selection retains independent empirical support.

  1. self definitional [Section 3.3.2, Eqs. (8)-(9); Section 3.2 hypothesis]
    "To extract visual factual knowledge, we subtract prior logits from the original logits at the target layer as visual factual knowledge. Subsequently, it is integrated into the original logits of the target layer for soft modulation, and the modulated logits are then proportionally merged into the final layer."

    The paper defines "visual factual knowledge" as the quantity logits_v = phi(h^M) - phi(hat h^M) in Eq. 9, then "extracts" exactly this quantity and adds it to the final logits in Eq. 8. The Section 3.2 hypothesis that higher-JSD layers harbor richer visual factual knowledge is therefore not a derived prediction: the measure of richness (JSD) and the extracted signal (logits_v) are the same original-vs-prior contrast by construction. Benchmark gains can test the heuristic, but the paper's framing presents the definition itself as evidence for the hypothesis.

  2. fitted input called prediction [Section 4.1 Implementation Details; results in Tables 1-3]
    "For the image captioning and VQA tasks, the hyperparameter α is tuned within the range of 0.6 to 5."

    The hyperparameter α directly scales the visual-fact correction in Eq. 8, and the paper reports hallucination reductions on POPE, MME, and CHAIR without describing a separate validation split. Tuning α on those same benchmarks makes the reported improvements partly an optimization result rather than an out-of-sample prediction. Because the paper does not report the chosen α per model and benchmark, the magnitude of the claimed gains cannot be separated from the fitting procedure.

1 more flagged steps
  1. self citation load bearing [Section 3.3.1 (candidate tokens) and Section 4.1 (candidate layers)]
    "We adopt the truncation strategy proposed in [25] to select candidate tokens, with the default strategy being top-p truncation. ... we select layers 20–28 as candidates for the preceding layers, following [25]."

    Reference [25] shares author H. Chen with the present paper, and the truncation strategy plus the 20–28 candidate-layer window are imported from that prior work as fixed design choices. These choices are load-bearing in every experiment because the JSD-based layer selection in Eq. 4 operates only inside this window. The central JSD contrast is new and is evaluated on external benchmarks, so this is partial rather than a full reduction to the self-citation.

full rationale

The core derivation in Eqs. (7)-(9) is not a formal derivation recycled as a prediction; it is an explicit decoding rule. The benchmark improvements on POPE, MME, and CHAIR, together with the ablation in Table 5, give the method independent empirical content, so this is not a case of the result being forced by definition alone. The main circularity burden is two-fold: the quantity called "visual factual knowledge" is defined as the same original-minus-prior contrast that the decoder injects, making the Section 3.2 "richness" claim a restatement of the construction; and α is tuned on the same benchmarks used to report gains, so the headline improvements are partly fitted. A third, weaker issue is the import of the candidate-layer window and truncation from the authors' prior work [25], which narrows where the JSD selection can operate. The text-only forward pass also changes positional IDs and attention context, so Eq. 9 may not isolate visual facts; this is a correctness or confound concern rather than a circularity and was not scored as a circular step. Overall score 4: some self-citation and benchmark-level fitting, but the JSD-based layer selection and external evaluations retain independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The ledger shows that EVA is not a first-principles derivation. It depends on the text-only-prior assumption, the top-p candidate restriction, and a hand-tuned alpha. The central JSD-max selection rule is an ad hoc hypothesis that is plausible but not formally established. No new physical entities or formal constructs are introduced.

free parameters (4)
  • alpha (proportion of extracted visual knowledge) = tuned in [0.6, 5], exact per-benchmark values not reported
    Controls the weight of the contrastive visual logits added to the final layer in Eq. 8. Tuning it on the same kinds of benchmarks used for reporting means the reported scores inherit tuned values.
  • Candidate layer window J = layers 20-28 for 7B/32-layer models
    Adopted from DeCo [25] without per-model or per-task justification; the method's behavior may depend on this hand-picked window.
  • top-p truncation threshold p = not reported
    Equation (2) restricts candidate tokens using top-p truncation, but the paper never states the p value used in the experiments.
  • Confidence threshold for token-count analysis = 0.9
    Used in Section 4.4 to filter data for the comparison in Table 4; this affects the supporting evidence for the visual-fact extraction claim.
assumptions (5)
  • domain assumption Ground-truth next tokens usually appear among the top-ranked tokens of the final-layer output distribution.
    Used to justify restricting candidate tokens to a top-p subset in Eq. 2; borrowed from DeCo [25].
  • domain assumption A text-only forward pass through the same layers gives a clean estimate of the language-model prior.
    The contrast in Eq. 9 subtracts text-only logits from original logits; if the text-only distribution already contains visual-fact-adjacent context, the subtraction removes correct information.
  • ad hoc to paper The layer with maximum JS divergence between original and text-only distributions contains the most visual factual knowledge.
    This is the central selection rule in Eq. 4, supported only by the running example in Figure 2 and the small token-count comparison in Table 4, not by a systematic mechanistic study.
  • domain assumption Early-exit hidden states can be used to form meaningful next-token probability distributions.
    The method builds on the early-exit literature [27, 28, 29] and assumes intermediate-layer classifiers are reliable enough for contrastive decoding.
  • domain assumption POPE, MME, and CHAIR metrics adequately measure object hallucination.
    The headline conclusion is entirely about these public benchmarks, and no independent measurement of output quality or length-adjusted caption quality is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/UAP2TLLQ

@misc{pith2026250715652,
  author       = {Pith},
  title        = {Pith review of: Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UAP2TLLQ}},
  note         = {Machine review of arXiv:2507.15652}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have made significant strides by combining visual recognition and language understanding to generate content that is both coherent and contextually accurate. However, MLLMs continue to struggle with object hallucinations, where models produce seemingly plausible but factually incorrect outputs, including objects that do not exist in the image. Recent work has revealed that the prior knowledge in MLLMs significantly suppresses visual information in deep layers, causing hallucinatory outputs. However, how these priors suppress visual information at the intermediate layer stage in MLLMs remains unclear. We observe that visual factual knowledge and the differences between intermediate-layer prior/original probability distributions show similar evolutionary trends in intermediate layers. Motivated by this, we introduce Decoding by Extracting Visual Facts (EVA), a simple, training-free method that dynamically selects intermediate layers with the most significant visual factual information. By contrasting the output distributions of the selected layer derived from the original input and pure-text input, EVA extracts visual factual knowledge and proportionally incorporates it into the final layer to correct the output logits. Importantly, EVA is model-agnostic, seamlessly integrates with various classic decoding strategies, and is applicable across different MLLMs. We validate EVA on widely-used benchmarks, and the results show that it significantly reduces hallucination rates compared to baseline methods, underscoring its effectiveness in mitigating hallucinations.

Figures

Figures reproduced from arXiv: 2507.15652 by the authors.

Figure 1
Figure 1. Framework of EVA. (a) Dynamically select an optimal target layer from intermediate layers. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the hierarchical evolution trends of hallucinated tokens ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. MME full set results on LLaVA-1.5 with nucleus sampling. EVA leads to substantial [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustration of hallucination correction by EVA with two samples from LLaVA-Bench. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: EVA’s performance in mitigating hallucinations of InstructBLIP on three decoing methods. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: EVA’s performance in mitigating hallucinations of MiniGPT-4 on three decoing methods. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: EVA’s performance in mitigating hallucinations of LLaVA-1.5 on three decoing methods. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: EVA’s performance in mitigating hallucinations of Qwen-VL on three decoing methods. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 12 canonical work pages

  1. [1]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge et al. , “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024

    arXiv 2024

  2. [2]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He et al. , “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800, 2024

    arXiv 2024

  3. [3]

    Deepseek-vl: towards real-world vision-language understanding,

    H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang et al. , “Deepseek-vl: towards real-world vision-language understanding,” arXiv preprint arXiv:2403.05525, 2024

    arXiv 2024

  4. [4]

    Chameleon: Mixed-modal early-fusion foundation models,

    C. Team, “Chameleon: Mixed-modal early-fusion foundation models,” arXiv preprint arXiv:2405.09818, 2024

    arXiv 2024

  5. [5]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  6. [6]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26 296–26 306

    work page 2024

  7. [7]

    Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation,

    E. Chern, J. Su, Y . Ma, and P. Liu, “Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation,” arXiv preprint arXiv:2407.06135, 2024

    arXiv 2024

  8. [8]

    Hallucination of multimodal large language models: A survey,

    Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou, “Hallucination of multimodal large language models: A survey,” arXiv preprint arXiv:2404.18930, 2024

    arXiv 2024

Show all 42 references
  1. [9]

    A survey on hallucination in large vision-language models,

    H. Liu, W. Xue, Y . Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng, “A survey on hallucination in large vision-language models,” arXiv preprint arXiv:2402.00253, 2024

    2024 arXiv

  2. [10]

    Evaluating object hallucination in large vision-language models,

    Y . Li, Y . Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  3. [11]

    Mitigating hallucination in large multi-modal models via robust instruction tuning,

    F. Liu, K. Lin, L. Li, J. Wang, Y . Yacoob, and L. Wang, “Mitigating hallucination in large multi-modal models via robust instruction tuning,” arXiv preprint arXiv:2306.14565, 2023

    2023 arXiv

  4. [12]

    A survey of hallucination in large foundation models,

    V . Rawte, A. Sheth, and A. Das, “A survey of hallucination in large foundation models,”arXiv preprint arXiv:2309.05922, 2023

    2023 arXiv

  5. [13]

    Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale,

    J. Chen, C. Gui, R. Ouyang, A. Gao, S. Chen, G. H. Chen, X. Wang, R. Zhang, Z. Cai, K. Ji et al., “Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale,” arXiv preprint arXiv:2406.19280, 2024

    2024 arXiv

  6. [14]

    Advancing medical imaging with language models: A journey from n-grams to chatgpt,

    M. Hu, S. Pan, Y . Li, and X. Yang, “Advancing medical imaging with language models: A journey from n-grams to chatgpt,” arXiv preprint arXiv:2304.04920, 2023

    2023 arXiv

  7. [15]

    Chatcad: Interactive computer-aided diagnosis on medical image using large language models,

    S. Wang, Z. Zhao, X. Ouyang, Q. Wang, and D. Shen, “Chatcad: Interactive computer-aided diagnosis on medical image using large language models,” arXiv preprint arXiv:2302.07257, 2023

    2023 arXiv

  8. [16]

    A survey on multimodal large language models for autonomous driving,

    C. Cui, Y . Ma, X. Cao, W. Ye, Y . Zhou, K. Liang, J. Chen, J. Lu, Z. Yang, K.-D. Liaoet al., “A survey on multimodal large language models for autonomous driving,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2024, pp. 958–979

    2024

  9. [17]

    Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving,

    W. Wang, J. Xie, C. Hu, H. Zou, J. Fan, W. Tong, Y . Wen, S. Wu, H. Deng, Z. Li et al. , “Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving,” arXiv preprint arXiv:2312.09245, 2023

    2023

  10. [18]

    Evaluating a large language model on searching for gui layouts,

    P. Brie, N. Burny, A. Sluÿters, and J. Vanderdonckt, “Evaluating a large language model on searching for gui layouts,” Proceedings of the ACM on Human-Computer Interaction , vol. 7, no. EICS, pp. 1–37, 2023

    2023

  11. [19]

    Dola: Decoding by contrasting layers improves factuality in large language models,

    Y .-S. Chuang, Y . Xie, H. Luo, Y . Kim, J. Glass, and P. He, “Dola: Decoding by contrasting layers improves factuality in large language models,” arXiv preprint arXiv:2309.03883, 2023

    2023 arXiv

  12. [20]

    In-context sharpness as alerts: An inner representation perspective for hallucination mitigation,

    S. Chen, M. Xiong, J. Liu, Z. Wu, T. Xiao, S. Gao, and J. He, “In-context sharpness as alerts: An inner representation perspective for hallucination mitigation,” arXiv preprint arXiv:2403.01548, 2024. 10

    2024 arXiv

  13. [21]

    Llms know more than they show: On the intrinsic representation of llm hallucinations,

    H. Orgad, M. Toker, Z. Gekhman, R. Reichart, I. Szpektor, H. Kotek, and Y . Belinkov, “Llms know more than they show: On the intrinsic representation of llm hallucinations,”arXiv preprint arXiv:2410.02707, 2024

    2024 arXiv

  14. [22]

    Llama slayer 8b: Shallow layers hold the key to knowledge injection,

    T. Chen, Z. Tan, T. Gong, Y . Wu, Q. Chu, B. Liu, J. Ye, and N. Yu, “Llama slayer 8b: Shallow layers hold the key to knowledge injection,” arXiv preprint arXiv:2410.02330, 2024

    2024 arXiv

  15. [23]

    Insights into llm long-context failures: When transformers know but don’t tell,

    T. Lu, M. Gao, K. Yu, A. Byerly, and D. Khashabi, “Insights into llm long-context failures: When transformers know but don’t tell,” arXiv preprint arXiv:2406.14673, 2024

    2024 arXiv

  16. [24]

    Knowledge mechanisms in large language models: A survey and perspective,

    M. Wang, Y . Yao, Z. Xu, S. Qiao, S. Deng, P. Wang, X. Chen, J.-C. Gu, Y . Jiang, P. Xieet al., “Knowledge mechanisms in large language models: A survey and perspective,” arXiv preprint arXiv:2407.15017, 2024

    2024 arXiv

  17. [25]

    Mllm can see? dynamic correction decoding for hallucination mitigation,

    C. Wang, X. Chen, N. Zhang, B. Tian, H. Xu, S. Deng, and H. Chen, “Mllm can see? dynamic correction decoding for hallucination mitigation,” arXiv preprint arXiv:2410.11779, 2024

    2024 arXiv

  18. [26]

    Unleashing region understanding in intermediate layers for mllm-based referring expression generation,

    Y . Liang, Z. Cai, J. Xu, G. Huang, Y . Wang, X. Liang, J. Liu, Z. Li, J. Wang, and S.-L. Huang, “Unleashing region understanding in intermediate layers for mllm-based referring expression generation,” Advances in Neural Information Processing Systems , vol. 37, pp. 120 578–12...

    2024

  19. [27]

    Branchynet: Fast inference via early exiting from deep neural networks,

    S. Teerapittayanon, B. McDanel, and H.-T. Kung, “Branchynet: Fast inference via early exiting from deep neural networks,” in 2016 23rd international conference on pattern recognition (ICPR). IEEE, 2016, pp. 2464–2469

    2016

  20. [28]

    Depth-adaptive transformer,

    M. Elbayad, J. Gu, E. Grave, and M. Auli, “Depth-adaptive transformer,” arXiv preprint arXiv:1910.10073, 2019

    1910 arXiv

  21. [29]

    Confident adaptive language modeling,

    T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V . Tran, Y . Tay, and D. Metzler, “Confident adaptive language modeling,” Advances in Neural Information Processing Systems , vol. 35, pp. 17 456–17 472, 2022

    2022

  22. [30]

    Hallucination augmented contrastive learning for multimodal large language model,

    C. Jiang, H. Xu, M. Dong, J. Chen, W. Ye, M. Yan, Q. Ye, J. Zhang, F. Huang, and S. Zhang, “Hallucination augmented contrastive learning for multimodal large language model,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 27 0...

    2024

  23. [31]

    Delve into visual contrastive decoding for hallucination mitigation of large vision-language models,

    Y .-L. Lee, Y .-H. Tsai, and W.-C. Chiu, “Delve into visual contrastive decoding for hallucination mitigation of large vision-language models,” arXiv preprint arXiv:2412.06775, 2024

    2024 arXiv

  24. [32]

    Mitigating object hallucina- tions in large vision-language models through visual contrastive decoding,

    S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing, “Mitigating object hallucina- tions in large vision-language models through visual contrastive decoding,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13 872–13 882

    2024

  25. [33]

    Mirage in the eyes: Hallucination attack on multi-modal large language models with only attention sink,

    Y . Wang, M. Zhang, J. Sun, C. Wang, M. Yang, H. Xue, J. Tao, R. Duan, and J. Liu, “Mirage in the eyes: Hallucination attack on multi-modal large language models with only attention sink,” arXiv preprint arXiv:2501.15269, 2025

    2025 arXiv

  26. [34]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data,

    Q. Yu, J. Li, L. Wei, L. Pang, W. Ye, B. Qin, S. Tang, Q. Tian, and Y . Zhuang, “Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 12 944–12 953

    2024

  27. [35]

    Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models,

    X. Zou, Y . Wang, Y . Yan, S. Huang, K. Zheng, J. Chen, C. Tang, and X. Hu, “Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models,” arXiv preprint arXiv:2410.03577, 2024

    arXiv

  28. [36]

    Helpd: Mitigating hallucination of lvlms by hierarchical feedback learning with vision-enhanced penalty decoding,

    F. Yuan, C. Qin, X. Xu, and P. Li, “Helpd: Mitigating hallucination of lvlms by hierarchical feedback learning with vision-enhanced penalty decoding,” arXiv preprint arXiv:2409.20429, 2024

    2024 arXiv

  29. [37]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation,

    Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu, “Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  30. [38]

    Instructblip: Towards general-purpose vision-language models with instruction tuning,

    W. Dai, J. Li, D. LI, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt,...

    2023

  31. [39]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing vision-language understanding with advanced large language models,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=1tZbq88f27

    2024

  32. [40]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huanget al., “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  33. [41]

    Mme-survey: A comprehensive survey on evaluation of multimodal llms,

    C. Fu, Y .-F. Zhang, S. Yin, B. Li, X. Fang, S. Zhao, H. Duan, X. Sun, Z. Liu, L. Wanget al., “Mme-survey: A comprehensive survey on evaluation of multimodal llms,” arXiv preprint arXiv:2411.15296, 2024

    2024 arXiv

  34. [42]

    Object hallucination in image captioning,

    A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko, “Object hallucination in image captioning,” arXiv preprint arXiv:1809.02156, 2018. 12 Appendix A Ablation Results of Dynamic Soft Modulation To quantify the effect of soft modulation, we evaluated the results o...

    2018 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.