Pith. sign in

REVIEW 5 major objections 7 minor 54 references

Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Non-hallucinated scene objects such as grass and sky causally induce object hallucination in large vision-language models, and a two-step foreground-background prompt blocks this effect, cutting CHAIR from 9.0 to 5.6 on AMBER with…

desk verdict Promising empirical prompt result, but the causal claim rests on a single-sample substitution that is not a front-door adjustment. read the letter →

arxiv 2412.02946 v1 pith:ORDBTZXL submitted 2024-12-04 cs.CV cs.AIcs.LGcs.MM

Po-Hsuan Huang , Jeng-Lin Li , Chin-Po Chen , Ming-Ching Chang , Wei-Chao Chen This is my paper · ORCID
classification cs.CVcs.AIcs.LGcs.MM
keywords objecthallucinationlargevision-languagemodelcausalinferenceconfounderforeground-backgroundpromptingembeddingeditingAMBERbenchmarkreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that object hallucination in large vision-language models is not purely random error but is causally induced by non-hallucinated context objects—trees, grass, sky—that co-occur with hallucinated objects in training data. It builds a structural causal model with a hidden context confounder and shows that intervening on the image, the text prompt, or the latent embedding can block this causal path. The headline result is a simple two-step foreground-background prompt that cuts CHAIR hallucination from 9.0 to 5.6 and HAL from 38.8 to 27.8 on the AMBER benchmark with InstructBLIP, while keeping coverage essentially unchanged. If correct, this means tuning-free prompt or image changes can substantially reduce hallucinations, and that model internals carry a detectable 'hallucination direction' that can be edited at inference time.

What carries the argument

The central object is the structural causal model of LVLM generation with a hidden context confounder $Z_c$ that influences both the target-object representation $Z_o$ and the answer $A$. The paper uses backdoor adjustment via image and text interventions, a front-door adjustment via a two-step foreground-background Chain-of-Thought prompt, and a t-test-based embedding saliency map for direct embedding editing, all aimed at severing the spurious path from context to hallucinated output.

What would settle it

A controlled test would hold the object region fixed while removing only the context (e.g., erase the grass but keep the boy) and check whether hallucination of 'frisbee' drops; if it does not, the context-driven causal claim loses support. Conversely, keeping the context but changing the object should still induce context-typical hallucinations if the mechanism is causal.

Watch

Extended reading notes

Core claim

The paper's central claim is that spurious correlations with non-hallucinated context objects causally produce hallucinated outputs. Using a structural causal model with a latent context confounder $Z_c$, it identifies a backdoor path $Z_c \to Z_o \to A$ that contaminates generation, and then demonstrates three interventions that block this path: pasting or removing small objects in the image, a foreground-background (FGBG) prompting scheme, and editing salient embedding dimensions. The FGBG prompt is the most effective across both InstructBLIP and mPLUG-Owl2 on AMBER and COCO, reducing hallucination by roughly half while preserving coverage. Embedding saliency analysis further shows that non-hallucinatory words like 'desk' or 'sign' sit closer to hallucinatory embeddings, supporting the claim that context objects are the hidden risk factors.

Load-bearing premise

The causal reading hinges on the assumption that a small image edit (pasting or removing an object) changes only the target-object representation and leaves the perceived context almost unchanged, since that is what lets the backdoor adjustment block the confounder.

Editorial extensions

If this is right

  • Prompt-level fixes can cut hallucination rates substantially without retraining, auxiliary detectors, or extra decoding overhead.
  • Non-hallucinated context words are warning signs; steering generation toward foreground description makes models more conservative and more accurate.
  • Embedding dimensions that statistically separate hallucinated from non-hallucinated generations can be edited at inference time, pointing to a general de-hallucination mechanism inside the model.
  • The causal interpretation implies that hallucination reduction should target context co-occurrence biases, not just object presence or decoding contrast.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the causal mechanism is real, hallucination benchmarks should report context-object co-occurrence statistics, not just per-object presence, since those co-occurrences are what the model latches onto.
  • The foreground-background split may transfer to other generative vision-language tasks, such as visual question answering or grounded captioning, where the foreground prompt could serve as a stable anchor that blocks context leakage.
  • The embedding editing result suggests a single learned 'de-hallucination direction' could be applied across models and samples, reducing the need for per-sample proxy datasets.
  • A testable extension would be to measure whether hallucination rates scale with the conditional probability $P(\text{hallucinated object} \mid \text{context})$ in the training distribution; if they do, the causal story gains direct support.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The manuscript proposes a causal framework for object hallucination in large vision-language models. It introduces a hidden context confounder Zc, a target-object representation Zo, and the generated answer A, and then tests three families of interventions intended to block the undesired path Zc → Zo: image intervention (pasting a small object or removing a hallucination-inducing object), text intervention (a foreground/background, FGBP, two-step prompt), and embedding intervention (editing salient latent dimensions). Experiments on AMBER and COCO with InstructBLIP and mPLUG-Owl2 report that FGBP lowers CHAIR from 9.0 to 5.6 and HAL from 38.8 to 27.8 on AMBER for InstructBLIP, image pasting lowers CHAIR to 5.5, and embedding editing gives smaller but consistent reductions. The paper also analyzes embedding saliency maps and retrieval-safety scores, arguing that non-hallucinatory context words lie closer to hallucination-prone embeddings.

Significance. The empirical part of the paper is potentially useful: two tuning-free interventions reduce hallucination metrics across two models and two datasets, and the FGBP result is not circular with respect to the test labels because the prompt does not use ground-truth object sets. The authors are also transparent about their key assumptions, such as P(Zc|I') ≈ P(Zc|I) and the single-sample replacement of E[X]. If the causal claims were supported, this would be a substantive advance. As it stands, however, the causal derivation in Section 3.5 is not valid as written, and the 'inducing' vocabulary in Table 1 is supported only by conditional association. The paper is therefore best viewed as a promising empirical intervention study whose causal framing currently overreaches.

major comments (5)
  1. [§3.5, Eq. (2)–(3)] The front-door adjustment is not actually computed. Equation (2) requires P(A|do(S)) = Σ_X P(X)P(A|S,X) = E_X[P(A|S,X)], a population-level backdoor adjustment over the input distribution. Equation (3) then replaces this expectation with P(A|S⊕E[X]), and the text states that the authors 'empirically use X in a single run to replace E[X]'. A single sample is not an expectation, so the reported FGBP reduction cannot be attributed to blocking Zc→Zo; the two-step prompt's format and length are confounded with the intended causal intervention. The authors should either estimate the expectation over a held-out input distribution or explicitly reframe FGBP as an empirical prompting intervention without the front-door claim.
  2. [§3.5, collider argument and notation] The derivation of P(S|do(X)) is not justified. The text writes that 'the collision structure of Zc → A ← S allows us to block the backdoor path and derive P(S|do(X)) = P(S|X)'; in standard do-calculus, conditioning on a collider opens a path rather than blocking one, and the equality requires the front-door criterion (S intercepts all X→A paths and there is no unobserved confounder of S and A), which is not established. The equations also shift between Q, the text query, and X, the input token sequence, without defining the mapping between them. The DAG in Figure 2(d) needs a formal statement of the assumed independencies and a step-by-step do-calculus derivation or a precise citation for each step.
  3. [§3.4, image intervention] The backdoor adjustment for the image intervention rests on the untested assumption P(Zc|I') ≈ P(Zc|I). Pasting a one-sixth-sized object, or removing and inpainting a hallucination-inducing object, can plausibly alter scene context statistics, especially when the removed object is large. If the intervention also changes Zc, the claimed backdoor blocking fails. The paper should measure the context shift (for example, via object-detection agreement or embedding distance between I and I') and report results on subsets where the assumption holds; without that, the image-pasting result remains an empirical effect with an unverified causal mechanism.
  4. [§4.1, Table 2] The text states that 'the object removal intervention, as shown in Table 2, results in lower CHAIR and HAL scores', but Table 2 contradicts this on AMBER: for InstructBLIP, object removal increases CHAIR from 9.0 to 11.7 and HAL from 38.8 to 46.2, and for mPLUG-Owl2 it increases CHAIR from 9.0 to 12.9 and HAL from 40.4 to 46.0. Only on COCO sentence-level CHAIRs is object removal consistently better. This discrepancy should be corrected, and the interpretation of object removal as evidence for the causal account needs to be substantially qualified.
  5. [§3.2, Table 1 and §3.4 object removal] Table 1's 'hallucinatory-inducing words' are defined by conditional association P(Oh|On), not by any intervention, so calling them 'inducing' presupposes the causal conclusion. Moreover, the object-removal intervention selects the object to remove using the same P(Oh|On) statistics computed on the AMBER set (§3.2), making the object-removal gains partly dependent on test-set statistics rather than on a generalizable causal relation. The authors should either estimate these statistics on an independent training split or explicitly label Table 1 as descriptive association and use neutral terminology.
minor comments (7)
  1. [§3.3] The sentence 'The path Zo ← Zc → A forms a backdoor path and thus and thus we aim to perform an intervention' contains a duplicated 'and thus' that should be removed.
  2. [§3.5] The phrase 'collision structure' should be 'collider structure' to use standard causal terminology.
  3. [Table 2] There is a typo in the row label 'BG prmopt', which should read 'BG prompt'; also, 'Amber' should be consistently capitalized as 'AMBER'.
  4. [§3.3, Eq. (1)] The definition of TCE is unclear: δ(P,P′) is defined as an indicator I(H(A) > H(A′)), but Eq. (1) writes an expectation over X without specifying whether the randomness is over the test set, the LVLM decoding, or both. Please clarify the probability space.
  5. [§4.2, Stopping prompt] The Stopping prompt derives Oh from a prior LVLM output compared with annotated ground truth, making it an oracle-informed baseline. The text calls it a 'hard upper bound', but the caption and main text should state this more prominently so readers do not interpret it as a fair unsupervised baseline.
  6. [Supplementary] The paper should provide the exact FG and BG prompt templates, the full Table 1 statistics, and sensitivity analyses for the embedding intervention hyperparameters ρ, K, and the saliency p-value threshold.
  7. [Figure 5 caption] The caption contains 'cross-modality casual relations'; this should be 'causal relations'.

Circularity Check

2 steps flagged · score 6.0 of 10

The FGBG causal-effect derivation in §3.5 replaces the front-door expectation with the sample's own embedding, so the claimed do(X) reduction reduces by construction to the observational prompt-conditioned output; the embedding intervention is fitted and evaluated on the same proxy data. The raw prompt improvement is independent, but the causal interpretation is not.

  1. self definitional [Section 3.5, Eq. (3) (Text (Q) Intervention)]
    "EX [P (A|S, X)] ≈ P (A|S, E[X]) ≈ P (A|S ⊕E[X]), (3) where ⊕ denotes vector concatenation. ... We thus empirically use X in a single run to replace E[X]."

    The front-door adjustment required by Eq. (2) is P(A|do(S)) = E_X[P(A|S,X)], a population-level expectation over inputs. Eq. (3) replaces E[X] with the current sample's embedding, and the text explicitly says 'we empirically use X in a single run to replace E[X]'. Therefore the reported P(A|do(X)) reduces by construction to the ordinary conditional output P(A|S⊕E[X]) under the two-step prompt. The intervention do(X) is never actually applied; the measured CHAIR/HAL drop is the model's response to a differently formatted prompt, not an estimate of blocking the Zc→Zo path. The causal claim is thus equivalent to the observational input by construction.

  2. fitted input called prediction [Section 3.6, Embedding Intervention; Table 2]
    "Depending on samples X in a proxy dataset with hallucination annotations, we derive a hallucinated group Xh = {x ∈ X|H(f(x)) > 0} and a non-hallucinated group Xn = {x ∈ X|H(f(x)) = 0}. ... Each query sample obtains the mean embedding from most similar and non-hallucinated samples in the proxy dataset."

    The proxy dataset is not stated to be disjoint from the evaluation benchmarks; the only datasets introduced in §3.1 are AMBER and COCO, and Table 2 reports the embedding intervention on exactly AMBER and COCO. Thus the saliency map and the non-hallucinated prototype E_K are fitted from the same samples on which hallucination is then measured. The reported reduction is an in-sample fit of the editing direction to the evaluation labels, not an out-of-sample prediction, so the improvement is constructed from the test distribution rather than independently predicted.

full rationale

The paper's headline FGBG prompt result (AMBER CHAIR 9.0 → 5.6, HAL 38.8 → 27.8 with InstructBLIP) does not read the AMBER ground-truth annotations, so that empirical reduction is not circular. There is also no load-bearing self-citation chain: the causality references [27,29,39,50] are standard external citations. The circularity is localized to two places. First, the front-door derivation in §3.5 is mathematically not performed: Eq. (3) replaces the required E_X[·] with the single input embedding, so the interventional distribution collapses to the observational conditional given the two-step prompt; the causal effect claim is therefore a relabeled prompt-format effect. Second, the embedding intervention in §3.6 constructs its editing direction from a proxy dataset that appears to be the same AMBER/COCO benchmark on which it is evaluated, making the reported embedding gains in-sample rather than predicted. These are genuine reductions of claimed predictions to their own inputs, but the main empirical FGBG improvement remains independent, so the overall circularity is partial rather than total.

Assumptions & free parameters 6 free parameters · 6 assumptions · 2 invented entities

Two of the three interventions are fitted on evaluation data: object-removal priority uses P(Oh|On) computed on AMBER, and embedding editing builds its saliency map and prototypes from hallucination-labeled samples on the same benchmark. The causal claims add unmeasured latent variables Zc and S. The reported numbers also depend on untested hyperparameters (rho, K, pasting geometry, p-threshold).

free parameters (6)
  • Editing strength rho = not reported
    Controls how much the embedding is pushed toward the non-hallucinated prototype in Section 3.6; no sensitivity analysis or selection procedure is given.
  • Top-K nearest neighbors = not reported
    K in the k-nearest-neighbor averaging for the non-hallucinated prototype E_K in Section 3.6.
  • Pasted object size and position = one-sixth of shortest side, top-left corner
    Hand-chosen geometry for image pasting in Section 3.4; no ablation across sizes or positions.
  • Pasted object category = single rabbit
    The image-pasting intervention uses one fixed object; other objects are not tested except in a semantic/non-semantic inpainting study in Table 3.
  • Saliency p-value threshold = 0.001
    Threshold for marking embedding dimensions as hallucination-salient in Section 3.6; no analysis of sensitivity.
  • mPLUG-Owl2 decoding hyperparameters = temperature 0.7, max new tokens 512
    Set to reproduce AMBER benchmark numbers; not a hallucination-model parameter but affects all compared numbers.
assumptions (6)
  • domain assumption The causal graph in Figure 2b correctly represents LVLM generation, with unobserved context Zc affecting both target object Zo and answer A.
    Invoked in Section 3.3 to justify backdoor adjustment; Zc is never measured or validated.
  • ad hoc to paper Conditional probability P(Oh|On) measures an inducing relation, so On can be called hallucination-inducing.
    Section 3.2 converts co-occurrence statistics into the arrow On -> Oh without controlling confounders.
  • ad hoc to paper P(Zc|I') ≈ P(Zc|I) for image interventions.
    Section 3.4 assumes pasting or removal leaves context background unchanged; no validation.
  • ad hoc to paper Front-door adjustment through mediator S is valid, and E[X] can be replaced by one observed sample X.
    Section 3.5, equations (2)-(3); the graph does not clearly support P(S|do(X)) = P(S|X) and the expectation approximation is unjustified.
  • domain assumption Hallucination metrics H (CHAIR, HAL, Cog, Cover) capture the intended notion of object hallucination.
    Used throughout Section 4 to score responses; conventions come from AMBER and COCO benchmarks.
  • domain assumption AMBER/COCO human annotations of present objects are correct.
    Ground truth O defines hallucinatory and non-hallucinatory object sets in Sections 3.2 and 3.4.
invented entities (2)
  • Zc, the hidden context factor
    purpose: Unobserved confounder that is claimed to induce hallucinated objects via context cues.
    Introduced in the causal graph of Section 3.3; no measurement protocol or falsifiable prediction outside the paper is given.
  • S, the foreground mediator
    purpose: Mediator variable for the front-door adjustment in text intervention.
    Introduced in Section 3.5 as the foreground description; it is not separated from ordinary text generation, so it has no independent falsifiable identity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis." pith.science (2026). https://pith.science/paper/ORDBTZXL

@misc{pith2026241202946,
  author       = {Pith},
  title        = {Pith review of: Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ORDBTZXL}},
  note         = {Machine review of arXiv:2412.02946}
}
read the original abstract

Recent advancements in large vision-language models (LVLM) have significantly enhanced their ability to comprehend visual inputs alongside natural language. However, a major challenge in their real-world application is hallucination, where LVLMs generate non-existent visual elements, eroding user trust. The underlying mechanism driving this multimodal hallucination is poorly understood. Minimal research has illuminated whether contexts such as sky, tree, or grass field involve the LVLM in hallucinating a frisbee. We hypothesize that hidden factors, such as objects, contexts, and semantic foreground-background structures, induce hallucination. This study proposes a novel causal approach: a hallucination probing system to identify these hidden factors. By analyzing the causality between images, text prompts, and network saliency, we systematically explore interventions to block these factors. Our experimental findings show that a straightforward technique based on our analysis can significantly reduce hallucinations. Additionally, our analyses indicate the potential to edit network internals to minimize hallucinated outputs.

Figures

Figures reproduced from arXiv: 2412.02946 by the authors.

Figure 1
Figure 1. The InstructBLIP [8] LVLM hallucinates a frisbee when describing a boy in the green field. There might be a spurious correlation between a boy and a frisbee. Meanwhile, the green field is another non-hallucinatory subject that might induce hallucinations. This underlying relation remains underexplored in the hallucination reduction research. above in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Ideal LVLM generation. (b) The causal graphical model for LVLM generation. (c-e) Deconfounded by (c) image intervention, (d) text intervention, and (e) embedding intervention. dating model parameters, relying on regenerated or refined answers from ChatGPT [1], indicating their success hinges on the strong summarization ability of ChatGPT. We ob￾serve that discriminative ability in vision question answer￾ing does… view at source ↗
Figure 3
Figure 3. Our proposed image, text, and embedding intervention approaches correspond to Figure [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Embedding Saliency with timestamps in rows and dimensions in columns for (a) Instructblip and (b) mPLUG-Owl2. (c) and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: A case with the FGBG approach continuing to hallucinate while image-pasting successfully reduces the hallucination, indicating [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 27 canonical work pages

  1. [1]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Hallucination of multimodal large language models: A survey

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024. 1

    arXiv 2024

  3. [3]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005. 2

    work page 2005

  4. [4]

    Let there be a clock on the beach: Reducing object halluci- nation in image captioning

    Ali Furkan Biten, Llu ´ıs G ´omez, and Dimosthenis Karatzas. Let there be a clock on the beach: Reducing object halluci- nation in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1381–1390, 2022. 1

    work page 2022

  5. [5]

    Deconfounded visual question generation with causal infer- ence

    Jiali Chen, Zhenjun Guo, Jiayuan Xie, Yi Cai, and Qing Li. Deconfounded visual question generation with causal infer- ence. In Proceedings of the 31st ACM International Confer- ence on Multimedia, pages 5132–5142, 2023. 3

    work page 2023

  6. [6]

    Truth forest: Toward multi-scale truthfulness in large language models through intervention without tuning

    Zhongzhi Chen, Xingwu Sun, Xianfeng Jiao, Fengzong Lian, Zhanhui Kang, Di Wang, and Chengzhong Xu. Truth forest: Toward multi-scale truthfulness in large language models through intervention without tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20967–20974, 2024. 3

    work page 2024

  7. [7]

    Can we edit multimodal large language models? In Proceed- ings of the 2023 Conference on Empirical Methods in Nat- ural Language Processing, pages 13877–13888, Singapore, Dec

    Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen, and Ningyu Zhang. Can we edit multimodal large language models? In Proceed- ings of the 2023 Conference on Empirical Methods in Nat- ural Language Processing, pages 13877–13888, Singapore, Dec. 2023. Association for Computational Linguistics. 6

    work page 2023

  8. [8]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. Ad- vances in Neural Information Processing Systems, 36, 2024. 1, 2, 3, 6

    work page 2024

Show all 54 references
  1. [9]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pages 18135–18143, 2024. 2

    2024

  2. [10]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. arXiv preprint arXiv:2311.17911 ,

    arXiv

  3. [11]

    Hallucination augmented contrastive learn- ing for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learn- ing for multimodal large language model. arXiv preprint arXiv:2312.06968, 2023. 1

    2023 arXiv

  4. [12]

    Sophia Koepke, Cordelia Schmid, and Zeynep Akata

    Jae Myung Kim, A. Sophia Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious corre- lations for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 2585–2595, June

  5. [13]

    V olcano: mitigating multimodal hallucina- tion through self-feedback guided revision

    Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Min- joon Seo. V olcano: mitigating multimodal hallucina- tion through self-feedback guided revision. arXiv preprint arXiv:2311.07362, 2023. 2

    2023 arXiv

  6. [14]

    Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shi- jian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding. arXiv preprint arXiv:2311.16922,

    arXiv

  7. [15]

    Inference-time intervention: Elic- iting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi ´egas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Elic- iting truthful answers from a language model. In Thirty- seventh Conference on Neural Information Processing Sys- tems, 2023. 3, 6

    2023

  8. [16]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pages 292–305, 2023. 2

    2023

  9. [17]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  10. [18]

    Revisiting the role of language priors in vision-language models

    Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models. 2024. 3

    2024

  11. [19]

    Show, deconfound and tell: Im- age captioning with causal inference

    Bing Liu, Dong Wang, Xu Yang, Yong Zhou, Rui Yao, Zhi- wen Shao, and Jiaqi Zhao. Show, deconfound and tell: Im- age captioning with causal inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18041–18050, 2022. 3

    2022

  12. [20]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representa- tions, 2023. 2

    2023

  13. [21]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 2

    2023

  14. [22]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024. 1

    2024 arXiv

  15. [23]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 5

    2023 arXiv

  16. [24]

    Negative object presence evaluation (nope) to measure object hallucination in vision-language models

    Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. arXiv preprint arXiv:2310.05338, 2023. 3

    2023 arXiv

  17. [25]

    Towards vision-language mechanistic interpretabil- ity: A causal tracing tool for blip

    Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision-language mechanistic interpretabil- ity: A causal tracing tool for blip. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2856–2861, 2023. 3

    2023

  18. [26]

    Direct and indirect effects

    Judea Pearl. Direct and indirect effects. Probabilistic and Causal Inference: The Works of Judea Pearl, page 373, 2001. 4

    2001

  19. [27]

    Causality

    Judea Pearl. Causality. Cambridge university press, 2009. 3, 4

    2009

  20. [28]

    The seven tools of causal inference, with re- flections on machine learning

    Judea Pearl. The seven tools of causal inference, with re- flections on machine learning. Communications of the ACM, 62:54 – 60, 2019. 4

    2019

  21. [29]

    Causal inference in statistics: A primer

    Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley & Sons,

  22. [30]

    Valor- eval: Holistic coverage and faithfulness evaluation of large vision-language models

    Haoyi Qiu, Wenbo Hu, Zi-Yi Dou, and Nanyun Peng. Valor- eval: Holistic coverage and faithfulness evaluation of large vision-language models. arXiv preprint arXiv:2404.13874,

    arXiv

  23. [31]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Confer- ence on Empirical Methods in Natural L...

    2018

  24. [32]

    A causal framework to quantify the robustness of mathematical reasoning with lan- guage models

    Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with lan- guage models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023

  25. [33]

    Aligning large multi- modal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023. 2

    2023 arXiv

  26. [34]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. 2

    2015

  27. [35]

    Cross modality bias in visual question answering: A causal view with possible worlds vqa

    Ali V osoughi, Shijian Deng, Songyang Zhang, Yapeng Tian, Chenliang Xu, and Jiebo Luo. Cross modality bias in visual question answering: A causal view with possible worlds vqa. IEEE Transactions on Multimedia, 2024. 3

    2024

  28. [36]

    Vigc: Visual instruction generation and correction

    Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al. Vigc: Visual instruction generation and correction. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 38, pages 5309–5317, 2024. 2

    2024

  29. [37]

    An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation. arXiv preprint arXiv:2311.07397,

    arXiv

  30. [38]

    Logical closed loop: Uncov- ering object hallucinations in large vision-language models

    Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. Logical closed loop: Uncov- ering object hallucinations in large vision-language models. arXiv preprint arXiv:2402.11622, 2024. 1

    2024 arXiv

  31. [39]

    Show, attend and tell: Neural image caption gen- eration with visual attention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption gen- eration with visual attention. In International conference on machine learning, pages 2048–2057. PMLR, 2015. 5

    2015

  32. [40]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023. 2

    2023 arXiv

  33. [41]

    Vigor: Improving visual ground- ing of large vision language models with fine-grained reward modeling

    Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, and Li Erran Li. Vigor: Improving visual ground- ing of large vision language models with fine-grained reward modeling. arXiv preprint arXiv:2402.06118, 2024. 2

    2024

  34. [42]

    Decon- founded image captioning: A causal retrospect

    Xu Yang, Hanwang Zhang, and Jianfei Cai. Decon- founded image captioning: A causal retrospect. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):12996–13010, 2021. 3

    2021

  35. [43]

    Causal attention for vision-language tasks

    Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for vision-language tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9847–9857, 2021. 3

    2021

  36. [44]

    mPLUG-Owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mPLUG-Owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 2

    2023 arXiv

  37. [45]

    mPLUG-Owl2: Revolutionizing multi-modal large lan- guage model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mPLUG-Owl2: Revolutionizing multi-modal large lan- guage model with modality collaboration. arXiv preprint arXiv:2311.04257, 2023. 1, 2, 3, 6

    2023 arXiv

  38. [46]

    Woodpecker: Hallucination correction for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045 ,

    arXiv

  39. [47]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wen- tao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. arXiv preprint arXiv:2311.13614 ,

    arXiv

  40. [48]

    Inpaint anything: Segment anything meets image inpainting

    Tao Yu, Runsen Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. ArXiv, abs/2304.06790,

    arXiv

  41. [49]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.00849, 2023. 1

    2023 arXiv

  42. [50]

    Causal prompting: Debiasing large language model prompting based on front-door adjustment

    Congzhi Zhang, Linhai Zhang, Deyu Zhou, and Guoqiang Xu. Causal prompting: Debiasing large language model prompting based on front-door adjustment. arXiv preprint arXiv:2403.02738, 2024. 5

    2024 arXiv

  43. [51]

    Mitigating object hallucination in large vision- language models via classifier-free guidance

    Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision- language models via classifier-free guidance. arXiv preprint arXiv:2402.08680, 2024. 2

    2024 arXiv

  44. [52]

    Causal-debias: Unifying debiasing in pretrained language models and fine-tuning via causal invariant learning

    Fan Zhou, Yuzhou Mao, Liu Yu, Yi Yang, and Ting Zhong. Causal-debias: Unifying debiasing in pretrained language models and fine-tuning via causal invariant learning. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2023

  45. [53]

    Analyzing and mitigating object hallucination in large vision-language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Con- ference on Learning Representations, 2024. 1, 5

    2024

  46. [54]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representa- tions, 2023. 1, 2

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.