Pith. sign in

REVIEW 4 major objections 6 minor 75 references

Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that object hallucinations in multimodal large language models are caused by entangled semantic representations formed from co-occurrence biases, and that causal-intervention modules that disentangle these representations…

desk verdict Plausible idea and a useful representational analysis, but the causal claim is underdetermined by the evaluation; deserves a serious but demanding referee. read the letter →

arxiv 2505.19474 v1 pith:553JSEMA submitted 2025-05-26 cs.AI

Xinmiao Hu , Chun Wang , Ruihe An , ChenYu Shao , Xiaojun Ye , Sheng Zhou , Liangcheng Li (Zhejiang University) This is my paper
classification cs.AI
keywords objecthallucinationmultimodallargelanguagemodelscausalinferencebackdooradjustmentrepresentationdisentanglementco-occurrencebiasconfounderdictionaryLLaVA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that object hallucinations in multimodal large language models are a representational problem: frequent object co-occurrences in training data push the hidden states of commonly paired objects close together, so activating one object representation can activate an absent companion. It supports this with PCA visualizations showing that "dining table" and its top-10 co-occurring objects cluster tightly after the projector and remain entangled through most transformer layers. The proposed fix is a causality-driven disentanglement framework that applies backdoor adjustment at two points: a Causal-Driven Projector in the visual pathway and a Causal Intervention Module in the final language-model transformer layer. If correct, the method offers an architectural, end-to-end way to reduce a class of hallucinations without external tools, synthetic data, or post-processing.

What carries the argument

The machinery is backdoor adjustment approximated by the Normalized Weighted Geometric Mean (NWGM). Concretely, the framework adds two learnable cross-attention modules: a Causal-Driven Projector that adds an expectation term to the visual projector output, and a Causal Intervention Module in the final LLM transformer layer that adds cross-attention outputs over visual and textual confounder dictionaries $D_v$ and $D_t$. Each dictionary is a $K \times \sigma$ matrix whose rows are average post-projector representations of object categories, aggregated from 5,000 LLaVA-Instruct samples, and cross-attention lets the input modulate which confounder entries matter. The declared function is to block the backdoor paths $h \leftarrow D_v \to W$ and $h \leftarrow D_t \to W$ while preserving the genuine chain $S \to h \to W$.

What would settle it

Train the same architecture with confounder dictionaries built from a separate, independently trained encoder or from randomly permuted category prototypes, and compare hallucination rates; if the reductions persist, the dictionary content is not doing causal work. Equivalently, invert the dataset's co-occurrence statistics in a controlled benchmark: the causal model should hallucinate far less than an entangled baseline when the test set reverses training co-occurrences.

Watch

Extended reading notes

Core claim

On the paper's own account, the underlying cause of object hallucinations is entangled semantic representations driven by dataset co-occurrence bias, and this entanglement is measurable in feature space. The authors show that after the trained projector, and persistently through later LLM layers, the average visual representation of an object such as "dining table" clusters tightly with its most frequently co-occurring objects, and that textual representations behave similarly. They then claim that blocking the confounding paths through backdoor adjustment, implemented as a Causal-Driven Projector plus a Causal Intervention Module, separates these clusters and reduces hallucination while preserving or improving performance across multiple benchmarks. The headline result is a 22.6% increase in MME-Perception for the LLaMA-2-13B version, alongside consistent reductions on CHAIR and POPE.

Load-bearing premise

The load-bearing premise is that the averaged object representations stored in $D_v$ and $D_t$, harvested from the model's own intermediate states, are genuine confounders; if they merely re-encode the same co-occurrence bias, the intervention is not causal.

Editorial extensions

If this is right

  • Installing only the projector-side intervention yields most of the MME-Perception gain, while installing only the transformer-side intervention yields little; the paper concludes both stages are needed to stop bias propagation.
  • On LLaVA with LLaMA-2-13B, the method raises MME-Perception from 711.22 to 872.09 and lowers CHAIRs from 30.3 to 28.2 while matching or improving general comprehension benchmarks.
  • Because the modules are architectural rather than data-generation or post-processing steps, the paper claims they can be adapted to other multimodal large language model architectures without external models.
  • PCA visualizations show that "dining table" and its top-10 co-occurring objects, tightly clustered in the original model through layer 40, become separated at the projector output and remain separated by layer 40.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same confounder-dictionary intervention should transfer to other hallucination types such as attributes, relations, or actions by building dictionaries over those categories; the paper only demonstrates object-level co-occurrence.
  • The paper's own limitation note that confounders may be contaminated by noisy environment- and dataset-dependent variables, combined with dictionaries built from the model's own 0.1-epoch checkpoint, leaves open that part of the improvement is a learned additive offset rather than true do-calculus blocking.
  • A testable consequence of the paper's structural causal model is that unfreezing the CLIP vision encoder during training should matter: if the path $F_v \leftarrow D_v$ is truly blocked by the frozen encoder, unfreezing should not change hallucination much, and if it does, the causal story needs revision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Causal-LLaVA, a modification of LLaVA that inserts a Causal-Driven Projector in the visual pathway and a Causal Intervention Module in the final LLM transformer layer. The authors argue that object hallucinations arise from entangled semantic representations caused by co-occurrence biases in instruction data, and that a backdoor-adjustment-style causal intervention using visual and textual confounder dictionaries Dv and Dt can block the confounding paths h←Dv→W and h←Dt→W. They report improvements on POPE, MME-P, CHAIR, and several general VQA benchmarks, plus PCA visualizations showing better separation of co-occurring object representations. The paper includes ablations on module placement and projection-matrix choices, and the code is publicly released.

Significance. If the causal interpretation were valid, the paper would offer a principled, training-time architectural remedy for object hallucination in MLLMs, with potentially broad applicability. The paper has concrete strengths: it releases code, provides a detailed account of the proposed modules, includes ablations, and supports its representational-entanglement narrative with layer-wise visualizations. However, the central claim is not currently supported. The confounder dictionaries are built from the model's own intermediate representations, the implementation contradicts the stated backdoor adjustment, and the main comparison against LLaVA uses different training hyperparameters. These are load-bearing issues that affect the paper's main contribution, not merely presentation.

major comments (4)
  1. [§5.1, Tables 1-2] The headline comparison is confounded by training hyperparameters. The text states that Causal-LLaVA 'maintain[s] LLaVA's official training configuration with two critical modifications': doubled batch size (256) and halved learning rate (1e-3). The LLaVA baselines use the official configuration, so the reported gains (e.g., MME-P +6.0% for 7B and +22.6% for 13B) may be due to the altered optimization settings rather than the causal modules. A controlled baseline—LLaVA trained with the same doubled batch size and halved learning rate but without the disentanglement modules—is needed to support the attribution claim.
  2. [§4.3, §5.1, Fig. 4(b)] The confounder dictionaries Dv and Dt are not exogenous confounders as required by the causal graph. Each entry is 'the average post-projector visual representation of an object category, aggregated from 5,000 instances sampled from the LLaVA-Instruct dataset', computed from 'an intermediate checkpoint from a non-causal model trained for 0.1 epoch'. These entries are outputs of the very model the method aims to correct; they are descendants of Fv, S, and h, not common causes. Conditioning on them is therefore not a backdoor adjustment over an independent confounder, and the cross-attention weights (Wk, Wv, Wo in Eq. 10) are trained end-to-end, so the module can implement an arbitrary input-conditioned additive offset. To support the causal claim, the dictionaries must be shown to be faithful proxies for the confounders Z in Eq. (2), for example by deriving them from an external or frozen encoder or from an independently annotated dataset, and the claimed blocking of the backdoor paths must be demonstrated rather than assumed.
  3. [§A, Eq. (18) vs §4.2, Eqs. (4)-(5)] The supplementary derivation reintroduces the confounding dependence that the backdoor adjustment is supposed to remove. Equation (5) approximates P(W|do(h)) using E[dv] and E[dt], which corresponds to the uniform P(Z=z) in Eq. (2). However, the supplementary text states that 'when computing E_D[g_D(D)] for both Dv and Dt, we condition d on h via E[D|h][g_D(D)]'. Conditioning on h is exactly P(Z=z|h), which is the term appearing in the observational decomposition of Eq. (4) and which the do-operator is meant to eliminate. This is an internal inconsistency in the causal argument. In addition, the NWGM derivation in Eq. (17) assumes g is linear, but the intervention modules use softmax cross-attention with trainable projections; the approximation error for a nonlinear g is not analyzed.
  4. [Tables 1-2, §5.2] Several of the reported improvements are small and are presented without any measure of uncertainty. For example, POPE-random improves by only +1.42% and +1.20% for the 7B and 13B LLaVA variants, CHAIR-i drops from 9.5 to 9.2, and VQAv2 improves by +0.7 and +2.3 points across the two LLaVA sizes. Without multiple seeds, confidence intervals, or significance tests, it is unclear whether these differences are reliable, especially given the simultaneous change in batch size and learning rate. The authors should report variance across runs or demonstrate that the effects are reproducible.
minor comments (6)
  1. [Abstract and §1] The paper uses both 'MLLM' and 'LVLM' for the same class of models; please unify the terminology throughout.
  2. [Figures 1-3, 6] The PCA plots would be clearer if the caption explained how many object instances are averaged per category, how the top-10 co-occurring objects are selected, and whether the visualization uses training or validation data.
  3. [§4.3, Eq. (7)] The notation 'E_Z[Z] modeled as ≈ CrossAttn' is awkward and ambiguous; please rewrite to clearly define the approximation and the dimensions of the matrices.
  4. [§4.3] The phrase 'similar as section 4.3' appears inside the same section; it should refer to an earlier subsection or be rephrased.
  5. [Appendix B, Table 3] The baseline CHAIR-s in the ablation table is 29.5, whereas Table 1 reports 33.0 for the same LLaVA-7B model; please explain the discrepancy or use consistent settings.
  6. [§4.2, Fig. 5] The text says the intervention is applied to 'the final LLM transformer layer', but the appendix analyzes layer 40 and Figure 5 shows '× L'; please clarify whether the Causal Intervention Module is inserted into one layer or multiple layers.

Circularity Check

2 steps flagged · score 6.0 of 10

Causal intervention reduces to self-derived conditional expectations: confounder dictionaries are averages of the model's own biased hidden states, and Eq. (7)/Appendix A condition on h instead of using the unconditional P(Z), so the claimed backdoor adjustment is a fitted additive offset by construction.

  1. self definitional [Section 4.3 (Disentangled Visual Projector; Disentangled LLM Transformer) and Section 5.1 (Implementation Details)]
    "Each entry in D is the average post-projector visual representation of an object category, aggregated from 5,000 instances sampled from the LLaVA-Instruct dataset. ... The confounder dictionary estimation utilizes an intermediate checkpoint from a non-causal model trained for 0.1 epoch."

    The confounders Dv and Dt in Eqs. (4)-(5) are supposed to be common causes Z whose influence backdoor adjustment removes via P(Z=z). Instead, each dictionary entry is an average of the post-projector or LLM hidden states of the very non-causal model whose biased representations the paper aims to correct. The bias to be removed is therefore defined as a summary statistic of the biased model's own internal representations, and the intervention conditions on those self-derived centroids. The claimed blocking of h←Dv→W and h←Dt→W is not an adjustment over an independent confounder but a self-referential reshaping of h by a function of h's own category averages.

  2. other [Eq. (5), Eq. (7), Appendix A (Eq. 18 discussion)]
    "This formulation allows fixed confounder vectors d∈D to be contextually modulated by X, enabling input-specific confounding estimation while avoiding static representations. ... when computing ED[gd(D)] for both Dv and Dt, we condition d on h via E[D|h][gd(D)]."

    Backdoor adjustment in Eq. (2) requires the unconditional P(Z=z), and the NWGM approximation in Eq. (3) uses Ez[z]. The implementation in Eq. (7) computes CrossAttn(X,D,D), a query-dependent weighted average E[D|X], and Appendix A explicitly conditions on h. The claimed causal quantity P(W|do(h))≈P(W|h,Edv[dv],Edt[dt]) is therefore executed as P(W|h,E[Dv|h],E[Dt|h]), which has the same confounded form as the observational P(W|h) in Eq. (4) with P(dv|h) and P(dt|h). By construction, the 'do-intervention' reduces to an input-conditioned additive offset rather than a backdoor-blocked distribution.

full rationale

The paper's empirical hallucination reductions are real benchmark outcomes and are not mathematical identities, so this is not a fully circular paper. However, the central causal derivation has two self-referential reductions. First, the confounders Dv and Dt are defined as category-averaged hidden states of the non-causal LLaVA model that the method modifies; they are summaries of the very entangled representations the paper sets out to remove, so the 'deconfounding' conditions on a statistic of the target variable rather than an independent common cause. Second, the formal backdoor-adjustment derivation requires P(Z=z) and Ez[z], but the implementation uses CrossAttn(X,D,D) and explicitly E[D|h], i.e., the observational conditional expectation. The claimed P(W|do(h)) therefore reduces by construction to P(W|h,E[Dv|h],E[Dt|h]), the same confounded form as Eq. (4); the causal intervention is effectively a learned additive offset. These issues undermine the causal interpretation of the method, though the benchmark gains and visualization results remain independent empirical evidence. No load-bearing self-citation chain is present; the cited NWGM and backdoor-adjustment works are by other authors. Overall score 6: one or more central 'causal' claims reduce by construction, while the empirical content is not itself forced.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim rests on several assumptions that are not independently verified: the validity of the NWGM approximation for nonlinear cross-attention, the correctness of the proposed SCM, the unbiasedness of the frozen CLIP encoder, and the representativeness of confounder dictionaries computed from the model's own early training. The dictionaries themselves are fitted to the model's behavior, creating a circular dependency between the confounders and the intervention meant to correct them.

free parameters (3)
  • Confounder dictionary entries Dv, Dt = Averages of object-category hidden features from 5,000 LLaVA-Instruct samples using a 0.1-epoch checkpoint
    The causal intervention depends on these entries, which are computed from the model's own intermediate representations, so they are effectively fitted to the model's behavior rather than defined independently.
  • Batch size and learning rate for Causal-LLaVA pretraining = batch_size=256, lr=1e-3
    The authors doubled the batch size and halved the learning rate relative to the LLaVA baseline 'to accommodate additional projector parameters'; this changes the training dynamics and confounds the comparison with the baseline.
  • Number of confounder classes K and sample size 5000 = K unspecified, 5000 instances
    These hyperparameters are chosen without sensitivity analysis or justification, yet they determine the size and quality of the confounder dictionary.
assumptions (4)
  • ad hoc to paper The NWGM approximation is valid for the cross-attention modules used in the causal intervention.
    Eqs. (3) and (5) rely on a linearity assumption, but the implementation uses nonlinear cross-attention (Eqs. 7 and 10); the approximation error is not quantified and may be large.
  • standard math The backdoor adjustment formula applies to the proposed Structural Causal Model in Section 4.2.
    Application of standard causal inference, but the graph is asserted rather than learned or validated, so the set of confounders may be incomplete or incorrect.
  • domain assumption CLIP vision encoder representations are unaffected by dataset biases because the encoder is frozen and trained on diverse data.
    Section 4.2 uses this to justify not intervening on Fv; it is an untested assumption that could fail for domains unlike the CLIP training distribution.
  • ad hoc to paper Object category averages from a 0.1-epoch-trained model are representative confounders.
    The confounder dictionary is built from an intermediate checkpoint of a non-causal model; the choice of 0.1 epoch is arbitrary and not justified, and the resulting averages may be noisy or biased.
invented entities (2)
  • Visual confounder dictionary Dv
    purpose: Represents object co-occurrence biases to be removed via causal intervention
    Entries are averaged hidden features from the model itself; there is no external validation that these averages correspond to true confounders.
  • Textual confounder dictionary Dt
    purpose: Represents textual biases (e.g., language priors) to be removed
    Similarly built from the model's own features; no external benchmark or independent measurement is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/553JSEMA

@misc{pith2026250519474,
  author       = {Pith},
  title        = {Pith review of: Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/553JSEMA}},
  note         = {Machine review of arXiv:2505.19474}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong performance in visual understanding tasks, yet they often suffer from object hallucinations--generating descriptions of objects that are inconsistent with or entirely absent from the input. This issue is closely related to dataset biases, where frequent co-occurrences of objects lead to entangled semantic representations across modalities. As a result, models may erroneously activate object representations that are commonly associated with the input but not actually present. To address this, we propose a causality-driven disentanglement framework that mitigates hallucinations through causal intervention. Our approach includes a Causal-Driven Projector in the visual pathway and a Causal Intervention Module integrated into the final transformer layer of the language model. These components work together to reduce spurious correlations caused by biased training data. Experimental results show that our method significantly reduces hallucinations while maintaining strong performance on multiple multimodal benchmarks. Visualization analyses further confirm improved separability of object representations. The code is available at: https://github.com/IgniSavium/Causal-LLaVA

Figures

Figures reproduced from arXiv: 2505.19474 by the authors.

Figure 1
Figure 1. left: A hallucination example where an MLLM incorrectly describes non-existent objects (chairs and a coffee cup) based on co-occurrence patterns. middle: Representations of original LLaVA’s final LLM Transformer layer. The dining table (green dot, id=67) is surrounded by its top-10 most frequently co-occurring objects (red dots). right: Disentangled representations in our framework, with previously clustered objects… view at source ↗
Figure 2
Figure 2. Original LLaVA average object visual representations PCA after Vision Encoder (left), [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Original LLaVA average object visual hidden state distributions PCA in LLM layers 15 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Unified illustration of causal mechanisms in MLLMs. (a) Backdoor adjustment blocks con [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustration of our innovative disentangled MLLM model architecture. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Semantic distributions PCA after disentanglement at three stages: Projector output (left), [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: (Table-Centric:Visual) Layer-wise visual representations: CLIP Vision Encoder output [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: (Table-Centric:Visual) LLM layer 3 (left), layer 5 (middle), and layer 8 (right) visualizations [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: (Table-Centric:Visual) Mid-depth layers: layer 10 (left), layer 12 (middle), and layer 15 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: (Table-Centric:Visual) Transition phase layers: layer 20 (left), layer 25 (middle), and layer [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: (Table-Centric:Visual) Final prediction layers: layer 35 (left), layer 39 (middle), and layer [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: (Table-Centric:Textual) Textual representations after Projector (left), LLM layer 1 (middle), [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: (Table-Centric:Textual) Layer 10 (left), layer 37 (middle), and layer 40 (right) showing [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: (Car-Centric:Visual) Early layers (0-10) showing initial entanglement patterns. [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: (Car-Centric:Visual) Mid-depth layers (15-25) maintaining co-occurrence structures. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: (Car-Centric:Visual) Final layers (30-40) retaining co-occurrence patterns. [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: (Disentangled) Layer 0 (left) to layer 10 (right) showing progressive separation. [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: (Disentangled) Mid-depth layers (15-25) showing increasing spatial differentiation. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: (Disentangled) Final layers (30-40) achieving complete spatial separation. [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 52 canonical work pages

  1. [1]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015

    2015

  2. [2]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023. URLhttps://api.semanticscholar.org/CorpusID:261101015

    work page 2023

  3. [3]

    Hallucination of multimodal large language models: A survey.ArXiv, abs/2404.18930, 2024

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey.ArXiv, abs/2404.18930, 2024. URL https://api.semanticscholar.org/CorpusID:269449935

    arXiv 2024

  4. [4]

    Causal feature learning: an overview.Behav- iormetrika, 44:137–164, 2017

    Krzysztof Chalupka, Frederick Eberhardt, and Pietro Perona. Causal feature learning: an overview.Behav- iormetrika, 44:137–164, 2017. URLhttps://api.semanticscholar.org/CorpusID:40056398

    work page 2017

  5. [5]

    Shikra: Unleash- ing multimodal llm’s referential dialogue magic.ArXiv, abs/2306.15195, 2023

    Ke Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleash- ing multimodal llm’s referential dialogue magic.ArXiv, abs/2306.15195, 2023. URL https://api. semanticscholar.org/CorpusID:259262082

    arXiv 2023

  6. [6]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Al- bert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.ArXiv, abs/2305.06500, 2023. URL https://api.semanticscholar.org/ CorpusID:258615266

    arXiv 2023

  7. [7]

    Neural modular control for embodied question answering.ArXiv, abs/1810.11181, 2018

    Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering.ArXiv, abs/1810.11181, 2018. URL https://api.semanticscholar. org/CorpusID:53080692

    arXiv 2018

  8. [8]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.ArXiv, abs/2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models.ArXiv, abs/2306.13394, 2023. URL https://api. semanticscholar.org/CorpusID:259243928

    arXiv 2023

Show all 75 references
  1. [9]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering.International Journal of Computer Vision, 127:398 – 414, 2016

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering.International Journal of Computer Vision, 127:398 – 414, 2016. URLhttps://api.semanticscholar.org/Corp...

    2016

  2. [10]

    Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people.2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3608–3617, 2018. URL h...

    2018

  3. [11]

    Unbiased classification through bias-contrastive and bias-balanced learning

    Youngkyu Hong and Eunho Yang. Unbiased classification through bias-contrastive and bias-balanced learning. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, 2021. URLhttps://openreview.net/forum?id=2OqZZAqxnn

    2021

  4. [12]

    Ciem: Contrastive instruction eval- uation method for better instruction tuning.ArXiv, abs/2309.02301, 2023

    Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction eval- uation method for better instruction tuning.ArXiv, abs/2309.02301, 2023. URL https://api. semanticscholar.org/CorpusID:261557047

    2023 arXiv

  5. [13]

    Qidong Huang, Xiao wen Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Neng H. Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation.2024 IEEE/CVF Conference on Computer Vi...

    2024

  6. [14]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6693–6702, 2019. URL https://api.semanticscholar.org/CorpusID: 152282269

    2019

  7. [15]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700–6709, 2019

    2019

  8. [16]

    Introducing idefics: An open reproduction of state-of-the-art visual language model

    IDEFICS Team. Introducing idefics: An open reproduction of state-of-the-art visual language model. https://huggingface.co/blog/idefics, 2023. Accessed: 2025-05-13. 10

    2023

  9. [17]

    Vcoder: Versatile vision encoders for multimodal large language models.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27992–28002, 2023

    Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Versatile vision encoders for multimodal large language models.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27992–28002, 2023. URLhttps://api.semanticscholar.org/CorpusID:266521081

    2024

  10. [18]

    Causal inference meets deep learning: A comprehensive survey.Research, 7, 2024

    Licheng Jiao, Yuhan Wang, Xu Liu, Lingling Li, Fang Liu, Wenping Ma, Yuwei Guo, Puhua Chen, Shuyuan Yang, and Biao Hou. Causal inference meets deep learning: A comprehensive survey.Research, 7, 2024. URLhttps://api.semanticscholar.org/CorpusID:272211277

    2024

  11. [19]

    Unbiased learning-to-rank with biased feedback.Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, 2016

    Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. Unbiased learning-to-rank with biased feedback.Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, 2016. URLhttps://api.semanticscholar.org/CorpusID:300418

    2016

  12. [20]

    Deep visual-semantic alignments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015

    2015

  13. [21]

    Learning not to learn: Training deep neural networks with biased data.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9004–9012, 2018

    Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9004–9012, 2018. URL https://api.semanticscholar.org/Corpu...

    2019

  14. [22]

    Sophia Koepke, Cordelia Schmid, and Zeynep Akata

    Jae Myung Kim, A. Sophia Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious correlations for cross-modal retrieval.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2585–2595, 2023. URL https://api.semanticsch...

    2023

  15. [23]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization.CoRR, abs/1412.6980,

    arXiv

  16. [24]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yan- nis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connect- ing language and vision using crowdsourced dense image annotations...

    2016

  17. [25]

    V olcano: Mitigating multimodal hal- lucination through self-feedback guided revision

    Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. V olcano: Mitigating multimodal hal- lucination through self-feedback guided revision. InNorth American Chapter of the Association for Computational Linguistics, 2023. URL https://api.semanticscholar.org/CorpusID:265150082

    2023

  18. [26]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Li Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13872–...

    2024

  19. [27]

    Jiangtong Li, Li Niu, and Liqing Zhang. From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21241–21250, 2022. URL https://api.semantics...

    2022

  20. [28]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning, 2023. URLhttps://api.semanticscholar.org/CorpusID:256390509

    2023

  21. [29]

    Towards deconfounded image-text matching with causal inference.Proceedings of the 31st ACM International Conference on Multimedia, 2023

    Wenhui Li, Xinqi Su, Dan Song, Lanjun Wang, Kun Zhang, and Anan Liu. Towards deconfounded image-text matching with causal inference.Proceedings of the 31st ACM International Conference on Multimedia, 2023. URLhttps://api.semanticscholar.org/CorpusID:264492480

    2023

  22. [30]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. Evaluating object hallucination in large vision-language models. InConference on Empirical Methods in Natural Language Processing, 2023. URLhttps://api.semanticscholar.org/CorpusID:258740697

    2023

  23. [31]

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages...

    2024

  24. [32]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean Conference on Computer Vision, 2014. URLhttps://api.semanticscholar.org/CorpusID:14113767. 11

    2014

  25. [33]

    Show, deconfound and tell: Image captioning with causal inference.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18020–18029, 2022

    Bing Liu, Dong Wang, Xu Yang, Yong Zhou, Rui Yao, Zhiwen Shao, and Jiaqi Zhao. Show, deconfound and tell: Image captioning with causal inference.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18020–18029, 2022. URL https://api.semanticscholar...

    2022

  26. [34]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. InInternational Conference on Learning Representations, 2023. URLhttps://api.semanticscholar.org/CorpusID:259251834

    2023

  27. [35]

    A survey on hallucination in large vision-language models.ArXiv, abs/2402.00253,

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rong-Zhi Li, and Wei Peng. A survey on hallucination in large vision-language models.ArXiv, abs/2402.00253,

    arXiv

  28. [36]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296,

    2024

  29. [37]

    Visual instruction tuning.ArXiv, abs/2304.08485, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.ArXiv, abs/2304.08485, 2023. URLhttps://api.semanticscholar.org/CorpusID:258179774

    2023 arXiv

  30. [38]

    Mmbench: Is your multi-modal model an all-around player? InEuropean Conference on Computer Vision, 2023

    Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? InEuropean Conference on Computer Vision, 2023. URL https: //api.seman...

    2023

  31. [39]

    Discovering causal signals in images.2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 58–66, 2016

    David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Discovering causal signals in images.2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 58–66, 2016. URLhttps://api.semanticscholar.org/CorpusID:1847130

    2017

  32. [40]

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.ArXiv, abs/2209.09513, 2022. URL https://api.semanticscholar.org/Corp...

    2022 arXiv

  33. [41]

    The deep regression bayesian network and its applications: Proba- bilistic deep learning for computer vision.IEEE Signal Processing Magazine, 35:101–111, 2018

    Siqi Nie, Meng Zheng, and Qiang Ji. The deep regression bayesian network and its applications: Proba- bilistic deep learning for computer vision.IEEE Signal Processing Magazine, 35:101–111, 2018. URL https://api.semanticscholar.org/CorpusID:22901106

    2018

  34. [42]

    Counterfactual vqa: A cause-effect look at language bias.2021 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 12695–12705, 2020

    Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xiansheng Hua, and Ji rong Wen. Counterfactual vqa: A cause-effect look at language bias.2021 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 12695–12705, 2020. URL https://api.semanticscholar.org/...

    2021

  35. [43]

    Basic books, 2018

    Judea Pearl and Dana Mackenzie.The book of why: the new science of cause and effect. Basic books, 2018

    2018

  36. [44]

    Two causal principles for improving visual dialog.2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10857–10866, 2019

    Jiaxin Qi, Yulei Niu, Jianqiang Huang, and Hanwang Zhang. Two causal principles for improving visual dialog.2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10857–10866, 2019. URLhttps://api.semanticscholar.org/CorpusID:208267677

    2020

  37. [45]

    Object hallucination in image captioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. InConference on Empirical Methods in Natural Language Processing, 2018. URL https://api.semanticscholar.org/CorpusID:52176506

    2018

  38. [46]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders.ArXiv, abs/2408.15998, 2024

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal llms with mixture of ...

    2024 arXiv

  39. [47]

    Towards vqa models that can read.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8309–8318, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8309–8318, 2019. URLhttps://api.semanticscholar...

    2019

  40. [48]

    Heikkila, and Li Liu

    Shuzhou Sun, Shuaifeng Zhi, Qing Liao, J. Heikkila, and Li Liu. Unbiased scene graph generation via two-stage causal modeling.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45: 12562–12580, 2023. URLhttps://api.semanticscholar.org/CorpusID:259148437. 12

    2023

  41. [49]

    Aligning large multimodal models with factually augmented rlhf.ArXiv, abs/2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf.ArXiv, abs/2309.14525, 2023. URL https://api.sem...

    2023 arXiv

  42. [50]

    Unbiased scene graph generation from biased training.2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3713–3722, 2020

    Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training.2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3713–3722, 2020. URL https://api.semanticscholar.org/CorpusID:211532586

    2020

  43. [51]

    Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne.Journal of Machine Learning Research, 9:2579–2605, 2008. URLhttps://api.semanticscholar.org/CorpusID:5855042

    2008

  44. [52]

    Vigc: Visual instruction generation and correction

    Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiao wen Dong, Weijia Li, Wei Li, Jiaqi Wang, and Conghui He. Vigc: Visual instruction generation and correction. InAAAI Conference on Artificial Intelligence, 2023. URLhttps://api.semanticscholar.org/CorpusID:...

    2023

  45. [53]

    Visual commonsense r-cnn.2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10757–10767, 2020

    Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense r-cnn.2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10757–10767, 2020. URLhttps://api.semanticscholar.org/CorpusID:211532471

    2020

  46. [54]

    Advancing fine-grained visual understanding with multi-scale alignment in multi- modal models.ArXiv, abs/2411.09691, 2024

    Wei Wang, Zhaowei Li, Qi Xu, Linfeng Li, Yiqing Cai, Botian Jiang, Hang Song, Xingcan Hu, Pengyu Wang, and Li Xiao. Advancing fine-grained visual understanding with multi-scale alignment in multi- modal models.ArXiv, abs/2411.09691, 2024. URL https://api.semanticscholar.org/Co...

    2024 arXiv

  47. [55]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding.ArXiv, abs/2403.18715, 2024

    Xintong Wang, Jingheng Pan, Liang Ding, and Christian Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding.ArXiv, abs/2403.18715, 2024. URL https://api.semanticscholar.org/CorpusID:268724017

    2024 arXiv

  48. [56]

    Noiseboost: Alleviating hallucination with noise perturbation for multimodal large language models.ArXiv, abs/2405.20081, 2024

    Kai Wu, Boyuan Jiang, Zhengkai Jiang, Qingdong He, Donghao Luo, Shengzhi Wang, Qingwen Liu, and Chengjie Wang. Noiseboost: Alleviating hallucination with noise perturbation for multimodal large language models.ArXiv, abs/2405.20081, 2024. URL https://api.semanticscholar.org/ C...

    2024 arXiv

  49. [57]

    Courville, Ruslan Salakhutdinov, Richard S

    Ke Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. InInternational Conference on Machine Learning, 2015. URL https://api.semant...

    2015

  50. [58]

    Deconfounded image captioning: A causal retrospect

    Xu Yang, Hanwang Zhang, and Jianfei Cai. Deconfounded image captioning: A causal retrospect. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:12996–13010, 2020. URL https: //api.semanticscholar.org/CorpusID:212633945

    2020

  51. [59]

    Causal attention for vision-language tasks.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9842–9852, 2021

    Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for vision-language tasks.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9842–9852, 2021. URLhttps://api.semanticscholar.org/CorpusID:232135026

    2021

  52. [60]

    A survey on causal inference.ACM Transactions on Knowledge Discovery from Data (TKDD), 15:1 – 46, 2020

    Liuyi Yao, Zhixuan Chu, Sheng Li, Yaliang Li, Jing Gao, and Aidong Zhang. A survey on causal inference.ACM Transactions on Knowledge Discovery from Data (TKDD), 15:1 – 46, 2020. URL https://api.semanticscholar.org/CorpusID:211066551

    2020

  53. [61]

    Woodpecker: Hallucination correction for multimodal large language models.Sci

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xingguo Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models.Sci. China Inf. Sci., 67, 2023. URLhttps://api.semanticscholar.org/CorpusID:264439367

    2023

  54. [62]

    Ferret: Refer and ground anything anywhere at any granularity.ArXiv, abs/2310.07704, 2023

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih- Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity.ArXiv, abs/2310.07704, 2023. URLhttps://api.semanticscholar.org/CorpusID:263834718

    2023 arXiv

  55. [63]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.ArXiv, abs/2308.02490,

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.ArXiv, abs/2308.02490,

    arXiv

  56. [64]

    Interventional few-shot learning.ArXiv, abs/2009.13000, 2020

    Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xiansheng Hua. Interventional few-shot learning.ArXiv, abs/2009.13000, 2020. URLhttps://api.semanticscholar.org/CorpusID:221970817

    2009 arXiv

  57. [65]

    Halle-control: Controlling object hallucination in large multimodal models

    Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, and Manling Li. Halle-control: Controlling object hallucination in large multimodal models. 2023. URL https://api. semanticscholar.org/CorpusID:265609270. 13

    2023

  58. [66]

    Analyzing and mitigating object hallucination in large vision-language models.ArXiv, abs/2310.00754, 2023

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models.ArXiv, abs/2310.00754, 2023. URLhttps://api.semanticscholar.org/CorpusID:263334335

    2023 arXiv

  59. [67]

    URLhttps://api.semanticscholar.org/CorpusID:260611572

  60. [68]

    dining table

    Qingfu Zhu, Weinan Zhang, Ting Liu, and William Yang Wang. Counterfactual off-policy training for neural dialogue generation. InConference on Empirical Methods in Natural Language Processing, 2020. URLhttps://api.semanticscholar.org/CorpusID:265039635. 14 Supplementary Materia...

    2020

  61. [71]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models.ArXiv, abs/2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models.ArXiv, abs/2304.10592, 2023. URL https://api.semanticscholar.org/CorpusID:258291930

    2023 arXiv

  62. [73]

    7 and 8)

    Early-stage entanglement: Layers 0-5 maintain tight clustering driven by projector-induced biases, consistent with main text findings (figs. 7 and 8)

  63. [74]

    9 and 10)

    Mid-layer persistence: Surprisingly, strong co-occurrence patterns persist until layer 25 despite increasing semantic abstraction (figs. 9 and 10)

  64. [75]

    dining table

    Final-layer retention: Even at the final prediction stage (layer 40), dining tables retain significant entanglement with co-occurring objects (fig. 11), confirming the lasting impact of training data biases. These results reinforce our main conclusion that instruction data bia...

  65. [2014]

    URLhttps://api.semanticscholar.org/CorpusID:6628106

  66. [2023]

    URLhttps://api.semanticscholar.org/CorpusID:263672058

  67. [2024]

    URLhttps://api.semanticscholar.org/CorpusID:267365472

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.