Pith. sign in

REVIEW 10 cited by

Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.03287 v2 pith:4P7EJVN5 submitted 2023-11-06 cs.LG cs.CLcs.CV

Chenhang Cui , Yiyang Zhou , Xinyu Yang , Shirley Wu , Linjun Zhang , James Zou , Huaxiu Yao This is my paper · ORCID
classification cs.LGcs.CLcs.CV
keywords gpt-4visionbiasinterferencechallengesimagesmodelsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While GPT-4V(ision) impressively models both visual and textual information simultaneously, it's hallucination behavior has not been systematically assessed. To bridge this gap, we introduce a new benchmark, namely, the Bias and Interference Challenges in Visual Language Models (Bingo). This benchmark is designed to evaluate and shed light on the two common types of hallucinations in visual language models: bias and interference. Here, bias refers to the model's tendency to hallucinate certain types of responses, possibly due to imbalance in its training data. Interference pertains to scenarios where the judgment of GPT-4V(ision) can be disrupted due to how the text prompt is phrased or how the input image is presented. We identify a notable regional bias, whereby GPT-4V(ision) is better at interpreting Western images or images with English writing compared to images from other countries or containing text in other languages. Moreover, GPT-4V(ision) is vulnerable to leading questions and is often confused when interpreting multiple images together. Popular mitigation approaches, such as self-correction and chain-of-thought reasoning, are not effective in resolving these challenges. We also identified similar biases and interference vulnerabilities with LLaVA and Bard. Our results characterize the hallucination challenges in GPT-4V(ision) and state-of-the-art visual-language models, and highlight the need for new solutions. The Bingo benchmark is available at https://github.com/gzcch/Bingo.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 13 citations worldwide. Full citation record

  1. Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Context-guided multi-turn interviews reveal more VLM hallucinations than static benchmarks, and those hallucinations increase with conversational history and false-premise questions.

  2. More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Reasoning models trade visual grounding for language-based inference, and this paper measures that trade-off with a new metric and benchmark.

  3. MolSight: A Graph-Aware Vision-Language Model for Unified Chemical Image Understanding

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    MolSight integrates a Molecular Topology Module and Molecular Grounding Module into VLMs to enhance molecular image understanding and claims to outperform prior models on chemical visual tasks.

  4. Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    Multimodal reasoning models hallucinate at high-entropy cognitive bifurcation points due to loss of visual semantic anchoring, and the V-STAR training paradigm with HVAR rewards and FRM reflection mitigates this by re...

  5. SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models

    cs.AI 2025-08 unverdicted novelty 6.0 of 10

    SEAM measures VLM reasoning consistency across modalities using paired semantically equivalent textual and visual notations, and finds systematic vision-language imbalance.

  6. From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Text

    cs.CL 2025-08 conditional novelty 6.0 of 10

    Vision-language models produce more positive chart summaries for high-income countries than for middle- or low-income countries, and a simple positive prompt only partly fixes the bias.

  7. VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A new benchmark with edited-image question pairs shows that multimodal reasoning models lose accuracy when visual cues change, suggesting their reasoning is often not faithfully tied to the image.

  8. Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    RVCD uses YOLO detections and retrieved single-concept AI images to adjust LVLM logits at decode time, cutting CHAIR hallucination rates by roughly half versus prior contrastive decoding baselines.

  9. X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

    cs.LG 2026-07 conditional novelty 5.0 of 10

    X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.

  10. MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Grouping instruction-tuning datasets by redundancy, uniqueness, or synergy of text-image interaction improves vision-language model accuracy over single-task and unselective multi-task tuning.

Pith tools