Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

A 22,831-question benchmark shows multimodal models rarely say 'None of the above' when every option is false.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A new 22,831-question visual benchmark shows that major multimodal LLMs struggle to reject false answer options, often scoring near random when abstaining is the only correct response.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A useful NOTA-style hallucination benchmark with a plausible central finding, but the label audit and the random-baseline arithmetic need fixing before the headline numbers are cited. the 3 major comments →

arxiv 2509.09658 v2 pith:ZIEIYDAD submitted 2025-09-11 cs.CV

Measuring Epistemic Humility in Multimodal Large Language Models

Bingkui Tong , Jiaer Xia , Sifeng Shang , Kaiyang Zhou This is my paper
classification cs.CV
keywords hallucination benchmarkmultimodal large language modelepistemic humilityNone of the abovefalse-option rejectionabstentionvisual question answeringscene graph
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HumbleBench is a new benchmark of 22,831 multiple-choice visual questions, each with five options and a mandatory 'None of the above' option. The paper's goal is to measure a capability that recognition-focused benchmarks do not: noticing when none of the concrete answers is supported by the image and refusing to choose. Evaluated on 19 current multimodal models, the best score is about 73%, and in a stress test where 'None of the above' is the only correct answer most models fall to or below random-guess level. The paper concludes that current models systematically hallucinate rather than abstain, and that standard accuracy numbers mask this failure.

Core claim

The central claim is that, in a forced-choice setting where the correct response is to reject all candidate answers, current multimodal large language models overwhelmingly choose a false concrete option instead of selecting 'None of the above'. The paper demonstrates this through HumbleBench: models plateau in the low 70s on normal questions, then collapse on a version of the benchmark where the 'None of the above' option is the only valid answer—several models score 0%, and the best reaches about 61%. When images are replaced with Gaussian noise so that all concrete options are necessarily wrong, models' behavior splits sharply: some still answer from text priors (e.g., naming a sky color)

What carries the argument

The benchmark's core mechanism is the universal 'None of the above' option in a five-choice format. Questions are built from a panoptic scene-graph dataset, which supplies object and relation labels; attribute descriptions are added by prompting a visual instruction-following model; and a large language model generates the questions and distractors. A manual filtering stage keeps 22,831 of roughly 42,000 generated questions, correcting or deleting the rest. Two stress tests isolate the target behavior: HumbleBench-E deletes every correct non-E answer so 'None of the above' is the only correct option, and HumbleBench-GN replaces all images with noise so no concrete option can be right. The co

Load-bearing premise

The benchmark's ground-truth answers are correct, which depends on the scene-graph annotations, the machine-generated attribute descriptions, and the manual filtering being accurate and unbiased; the paper reports no inter-annotator agreement or follow-up audit of residual labeling errors.

What would settle it

Have independent annotators re-examine a random sample of, say, 500 HumbleBench questions with full image access and record agreed correct answers, especially NOTA cases; if a substantial share of answers are wrong or ambiguous, the accuracy gap and abstention-failure conclusion could be artifacts of label noise rather than model behavior.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If HumbleBench's measurement is valid, current MLLMs are not safe to deploy in settings where the supplied answer set may be wrong; they will fabricate a choice rather than abstain.
  • Recognition-style benchmarks systematically overstate reliability, because they never present a question with no valid concrete answer.
  • Model scale and reasoning-oriented fine-tuning do not by themselves solve false-option rejection; some smaller models outperform larger ones, and some reasoning models underperform their base models.
  • The Gaussian-noise results imply that many models lean on language priors instead of visual grounding when the image is uninformative, a distinct failure mode worth targeting in training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to add abstention-aware fine-tuning—rewarding correct 'None of the above' responses on HumbleBench-style data—to see whether the abstention skill transfers to other benchmarks or remains isolated.
  • The same forced-choice-with-abstention format could be ported to high-stakes visual domains such as medical imaging or driving scenes, where answering confidently with a false option is more harmful than declining to answer.
  • The large spread between standard accuracy and NOTA-only accuracy across models suggests that recognition and abstention may be partially independent skills; a model could be strong at one and weak at the other.
  • Because the benchmark's ground truth is generated by models and then filtered by humans, an independent label audit would clarify how much of the reported failure is model behavior versus residual annotation error.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces HumbleBench, a 22,831-question multiple-choice benchmark built from the Panoptic Scene Graph (PSG) dataset, covering object, relation, and attribute questions, each with a mandatory "None of the above" option. It reports two stress tests: HumbleBench-E, where the originally correct non-E option is removed so E is always correct, and HumbleBench-GN, where images are replaced by Gaussian noise. The authors evaluate 19 MLLMs and find that the best models reach only ~72–73% original accuracy, while most models perform at or below chance on the NOTA stress tests. The central claim is that current MLLMs overwhelmingly select false options rather than abstaining, revealing a failure of false-option rejection and visual grounding.

Significance. If the benchmark labels are valid, this is a useful and timely complement to existing hallucination benchmarks: it is the largest dataset of its kind, publicly released, spans three hallucination types, and explicitly tests abstention rather than only recognition. The stress-test design is a genuine contribution, and the finding that many strong models collapse when NOTA becomes the only correct answer is striking. The paper ships code and data, which strengthens reproducibility. However, the validity of every quantitative claim depends on ground-truth label quality, particularly for attribute questions, and this is currently not sufficiently evidenced. The paper also does not address possible train/test contamination arising from the COCO/PSG image source. These gaps make the central result plausible but not yet fully established.

major comments (3)
  1. [§2.1, Table 2, Appendix A.4] The ground-truth labels, especially the 35.39% attribute questions, are the linchpin of the benchmark. Attribute labels are generated by InstructBLIP, a hallucination-prone model, and the only safeguard is the manual filtering process. The paper reports that 45.44% of candidates were deleted and 10.82% modified, but it does not report annotator count, expertise, independence, inter-annotator agreement, or a post-filter residual-error audit. A systematic residual error in attribute labels would directly break HumbleBench-E: if the original correct option is removed but another option is actually true, the ground truth becomes wrong, and a model selecting that option would be misclassified as hallucinating. Please add a residual-error audit on a random sample (e.g., double annotation with agreement metrics) and quantify the possible label-noise contribution to the reported accuracies.
  2. [§2.1, data source; COCO contamination] PSG is built from COCO images, and most evaluated MLLMs are pretrained on large image-text corpora that include COCO. The paper does not analyze whether HumbleBench images or derived captions appear in the training data of the evaluated models. This overlap could inflate absolute accuracy and distort comparisons on HumbleBench, and it leaves open the possibility that some 'correct' answers are retrieved from memory rather than computed from the image. Although the GN stress test is less sensitive to this, the original benchmark and the NOTA stress-test construction are affected. Please report a leakage analysis (e.g., near-duplicate detection against known training corpora) or performance stratified by image novelty, and otherwise bound this effect.
  3. [§3.2, Fig. 4; §3.3, Fig. 5] The 'random guess' baseline of 20% is not a behaviorally meaningful null model for the stress tests. In the original HumbleBench only 14.57% of correct answers are E, so models may have a strong prior against selecting E; low E-selection in HumbleBench-E could reflect this label prior rather than an inability to reject false options. Conversely, a model that always echoes E would score 100% on HumbleBench-GN without any visual grounding, so high GN accuracy alone does not demonstrate visual faithfulness. Please report the models' E-selection rate on the original HumbleBench as a control, and analyze choice-position effects (e.g., permute E's position) before concluding that low accuracy reflects hallucination rather than decision bias.
minor comments (6)
  1. [§3.2, Fig. 4] The text and figure caption interchangeably call HumbleBench-E accuracy 'NOTA accuracy' and 'E-selection rate.' Define the metric unambiguously and use one term throughout.
  2. [Fig. 2] The caption says 'Distribution of answer choices,' but the plot shows the distribution of correct answers in the benchmark, not model predictions. Rename to avoid confusion.
  3. [Appendix A.1] The prompt uses 'Category' for object-existence/count types while the main text uses 'Object' for that category. Align the terminology.
  4. [Appendix A.2] A single static noise image is used for all HumbleBench-GN questions. Using one image rather than multiple random draws introduces a potential confound; consider sampling a new noise image per question or reporting sensitivity.
  5. [Table 3] No confidence intervals or repeated runs are reported. For the small gaps that motivate the 'reasoning models do not always work better' discussion (e.g., LLaVA-CoT vs. LLaMA-3.2), include variance estimates or significance tests.
  6. [Abstract and §5] The paper uses 'epistemic humility' interchangeably with 'abstention.' A paragraph distinguishing the normative concept from the operational metric would avoid overclaiming.

Circularity Check

0 steps flagged

No significant circularity: HumbleBench is an empirical benchmark construction with direct measurements; no derivation reduces to its inputs.

full rationale

The paper does not contain a mathematical derivation or fitted-prediction chain. HumbleBench is constructed from external data sources (PSG scene graphs, InstructBLIP attribute descriptions, GPT-4-Turbo question generation) and human filtering, and the reported accuracies are direct behavioral measurements on that benchmark. The 'None of the above' design and the stress tests (HumbleBench-E and HumbleBench-GN) define the correct answer by construction, but that is an experimental protocol rather than a circular derivation: the models' responses are not forced by the construction, and the paper's empirical findings (e.g., low NOTA accuracy) are contingent observations. The only notable author-overlap citation is PSG, whose coauthor list includes Kaiyang Zhou; however, PSG is an external, publicly available dataset used as a data source, not as a self-supporting theorem or uniqueness claim. Concerns about attribute-label quality or manual-filtering rigor are benchmark-validity risks, not circularity. No load-bearing step reduces to an input of the paper, so the appropriate score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The benchmark's validity rests on the accuracy of PSG annotations, InstructBLIP attributes, and human filtering. The paper does not quantify residual label noise nor test for training-data leakage, so these are the main unverified pillars.

axioms (5)
  • domain assumption PSG panoptic scene graph annotations provide accurate ground-truth object and relation labels.
    Used as the source of ground truth for objects and relations; if PSG labels are wrong, benchmark answers are wrong. Section 2.1 'Data Source'.
  • domain assumption InstructBLIP-generated attribute descriptions are sufficiently accurate to serve as attribute ground truth after manual filtering.
    Attribute labels are not human-annotated per object; they come from InstructBLIP. Manual filtering checks resulting questions but may not independently verify every attribute against the image. Section 2.1 'Information Extraction'.
  • domain assumption Human annotators reliably correct or delete erroneous questions, with no systematic bias.
    The quality control step is described but no inter-annotator agreement or error rate is reported. Section 2.1 'Manual Filtering' and Table 2.
  • domain assumption The evaluated models have not previously memorized the PSG/COCO images or their annotations during pretraining.
    The benchmark images come from a public dataset (COCO via PSG) widely used in MLLM training; no contamination analysis is provided. Section 2.1 'Data Source' and Section 3 'Evaluation'.
  • domain assumption Answering 'None of the above' in a forced-choice setting is a valid operationalization of epistemic humility/abstention.
    The paper equates NOTA selection with humility; this is a conceptual assumption about what the benchmark measures. Section 1 and abstract.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Epistemic Humility in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/ZIEIYDAD

@misc{pith2026250909658,
  author       = {Pith},
  title        = {Pith review of: Measuring Epistemic Humility in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZIEIYDAD}},
  note         = {Machine review of arXiv:2509.09658}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Hallucinations in multimodal large language models (MLLMs) -- where the model generates content inconsistent with the input image -- pose significant risks in real-world applications, from misinformation in visual question answering to unsafe errors in decision-making. Existing benchmarks primarily test recognition accuracy, i.e., evaluating whether models can select the correct answer among distractors. This overlooks another important capability for trustworthy AI: recognizing when none of the provided options is supported by the image and abstaining from committing to a false choice, a humility-related behavior. We present HumbleBench, a new hallucination benchmark designed to evaluate false-option rejection in MLLMs under a forced-choice multiple-choice setting with a ``None of the above'' option. Built from a panoptic scene graph dataset, we leverage fine-grained scene graph annotations for objects and relations, use candidate attribute cues, and prompt GPT-4-Turbo to generate multiple-choice questions, followed by a rigorous manual filtering process. Each question includes a ``None of the above'' option, requiring models not only to recognize correct visual information but also to identify when no provided answer is valid. We evaluate a variety of state-of-the-art MLLMs -- including general-purpose, specialized reasoning, and proprietary models -- on HumbleBench and report empirical findings for the community. By incorporating explicit false-option rejection, HumbleBench fills a key gap in current evaluation suites by assessing a narrower but important abstention-oriented behavior that is relevant to trustworthy multimodal reasoning. Our code and dataset are released publicly and can be accessed at \href{https://github.com/maifoundations/HumbleBench}{https://github.com/maifoundations/HumbleBench}.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research

    cs.HC 2026-04 unverdicted novelty 5.0

    AVA is a specialized GenAI platform for development policy research that provides verifiable syntheses from World Bank reports and is associated with 2.4-3.9 hours of weekly time savings in a large-scale user evaluation.

Reference graph

Works this paper leans on

48 extracted references · 23 linked inside Pith · cited by 1 Pith paper

  1. [1]

    https://arxiv.org/ abs/2306.13394

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., Ji, R.: MME: A Comprehensive 12 Evaluation Benchmark for Multimodal Large Language Models (2024). https://arxiv.org/ abs/2306.13394

    Pith/arXiv arXiv 2024

  2. [2]

    arXiv preprint arXiv:2009.03300 (2020)

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.: Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300 (2020)

    Pith/arXiv arXiv 2009

  3. [3]

    https://arxiv.org/abs/2509

    Kalai, A.T., Nachum, O., Vempala, S.S., Zhang, E.: Why Language Models Hal- lucinate (2025). https://arxiv.org/abs/2509. 04664

    2025

  4. [4]

    Philosophy and Phenomenological Research 94(3), 509–539 (2017)

    Whitcomb, D., Battaly, H., Baehr, J., Howard-Snyder, D.: Intellectual humility. Philosophy and Phenomenological Research 94(3), 509–539 (2017)

    2017

  5. [5]

    The Journal of Positive Psychology 15(2), 155–170 (2020)

    Krumrei-Mancuso, E.J., Haggard, M.C., LaBouff, J.P., Rowatt, W.C.: Links between intellectual humility and acquiring knowl- edge. The Journal of Positive Psychology 15(2), 155–170 (2020)

    2020

  6. [6]

    In: European Conference on Com- puter Vision, pp

    Yang, J., Ang, Y.Z., Guo, Z., Zhou, K., Zhang, W., Liu, Z.: Panoptic scene graph gen- eration. In: European Conference on Com- puter Vision, pp. 178–196 (2022). Springer

    2022

  7. [7]

    Advances in neural information processing systems36, 49250–49267 (2023)

    Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in neural information processing systems36, 49250–49267 (2023)

    2023

  8. [8]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    https://llava-vl.github.io/blog/ 2024-01-30-llava-next/

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: LLaV A-NeXT: Improved reasoning, OCR, and world knowl- edge (2024). https://llava-vl.github.io/blog/ 2024-01-30-llava-next/

    2024

  10. [10]

    In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pp

    Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J.S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L.,et al.: Molmo and pixmo: Open weights and open data for state- of-the-art vision-language models. In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pp. 91–104 (2025)

    2025

  11. [11]

    arXiv preprint arXiv:2412.10302 (2024)

    Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al.: Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 (2024)

    Pith/arXiv arXiv 2024

  12. [12]

    arXiv preprint arXiv:2504.10479 (2025)

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open- source multimodal models. arXiv preprint arXiv:2504.10479 (2025)

    Pith/arXiv arXiv 2025

  13. [13]

    arXiv e-prints, 2407 (2024)

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv e-prints, 2407 (2024)

    2024

  14. [14]

    arXiv preprint arXiv:2412.08905 (2024)

    Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R.J., Javaheripi, M., Kauffmann, P., et al.: Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    arXiv preprint arXiv:2503.19786 (2025)

    Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Mate- jovicova, T., Ram´ e, A., Rivi` ere, M., et al.: Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025)

    Pith/arXiv arXiv 2025

  16. [16]

    Advances in Neural Information Processing Systems37, 87310–87356 (2024)

    Tong, P., Brown, E., Wu, P., Woo, S., IYER, A.J.V., Akula, S.C., Yang, S., Yang, J., Mid- depogu, M., Wang, Z.,et al.: Cambrian-1: A fully open, vision-centric exploration of multi- modal llms. Advances in Neural Information Processing Systems37, 87310–87356 (2024)

    2024

  17. [17]

    arXiv preprint arXiv:2410.07073 (2024)

    Agrawal, P., Antoniak, S., Hanna, E.B., Bout, B., Chaplot, D., Chudnovsky, J., Costa, D., De Monicault, B., Garg, S., Gervet, T., et al.: Pixtral 12b. arXiv preprint arXiv:2410.07073 (2024)

    Pith/arXiv arXiv 2024

  18. [18]

    arXiv preprint arXiv:2408.12637 (2024)

    Lauren¸ con, H., Marafioti, A., Sanh, V., Tron- chon, L.: Building and better understanding 13 vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637 (2024)

    Pith/arXiv arXiv 2024

  19. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., Han, S.: Vila: On pre-training for visual language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689– 26699 (2024)

    2024

  20. [20]

    arXiv preprint arXiv:2405.20797 (2024)

    Lu, S., Li, Y., Chen, Q.-G., Xu, Z., Luo, W., Zhang, K., Ye, H.-J.: Ovis: Structural embedding alignment for multi- modal large language model. arXiv preprint arXiv:2405.20797 (2024)

    Pith/arXiv arXiv 2024

  21. [21]

    arXiv preprint arXiv:2412.18319 (2024)

    Yao, H., Huang, J., Wu, W., Zhang, J., Wang, Y., Liu, S., Wang, Y., Song, Y., Feng, H., Shen, L., et al.: Mulberry: Empowering mllm with o1-like reasoning and reflection via col- lective monte carlo tree search. arXiv preprint arXiv:2412.18319 (2024)

    Pith/arXiv arXiv 2024

  22. [22]

    arXiv preprint arXiv:2503.10615 (2025)

    Yang, Y., He, X., Pan, H., Jiang, X., Deng, Y., Yang, X., Lu, H., Yin, D., Rao, F., Zhu, M., et al.: R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615 (2025)

    Pith/arXiv arXiv 2025

  23. [23]

    arXiv preprint arXiv:2505.14677 (2025)

    Xia, J., Zang, Y., Gao, P., Li, Y., Zhou, K.: Visionary-r1: Mitigating shortcuts in visual reasoning with reinforcement learning. arXiv preprint arXiv:2505.14677 (2025)

    arXiv 2025

  24. [24]

    arXiv preprint arXiv:2411.10440 (2024)

    Xu, G., Jin, P., Li, H., Song, Y., Sun, L., Yuan, L.: Llava-cot: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440 (2024)

    Pith/arXiv arXiv 2024

  25. [25]

    arXiv preprint arXiv:2503.12937 (2025)

    Zhang, J., Huang, J., Yao, H., Liu, S., Zhang, X., Lu, S., Tao, D.: R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937 (2025)

    Pith/arXiv arXiv 2025

  26. [26]

    arXiv preprint arXiv:2507.01006 (2025)

    Hong, W., Yu, W., Gu, X., Wang, G., Gan, G., Tang, H., Cheng, J., Qi, J., Ji, J., Pan, L., et al.: Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scal- able reinforcement learning. arXiv preprint arXiv:2507.01006 (2025)

    Pith/arXiv arXiv 2025

  27. [27]

    arXiv preprint arXiv:2408.03314 (2024)

    Snell, C., Lee, J., Xu, K., Kumar, A.: Scal- ing llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 (2024)

    Pith/arXiv arXiv 2024

  28. [28]

    arXiv preprint arXiv:2306.14565 (2023)

    Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., Wang, L.: Mitigating hallucination in large multi-modal models via robust instruc- tion tuning. arXiv preprint arXiv:2306.14565 (2023)

    Pith/arXiv arXiv 2023

  29. [29]

    Zhai, B., Yang, S., Zhao, X., Xu, C., Shen, S., Zhao, D., Keutzer, K., Li, M., Yan, T., Fan, X.: Halle-switch: Rethinking and control- ling object existence hallucinations in large vision-language models for detailed caption (2023)

    2023

  30. [30]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., Bai, X.: Monkey: Image resolution and text label are impor- tant things for large multi-modal models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26763–26773 (2024)

    2024

  31. [31]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Jain, J., Yang, J., Shi, H.: Vcoder: Versatile vision encoders for multimodal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27992–28002 (2024)

    2024

  32. [32]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Jiang, C., Xu, H., Dong, M., Chen, J., Ye, W., Yan, M., Ye, Q., Zhang, J., Huang, F., Zhang, S.: Hallucination aug- mented contrastive learning for multimodal large language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27036–27046 (2024)

    2024

  33. [33]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L.,et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185–24198 (2024) 14

    2024

  34. [34]

    In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp

    Favero, A., Zancato, L., Trager, M., Choud- hary, S., Perera, P., Achille, A., Swami- nathan, A., Soatto, S.: Multi-modal halluci- nation control by visual information ground- ing. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 14303–14312 (2024)

    2024

  35. [35]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., Bing, L.: Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872–13882 (2024)

    2024

  36. [36]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., Yu, N.: Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418–13427 (2024)

    2024

  37. [37]

    arXiv preprint arXiv:2403.18715 (2024)

    Wang, X., Pan, J., Ding, L., Biemann, C.: Mitigating hallucinations in large vision- language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715 (2024)

    Pith/arXiv arXiv 2024

  38. [38]

    ACM computing surveys 55(12), 1–38 (2023)

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of hallucination in natural lan- guage generation. ACM computing surveys 55(12), 1–38 (2023)

    2023

  39. [39]

    arXiv preprint arXiv:2402.00253 (2024)

    Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L., Li, R., Peng, W.: A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253 (2024)

    Pith/arXiv arXiv 2024

  40. [40]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Kaul, P., Li, Z., Yang, H., Dukler, Y., Swami- nathan, A., Taylor, C., Soatto, S.: Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27228–27238 (2024)

    2024

  41. [41]

    In: European Conference on Computer Vision, pp

    Wang, Z., Bingham, G., Yu, A.W., Le, Q.V., Luong, T., Ghiasi, G.: Haloquest: A visual hallucination dataset for advancing multi- modal reasoning. In: European Conference on Computer Vision, pp. 288–304 (2024). Springer

    2024

  42. [42]

    arXiv preprint arXiv:2408.09429 (2024)

    Zheng, K., Chen, J., Yan, Y., Zou, X., Hu, X.: Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models. arXiv preprint arXiv:2408.09429 (2024)

    Pith/arXiv arXiv 2024

  43. [43]

    arXiv preprint arXiv:2305.10355 (2023)

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.-R.: Evaluating object halluci- nation in large vision-language models. arXiv preprint arXiv:2305.10355 (2023)

    Pith/arXiv arXiv 2023

  44. [44]

    arXiv preprint arXiv:2503.23573 (2025)

    Augustin, M., Neuhaus, Y., Hein, M.: Dash: Detection and assessment of system- atic hallucinations of vlms. arXiv preprint arXiv:2503.23573 (2025)

    Pith/arXiv arXiv 2025

  45. [45]

    CoRR (2024)

    Liu, J., Fu, Y., Xie, R., Xie, R., Sun, X., Lian, F., Kang, Z., Li, X.: Phd: A prompted visual hallucination evaluation dataset. CoRR (2024)

    2024

  46. [46]

    In: Proceedings of the 32nd ACM International Conference on Multime- dia, pp

    Ding, P., Wu, J., Kuang, J., Ma, D., Cao, X., Cai, X., Chen, S., Chen, J., Huang, S.: Hallu-pi: Evaluating hallucination in multi- modal large language models within per- turbed inputs. In: Proceedings of the 32nd ACM International Conference on Multime- dia, pp. 10707–10715 (2024)

    2024

  47. [47]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y.,et al.: Hallusionbench: an advanced diagnostic suite for entangled lan- guage hallucination and visual illusion in large vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375– 14385 (2024)

    2024

  48. [48]

    arXiv preprint arXiv:2410.09962 (2024) 15

    Qiu, H., Huang, J., Gao, P., Qi, Q., Zhang, X., Shao, L., Lu, S.: Longhalqa: Long- context hallucination evaluation for multi- modal large language models. arXiv preprint arXiv:2410.09962 (2024) 15

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.