Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hyper-detailed image captions can be made more factual by splitting them into atomic claims, verifying each against the image, and rewriting the caption to keep only the verified claims.

desk verdict Solid empirical paper with a useful training-free caption correction method and a coverage benchmark; the main caveat is that the factuality metric and the method share a decomposition mechanism, but the metric has independent human validation and the limitations are honestly disclosed. read the letter →

arxiv 2412.15484 v4 pith:A6QQDHT3 submitted 2024-12-20 cs.CV

Saehyung Lee , Seunghyun Yoon , Trung Bui , Jing Shi , Sungroh Yoon This is my paper · ORCID
classification cs.CV
keywords hyper-detailedimagecaptioninghallucinationdetectionmultiagentsystematomicpropositionverificationfactualityevaluationcoveragemultimodallargelanguagemodelsVQAbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that hallucinations in long, hyper-detailed image captions are less a decoding problem than a detection problem: once a caption runs hundreds of tokens, standard hallucination scores stop working and the model drifts from the image. Its proposed system, CapMAS, has an LLM decompose a caption into atomic propositions, an MLLM fact-check each proposition against the image, and an LLM rewrite the caption from only the verified propositions. The paper also introduces a factuality metric that uses GPT-4o to decompose captions and judge each claim against both the image and a reference caption, plus a coverage benchmark of nearly 20,000 human-refined multiple-choice questions answered from the caption alone. It reports that this metric matches human factuality judgments better than FaithScore and FactScore, and that CapMAS improves factuality for every captioner tested, including GPT-4V, while sacrificing little coverage. A separate finding is that rankings on VQA benchmarks do not predict detailed-captioning ability, which calls into question VQA-centric model evaluation.

What carries the argument

The load-bearing object is the atomic proposition, a claim that must be either true or false. CapMAS's pipeline surrounds it with three agents: a decomposer LLM that splits a caption into propositions, a fact-checker MLLM that converts each proposition into a 'True or False?' question and reads hallucination from the probability gap between the True and False tokens, and a corrector LLM that removes unverified content. The threshold $\pi$ controls a loose factuality-coverage trade-off, with smaller $\pi$ giving stricter fact-checking and lower coverage. For evaluation, the paper adds a GPT-4o-based factuality score that reuses the same decomposition-plus-verification logic with both image and reference caption, and a coverage score computed by having an LLM answer human-refined multiple-choice questions using only the caption.

What would settle it

Take a fresh set of several hundred detailed captions from multiple MLLMs, have human annotators label every atomic proposition as true or false, and compute the proposed metric's per-caption scores from those labels without excluding close pairs. If the Spearman correlation with human preferences drops to the level of FaithScore or FactScore, or if proposition-level agreement with humans is poor, the central evaluation claim fails; the same experiment would also show whether the threshold $\pi$ generalizes beyond five tuning examples.

Watch

Extended reading notes

Core claim

The central discovery is that decomposing a long caption into atomic propositions and verifying each proposition in isolation exposes hallucinations that token-confidence and consistency methods miss, especially after the 192nd token. CapMAS operationalizes this with an LLM decomposer, an MLLM fact-checker that scores each unit by $-\log(\min(p(\text{T}|x,Q(u))-p(\text{F}|x,Q(u)),\epsilon))$, a threshold $\pi$ that separates true from false units, and an LLM corrector that rewrites the caption using only the true units. The same decomposition idea drives the evaluation metric, where GPT-4o judges atomic propositions against both image and reference; the paper reports a Spearman correlation of 70.2 with human factuality preferences, versus 62.5 for FaithScore and 67.9 for FactScore. The paper further claims that decoding-based and training-based hallucination fixes validated on VQA can fail or even backfire on detailed captions, while CapMAS improves factuality for LLaVA, InternVL, and GPT-4V.

Load-bearing premise

The paper's headline results assume that GPT-4o can decompose any caption into atomic propositions and judge each one correctly against the image and a reference caption; this was validated only on 100 DOCCI images with caption pairs of similar factuality excluded, and the CapMAS threshold $\pi$ was tuned on five examples.

Editorial extensions

If this is right

  • Factuality fixes validated on short-answer VQA tasks should be re-audited on long-form captioning; the paper shows that applying VCD actually reduces factuality there.
  • CapMAS can be dropped onto any captioning model without training, including a closed model like GPT-4V, and it raises factuality while holding coverage roughly constant.
  • Model leaderboards built from VQA benchmarks should not be read as captioning ability; Idefics2 and MiniCPM-V rank well on VQA but poorly on detailed captioning relative to GPT-4V.
  • Factuality and coverage are separable evaluation axes with a loose trade-off controlled by one threshold, so a caption can be fully factual and still miss most of the image's content.
  • The proposed factuality metric should be preferred over reference-only or image-only metrics because it combines both sources and avoids the stylistic bias of reference-only scoring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same decompose-verify-rewrite loop could be applied to other long multimodal generations, such as video descriptions or dense scene graphs, where the latter half of the output is the most hallucination-prone.
  • A direct testable extension is to measure whether caption-only accuracy on the proposed coverage benchmark predicts downstream usability for blind users, connecting coverage scores to the accessibility outcome the paper motivates but does not measure.
  • Because the factuality metric's judge is GPT-4o, the method is tied to a closed model; an obvious extension is to train a smaller open verifier on the paper's atomic-proposition labels and check whether the correlation with human judgments is preserved.
  • The paper's dissociation between VQA ranking and captioning ranking suggests that benchmark suites should add long-form generation tasks; one could test this by checking whether composite rankings that include captioning scores predict human preference better than VQA-only rankings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes CapMAS, a training-free multiagent pipeline that decomposes a hyper-detailed image caption into atomic propositions, verifies each proposition with an MLLM against the image, and uses an LLM to rewrite the caption so that only verified content is retained. The paper also introduces a factuality evaluation metric built on GPT-4o that decomposes captions into atomic propositions and judges each against the image and a reference caption, and a coverage benchmark of multiple-choice questions derived from IIW-400 with human-validated answers. Experiments claim that CapMAS improves factuality across several MLLMs including GPT-4V, that the proposed metric correlates better with human judgments than FaithScore and FactScore, that existing hallucination mitigation methods underperform on detailed captions, and that VQA-centric benchmark rankings do not predict detailed captioning ability.

Significance. If the results hold, the paper would make three useful contributions: a plug-and-play, training-free correction method that appears to reduce hallucination in long captions; a dual factuality/coverage evaluation framework with a new human-annotated QA benchmark; and evidence that VQA-centric benchmarking is insufficient for assessing detailed captioning. The code and data are released, and the metric is validated with human judgments, which is a concrete step beyond purely model-based evaluation. The main caveat is that the validation is narrowly scoped (100 filtered DOCCI images for the metric, five examples for the threshold), so the headline improvements rest on a metric whose behavior on corrected captions has not been independently checked; this limits the current significance but is addressable with additional human evaluation.

major comments (5)
  1. [Section 4.3 / Table 4 / Appendix A] The central claim that CapMAS significantly enhances factuality is measured entirely with the proposed GPT-4o factuality metric, yet the metric's human validation (Table 3, Appendix A) is based on only 100 DOCCI images from which caption pairs of similar factuality were excluded, and it compares only original LLaVA-v1.5-7B vs InstructBLIP captions. No human evaluation is reported for original vs CapMAS-corrected captions. Because both CapMAS and the metric rely on atomic-proposition decomposition and truth verification, the observed gains in Table 4 (e.g., LLaVA-NeXT-7B factuality from 59.9 to 72.2) could be inflated by GPT-4o favoring the style or content of corrected captions rather than by a genuine reduction in hallucination. Please provide a human evaluation on a sample of corrected vs original captions, or otherwise demonstrate that the metric's judgments transfer to the correction setting.
  2. [Section 4.1 / Section 4.2 / Tables 3-6] The paper uses the word "significantly" repeatedly, but no confidence intervals, standard errors, or hypothesis tests are reported. The hyperparameter π is tuned on only five examples (Section 4.1), and the Spearman correlations in Table 3 (70.2 vs 67.9 for FactScore) may not be statistically distinguishable given the small filtered sample. Please report bootstrap confidence intervals and significance tests for the correlation differences and for the factuality/coverage improvements in Tables 4 and 5.
  3. [Section 4.4] The comparison with VisualFactChecker (VFC) is not a direct experiment. The text states that VFC could not be reproduced and that the authors "infer" superiority from two design differences: VFC targets only object hallucinations and uses an object detector, whose detection AUROC is lower than the Isolation method in Table 1. This inference conflates component-level hallucination detection with end-to-end caption quality and is insufficient to support the claim that CapMAS outperforms VFC. Please either include VFC results if resources become available, or restrict the claim to a design-level qualitative statement.
  4. [Section 3.3 / Table 2] The meta-evaluation reveals that the proposed metric assigns a factuality score of 62.8 to clean, human-written DOCCI captions. This means GPT-4o considers approximately 37% of atomic propositions in human captions to be false, indicating poor absolute calibration. Since Table 4 reports absolute factuality gains, it is important to discuss this calibration issue and to consider reporting relative improvements or a calibration-corrected score, especially because the gains could partly reflect changes in caption style rather than a reduction of hallucinations.
  5. [Section 4.5 / Table 6] The claim that VQA-centric benchmark performance "may not correlate" with detailed captioning ability is based on a visual comparison of ranks across only eight models, with no correlation coefficient or statistical test reported. Please compute a rank correlation (e.g., Spearman) between the detailed captioning average and OpenCompass or individual VQA benchmarks, and report its value and significance; otherwise, soften the claim to a qualitative counterexample.
minor comments (6)
  1. [Section 3.2 / Eq. (1)] The hallucination score formula uses min(p(T|x,Q(u)) - p(F|x,Q(u)), epsilon). Since p(T)-p(F) can be negative, taking the min with a small positive epsilon can produce a negative argument to the log. It appears that max(p(T)-p(F), epsilon) was intended to avoid taking the log of a non-positive number. Please correct or clarify.
  2. [Section 3.1 / Figure 2] The claim that hallucinations after the 192nd token are undetectable is based on visual inspection of the plotted means without error bars. Please report the dataset size, the number of hallucination/non-hallucination objects per bin, and consider adding confidence bands.
  3. [Section 3.3 / Coverage] The text says an LLM answers the multiple-choice questions using only the captions, but the model and the prompt for this coverage evaluation step are not specified in Section 4.1 or in the prompt templates of Appendix G. Please specify the model and provide the prompt template.
  4. [Section 4.1] The sentence "We adopt LLaVA-v1.5-7B, LLaVA-NeXT-7B, LLaVA-NeXT-13B, InternVL-Chat-V1.5, and GPT-4V as the models for both captioning and CapMAS's fact-checking" is ambiguous, because Table 4 also uses different MLLMs as fact-checkers for GPT-4V captions. Please clarify which models are used for each role.
  5. [Section 4.4 / Table 5] The methods are evaluated on different base models (LLaVA-v1.5-7B for VCD/OPERA/SPARC, MiniGPT-4 for LURE/LRV), which confounds the comparison. Please acknowledge this limitation explicitly or report all methods on a common base model.
  6. [Throughout] The paper contains typographical and formatting issues, including "V olcano" with an extra space in Section 4.4 and Table 5, and inconsistent capitalization of "LLaVA" and "FactScore/FaithScore". Please proofread and standardize the notation.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the factuality metric has independent human validation and CapMAS's gains are measured against external baselines; the shared decomposition logic is a validity risk, not a definitional reduction.

full rationale

The central claim that CapMAS improves factuality is measured by a GPT-4o-based metric that decomposes captions into atomic propositions and judges each against image and reference caption, while CapMAS uses the same decompose-and-verify idea via an LLM decomposer, an MLLM fact-checker, and an LLM corrector. This overlap is real, but it is not a circular reduction: the metric is not defined from CapMAS's own outputs or its fact-checker labels, and in the main experiments the fact-checker is a different MLLM (LLaVA-NeXT, InternVL, etc.) while the metric is GPT-4o, so their agreement is empirical rather than by construction. The metric is also independently validated against 100 human-judged DOCCI pairs in Table 3 and Appendix A, though that validation is filtered and does not cover the exact corrected-caption setting. The single CapMAS hyperparameter pi is tuned on five examples, but the ablation in Appendix C shows factuality gains across pi values, so the headline result is not a fitted artifact. Self-citations such as Jung et al. 2025 and Lee et al. 2024 appear only as baseline comparisons and do not carry the argument. The remaining concern is external validity of the metric for corrected captions, not definitional circularity; under the requirement to exhibit a specific equation-level reduction, no such step is present in the paper.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central results rest on the assumptions that atomic decomposition is reliable, that MLLM token probabilities for True/False reflect truth, that GPT-4o can act as a trustworthy judge, and that the coverage QA dataset accurately captures visual content. pi is the only explicit fitted parameter and is tuned on a very small validation set.

free parameters (1)
  • pi (threshold for classifying atomic propositions as True or False) = 1.0 in main results; appendix explores 0.5 and 0.3
    Controls how strict CapMAS is when deciding a proposition is false; tuned on a validation set of five examples from the DCI dataset (Section 4.1).
assumptions (5)
  • domain assumption A detailed caption can be decomposed into atomic propositions, each independently true or false.
    Assumed in Sections 3.2 and 3.3; both CapMAS and the factuality metric depend on reliable LLM decomposition.
  • domain assumption MLLM token probabilities for True and False correctly reflect proposition truth against the image.
    CapMAS computes hallucination scores from p(T) and p(F) in Equation (1) with no calibration evidence.
  • domain assumption GPT-4o can judge atomic proposition truth from image and reference caption accurately enough to serve as a metric.
    The factuality metric in Section 3.3 relies on GPT-4o as judge; validation is on 100 filtered DOCCI images.
  • domain assumption Long generated sequences become increasingly grounded in the model's own text rather than the image.
    Hypothesis in Section 3.1 motivated by cited works (Liu et al., 2024c; Zhong et al., 2024) rather than directly measured in this paper.
  • domain assumption The human-refined GPT-4o-generated multiple-choice questions in the coverage dataset accurately capture visual information.
    The coverage metric assumes that answering all questions from a caption implies full visual coverage; benchmark construction described in Section 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage." pith.science (2026). https://pith.science/paper/A6QQDHT3

@misc{pith2026241215484,
  author       = {Pith},
  title        = {Pith review of: Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6QQDHT3}},
  note         = {Machine review of arXiv:2412.15484}
}
read the original abstract

Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations. Our analysis reveals that existing hallucination detection methods struggle with detailed captions. We attribute this to the increasing reliance of MLLMs on their generated text, rather than the input image, as the sequence length grows. To address this issue, we propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions. Additionally, we introduce an evaluation framework and a benchmark dataset to facilitate the systematic analysis of detailed captions. Our experiments demonstrate that our proposed evaluation method better aligns with human judgments of factuality than existing metrics and that existing approaches to improve the MLLM factuality may fall short in hyper-detailed image captioning tasks. In contrast, our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V. Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating that an MLLM's performance on VQA benchmarks may not correlate with its ability to generate detailed image captions. Our code and data are available at https://github.com/adobe-research/CapMAS.

Figures

Figures reproduced from arXiv: 2412.15484 by the authors.

Figure 1
Figure 1. The process of generating a data sample for evaluating hallucination detection methods in detailed image captioning tasks. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The hallucination scores of the Confidence and Consis [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of CapMAS. The decomposer LLM breaks an initial caption into atomic units. These units are converted into [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: An example of our coverage evaluation data sample. The dataset consists of multiple-choice questions with four or fewer options. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: An example of a caption generated by CapMAS, with LLaVA-NeXT-7B as both the captioning and fact-checking model and [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: The five prompt inputs used to generate captions in our experiments. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: The prompt input for LLaMA-3-8B serving as the decomposer. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: The prompt input for LLaMA-3-8B serving as the corrector. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The prompt input for LLaMA-3-8B serving as the summerizer. We use the prompt employed in the work of ( [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: The prompt input for GPT-4o used to create the meta-evaluation dataset of Table [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: The prompt input for GPT-4 used to create the dataset of Figure [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models

    cs.CV 2025-02 conditional novelty 7.0 of 10

    SPARC selectively and progressively reinforces attention to relevant image tokens during decoding, improving both precision and recall in detailed image captioning compared to baselines and prior hallucination-mitigat...

  2. CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

    cs.CV 2026-08 conditional novelty 5.0 of 10

    Caption quality can be split into Coverage and Precision; in controlled training runs, Coverage predicts VLM understanding while Precision predicts T2I generation.

Reference graph

Works this paper leans on

60 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  3. [3]

    and Lavie, A

    Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp.\ 65--72, 2005

    2005

  4. [4]

    B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  5. [5]

    Clair: Evaluating image captions with large language models

    Chan, D., Petryk, S., Gonzalez, J., Darrell, T., and Canny, J. Clair: Evaluating image captions with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 13638--13646, 2023

    work page 2023

  6. [6]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 a

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 a

    arXiv 2024

  7. [7]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024 b

    arXiv 2024

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24185--24198, 2024 c

    2024

Show all 60 references
  1. [9]

    Efficient and effective text encoding for chinese llama and alpaca

    Cui, Y., Yang, Z., and Yao, X. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177, 2023

    2023 arXiv

  2. [10]

    Instruct BLIP : Towards general-purpose vision-language models with instruction tuning

    Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instruct BLIP : Towards general-purpose vision-language models with instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net...

    2023

  3. [11]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024

    Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., and Chen, K. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. URL https://arxiv.org/abs/2407.11691

    2024 arXiv

  4. [12]

    K., Bitton, Y., Montgomery, C., Onoe, Y., Bunner, A., Krishna, R., Baldridge, J., and Soricut, R

    Garg, R., Burns, A., Ayan, B. K., Bitton, Y., Montgomery, C., Onoe, Y., Bunner, A., Krishna, R., Baldridge, J., and Soricut, R. Imageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.02793, 2024

    2024 arXiv

  5. [13]

    S., Lin, T.-Y., Liu, M.-Y., and Cui, Y

    Ge, Y., Zeng, X., Huffman, J. S., Lin, T.-Y., Liu, M.-Y., and Cui, Y. Visual fact checker: Enabling high-fidelity detailed caption generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14033--14042, 2024

    2024

  6. [14]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Confere...

    2024

  7. [15]

    Clipscore: A reference-free evaluation metric for image captioning

    Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.\ 7514--7528, 2021

    2021

  8. [16]

    Z., Sohel, F., Shiratuddin, M

    Hossain, M. Z., Sohel, F., Shiratuddin, M. F., and Laga, H. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR), 51 0 (6): 0 1--36, 2019

    2019

  9. [17]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision an...

    2024

  10. [18]

    F aith S core: Fine-grained evaluations of hallucinations in large vision-language models

    Jing, L., Li, R., Chen, Y., and Du, X. F aith S core: Fine-grained evaluations of hallucinations in large vision-language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 5042--5063, Miam...

    2024 doi

  11. [19]

    Visual attention never fades: Selective progressive attention recalibration for detailed image captioning in multimodal large language models

    Jung, M., Lee, S., Kim, E., and Yoon, S. Visual attention never fades: Selective progressive attention recalibration for detailed image captioning in multimodal large language models. arXiv preprint arXiv:2502.01419, 2025

    2025 arXiv

  12. [20]

    A diagram is worth a dozen images

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, pp.\ 235--251. Springer, 2016

    2016

  13. [21]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Lauren c on, H., Tronchon, L., Cord, M., and Sanh, V. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    2024 arXiv

  14. [22]

    Volcano: Mitigating multimodal hallucination through self-feedback guided revision

    Lee, S., Park, S., Jo, Y., and Seo, M. Volcano: Mitigating multimodal hallucination through self-feedback guided revision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume ...

    2024

  15. [23]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding

    Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13872--13882, 2024

    2024

  16. [24]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 a

    2023

  17. [25]

    X., and Wen, J.-R

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023 b

    2023 arXiv

  18. [26]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.\ 74--81, 2004

    2004

  19. [27]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023 a

    2023

  20. [28]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 a . URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  21. [29]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 b

    2024

  22. [30]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023 b

    2023 arXiv

  23. [31]

    Paying more attention to image: A training-free method for alleviating hallucination in lvlms

    Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. arXiv preprint arXiv:2407.21771, 2024 c

    2024 arXiv

  24. [32]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023 c

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023 c

    2023 arXiv

  25. [33]

    On the hidden mystery of ocr in large multimodal models, 2024 d

    Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X., lin Liu, C., Jin, L., and Bai, X. On the hidden mystery of ocr in large multimodal models, 2024 d . URL https://arxiv.org/abs/2305.07895

    2024 arXiv

  26. [34]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024

    2024

  27. [35]

    Self-refine: Iterative refinement with self-feedback

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36: 0 46534--46594, 2023

    2023

  28. [36]

    On faithfulness and factuality in abstractive summarization

    Maynez, J., Narayan, S., Bohnet, B., and McDonald, R. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 1906--1919, 2020

    1906

  29. [37]

    FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generation

    Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generation. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 ...

    2023 doi

  30. [38]

    Docci: Descriptions of connected and contrasting images

    Onoe, Y., Rane, S., Berger, Z., Bitton, Y., Cho, J., Garg, R., Ku, A., Parekh, Z., Pont-Tuset, J., Tanzer, G., et al. Docci: Descriptions of connected and contrasting images. arXiv preprint arXiv:2404.19753, 2024

    2024 arXiv

  31. [39]

    Bleu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

    2002

  32. [40]

    Aloha: A new measure for hallucination in captioning models

    Petryk, S., Chan, D., Kachinthaya, A., Zou, H., Canny, J., Gonzalez, J., and Darrell, T. Aloha: A new measure for hallucination in captioning models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    2024

  33. [41]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  34. [42]

    and Gurevych, I

    Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084

    2019 arXiv

  35. [43]

    A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions

    Urbanek, J., Bordes, F., Astolfi, P., Williamson, M., Sharma, V., and Romero-Soriano, A. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2670...

    2024

  36. [44]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  37. [45]

    Cider: Consensus-based image description evaluation

    Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4566--4575, 2015

    2015

  38. [46]

    Show and tell: A neural image caption generator

    Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3156--3164, 2015

    2015

  39. [47]

    V., Chi, E

    Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  40. [48]

    Show, attend and tell: Neural image caption generation with visual attention

    Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp.\ 2048--2057. PMLR, 2015

    2015

  41. [49]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  42. [51]

    A survey on multimodal large language models

    Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023 b

    2023 arXiv

  43. [52]

    Woodpecker: Hallucination correction for multimodal large language models

    Yin, S., Fu, C., Zhao, S., Xu, T., Wang, H., Sui, D., Shen, Y., Li, K., Sun, X., and Chen, E. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045, 2023 c

    2023 arXiv

  44. [53]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  45. [54]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding an...

    2024

  46. [55]

    When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023

    Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023

    2023

  47. [56]

    Enhancing uncertainty-based hallucination detection with stronger focus

    Zhang, T., Qiu, L., Guo, Q., Deng, C., Zhang, Y., Zhang, Z., Zhou, C., Wang, X., and Fu, L. Enhancing uncertainty-based hallucination detection with stronger focus. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 915--932, 2023

    2023

  48. [57]

    Knowing what llms do not know: A simple yet effective self-detection method

    Zhao, Y., Yan, L., Sun, W., Xing, G., Meng, C., Wang, S., Cheng, Z., Ren, Z., and Yin, D. Knowing what llms do not know: A simple yet effective self-detection method. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingu...

    2024

  49. [58]

    Investigating and mitigating the multimodal hallucination snowballing in large vision-language models

    Zhong, W., Feng, X., Zhao, L., Li, Q., Huang, L., Gu, Y., Ma, W., Xu, Y., and Qin, B. Investigating and mitigating the multimodal hallucination snowballing in large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    2024

  50. [59]

    Analyzing and mitigating object hallucination in large vision-language models

    Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  51. [60]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  52. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.