Pith. sign in

REVIEW 3 major objections 4 minor 54 references

Unblocking Fine-Grained Evaluation of Detailed Captions: An Explaining AutoRater and Critic-and-Revise Pipeline

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model trained on human-annotated synthetic captions can judge every sentence of a paragraph-length image caption for factual accuracy, explain the error, and guide an LLM to correct it.

desk verdict Real benchmark and transferable factuality model, but the Neutral-as-error conflation needs a hard look before the error rates are taken at face value. read the letter →

arxiv 2506.07631 v1 pith:MQ5DIFWG submitted 2025-06-09 cs.CL cs.CV

Brian Gordon , Yonatan Bitton , Andreea Marzoca , Yasumasa Onoe , Xiao Wang , Daniel Cohen-Or , Idan Szpektor This is my paper · ORCID
classification cs.CLcs.CV
keywords vision-languagemodelsimagecaptioningfactualityevaluationhallucinationdetectioncritiquegenerationparagraphcaptionscaptioncorrectionsynthetic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models now produce paragraph-length image captions, but existing factuality checks are built for short texts and miss small errors. This paper claims that a single model, VNLI-Critique, can judge each sentence of such a caption against the image and generate a text critique explaining why a sentence is wrong. To train and test it, the authors built DOCCI-Critique, a benchmark of 1,400 captions from 14 VLMs with 10,216 sentence-level human labels and error rationales. They report top performance on an external hallucination benchmark, near-human ranking of captioning models, and human-confirmed factuality gains when the critiques drive a separate LLM to revise captions. If the claim holds, per-sentence human evaluation of detailed captions can be automated and large synthetic caption datasets can be corrected at scale.

What carries the argument

The central machinery is dual-task instruction tuning of a roughly 10-billion-parameter vision-language model on human-annotated synthetic captions. One prompt asks whether the target sentence aligns with the image; the other asks the model to explain the inaccuracies in a flagged sentence. The paragraph prefix is included as context in both prompts, which lets the model resolve ambiguity and co-reference that sentence-level metrics miss. The same model therefore provides both the factual signal and the explanatory text that drives the critic-and-revise loop.

What would settle it

Two concrete checks would settle the claim: re-annotate a sample of DOCCI-Critique with 'Neutral' kept separate from 'Contradiction' and see whether VNLI-Critique's errors concentrate on sentences that are true but unverifiable from the image; and evaluate the model on captions generated by a VLM family outside the synthetic training pool, comparing sentence-level predictions with fresh human labels.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that fine-grained, sentence-level factuality checking of paragraph captions can be learned from human annotations on synthetic captions and transferred to unseen models and datasets. VNLI-Critique classifies each sentence as aligned or not with the image, using the preceding paragraph as context, and for flagged sentences produces an explanatory critique. The paper reports a Macro-F1 of 0.76 on M-HalDetect, a Spearman correlation of 0.98 with human VLM rankings on DOCCI-Critique, and human-verified increases in sentence accuracy from 15% to 61% on DetailCaps-4870 and from 24% to 75% on PixelProse after revision. The supporting claim is that these results are due to robust generalization rather than in-distribution memorization.

Load-bearing premise

The benchmark counts a sentence as factually wrong when a majority of annotators label it either 'Neutral' or 'Contradiction', so claims that the image simply does not verify are treated as errors; if that conflation is wrong, the model's behavior and the headline gains inherit the bias.

Editorial extensions

If this is right

  • Automatic leaderboards for captioning models become feasible: VNLI-Critique ranks 14 VLMs with a Spearman correlation of at least 0.968 against human factuality judgments on all three ranking criteria.
  • Factuality checking transfers to new domains: the same model sets the best compared Macro-F1 on M-HalDetect and performs competitively on CHOCOLATE chart-caption claim verification without chart-specific training.
  • Synthetic caption datasets can be repaired at scale: the critic-and-revise pipeline lifts human-confirmed accuracy of flagged sentences from 15% to 61% on DetailCaps-4870 and from 24% to 75% on PixelProse.
  • Critiques are usable as explanations, not just labels: human raters judged most VNLI-Critique critiques correct and relevant on both DOCCI-Critique and M-HalDetect sentences.
  • Because evaluation happens sentence by sentence in paragraph context, fine-grained errors such as a wrong object location or a single wrong attribute can be isolated instead of hidden in a whole-response score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the most consequential test is to split the benchmark's 'Neutral' label from 'Contradiction': if most neutral sentences are true statements the image simply does not settle, the model may be learning to call unverifiable claims false, which would bias both the error rates and the correction gains.
  • A related extension is to audit the synthetic training captions for diversity of failure modes; since they come from one model family, rarer error types from other VLM families may be underrepresented, and testing on a broader generator pool would stress the generalization claim.
  • The critic-and-revise design implies a practical data-cleaning layer for large synthetic caption corpora; the authors show sentence-level gains but do not test whether a downstream model trained on corrected captions improves, which is a natural next experiment.
  • The unused annotation label for whether a sentence needs image grounding suggests a hybrid verifier that routes world-knowledge claims such as 'flamingos eat brine shrimp' to a knowledge base instead of the image; implementing that two-stage check could reduce false positives on true but visually unverifiable sentences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces DOCCI-Critique, a benchmark of 1,400 paragraph-length VLM captions over 100 images with 10,216 sentence-level human factuality judgments and error rationales, and VNLI-Critique, a PaliGemma-2-based model fine-tuned for sentence-level factuality classification and critique generation. The authors report three main results: state-of-the-art factuality classification on the external M-HalDetect benchmark and competitive results on CHOCOLATE; an AutoRater whose VLM rankings correlate highly with human rankings on DOCCI-Critique (0.98 Spearman); and a Critic-and-Revise pipeline in which VNLI-Critique critiques guide an LLM to revise incorrect sentences, with human-verified accuracy gains of 46% on DetailCaps-4870 and 51% on PixelProse. The paper also reports a human evaluation of critique quality and a limitations section discussing benchmark size and possible extensions.

Significance. If the central claims hold, this is a useful contribution. The benchmark is novel in combining paragraph-level context, sentence-level granularity, and explanatory rationales across 14 VLMs. The emphasis on a 10B critic with state-of-the-art external benchmark performance and a two-stage, interpretable Critic-and-Revise pipeline is practically valuable. The paper includes several commendable strengths: a multi-annotator protocol with five judges per sentence, human evaluation of both generated critiques and revised captions, and validation on external benchmarks (M-HalDetect, CHOCOLATE) that goes beyond in-distribution performance. The cost and training details are transparent. However, two load-bearing issues need to be addressed before the claims can be taken at face value: the benchmark's label definition collapses 'Neutral' with 'Contradiction', and the headline revision gains are measured only on the subset of sentences that the critic chose to flag.

major comments (3)
  1. [Section 3, label definition] The benchmark's core label is defined by collapsing two nonequivalent annotation categories: 'A sentence is classified as "Non-entailment" if a majority labeled it "Neutral" or "Contradiction"', where 'Neutral' is defined as 'not verifiable/contradicted'. This is load-bearing because DOCCI-Critique's error rates, VNLI-Critique's training targets, and the critique-generation rationales all inherit this convention: a sentence whose content is merely unverifiable from the image is treated as factually incorrect. For example, a claim about an occluded object's color would be labeled non-entailment even without visual evidence of falsity. The limitations section (Section 6) does not flag this conflation, and it sits at the interface between benchmark construction, model training, and all downstream claims. Please report the distribution of 'Neutral' versus 'Contradiction' labels in DOCCI-Critique, analyze how the benchmark statistics and VNLI-Critique's performance change when 'Neutral' sentences are excluded or treated as a separate class, and discuss explicitly whether unverifiable claims should count as factual errors for the intended evaluation use cases.
  2. [Section 5.2, Table 6] The headline gains (+46% on DetailCaps-4870, +51% on PixelProse) are computed only over sentences that VNLI-Critique flagged as inaccurate, not over whole captions. Table 6's 'Original' and 'Fixed' columns refer to this flagged subset, and no whole-caption accuracy, recall for missed errors, or rate of newly introduced errors from revision is reported. Consequently, the abstract's phrase 'substantial improvements in caption factuality' is broader than the evidence presented. Please report whole-caption human factuality before and after revision (including sentences not flagged by the critic), and separate the component of the improvement due to flagging precision from the component due to revision quality.
  3. [Section 4.2, Table 3] The AutoRater claim of 0.98 Spearman correlation is computed over only 14 VLM rankings on 100 images. With 14 data points, Spearman correlation has wide confidence intervals, and the paper does not provide intervals or a significance statement beyond p-values in the appendix. Please add a measure of uncertainty (e.g., bootstrap confidence intervals) or otherwise temper the claim that the AutoRater yields 'reliable' rankings, given the small number of ranked items.
minor comments (4)
  1. [Table 6] The 'Original' row for Human Judge (15%) is confusing because the text describes a '15% false positive rate' for the flagging step. Please clarify the definition of the Original column and the relationship between false positive rate and the accuracy reported.
  2. [Appendix C.1] The annotation interface in Figure 3 labels a sentence about non-visible hooves as 'Contradiction', while the protocol in Section 3 defines 'Neutral' as 'not verifiable/contradicted'. This inconsistency between the interface example and the label definition should be reconciled, as it bears directly on the major comment about Neutral.
  3. [Section 4.3 and Table 5] The critique-quality human evaluation samples sentences 'previously identified by human annotators as factually incorrect'; because of the Neutral/Contradiction collapse, this sample may include sentences that are merely unverifiable. Please state how many of the sampled sentences were majority-Neutral and whether critique quality differs for those cases.
  4. [References and notation] Reference [12] contains an unresolved '20xx' date; the appendix tables use nonstandard notation such as '0.985e−10' for p-values, which should be formatted consistently; and the main text alternates between 'PaliGemma-2' and 'PaliGemma 10B' when describing the model architecture.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model is trained on separate human-annotated synthetic captions, evaluated on external benchmarks and independent human judgments, and the self-consistency row is not load-bearing.

full rationale

VNLI-Critique is trained on a separately constructed synthetic-caption corpus (over 70 PaliGemma-2 variants on DOCCI training data) and evaluated on DOCCI-Critique (qual-dev split), on external M-HalDetect and CHOCOLATE, and through independent human evaluations of generated critiques and revised captions. The 0.98 AutoRater correlation is an evaluation against human labels of the same type used in training, but the labels are independent human judgments, not model outputs, so predicting them is standard supervised learning rather than a fitted quantity renamed as prediction. Table 6's 'VNLI-Critique as Judge' row is explicitly labeled self-consistency and is not used to support the headline human-confirmed gains. The treatment of 'Neutral' as 'Non-entailment' is a substantive benchmark-validity choice, not a circular derivation: it does not make any equation reduce to itself. Self-citations to PaliGemma-2 and DOCCI are citations to public resources, and no load-bearing uniqueness or ansatz claim rests on them. Hence no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted constants or invented physical entities appear in the paper. The central claims rest on data collection and modeling assumptions: human annotation reliability, the neutral-as-error conflation, image sampling, training distribution coverage, and the safety of the revision step. These are enumerated above because they are load-bearing for the benchmark, the model, and the pipeline results.

assumptions (5)
  • domain assumption Majority vote of five annotators provides reliable ground-truth factuality labels.
    Section 3 and Appendix C describe five Prolific annotators per sentence; no inter-annotator agreement statistics are reported, yet the benchmark labels are treated as ground truth for training and evaluation.
  • ad hoc to paper 'Neutral' (unverifiable or not contradicted) is counted as non-entailment/error when a majority chooses it.
    Section 3: 'A sentence is classified as Non-entailment if a majority labeled it Neutral or Contradiction'. This design choice conflates unverifiable with inaccurate and shapes all downstream results.
  • domain assumption The 100 DOCCI qual-dev images are representative of real-world images for evaluating detailed caption factuality.
    Section 3 selects 100 images from DOCCI; Section 6 acknowledges that expanding the image base would improve statistical power and coverage.
  • domain assumption Captions generated by 70 PaliGemma-2 variants on DOCCI training data provide a diverse enough training distribution for error types found in the 14 benchmark VLMs.
    Section 4.1 constructs the training data from PaliGemma-2 variants; the external benchmark transfer (M-HalDetect, CHOCOLATE) is used as evidence, but the coverage of error diversity is not directly quantified.
  • domain assumption The revision LLM (Gemini-2.0-Flash) corrects the errors named in the critique without introducing new factual errors outside the flagged sentence.
    Section 5.1-5.2 evaluate revised sentences, but whole-caption accuracy and errors in non-flagged sentences are not measured, so the no-new-error assumption is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unblocking Fine-Grained Evaluation of Detailed Captions: An Explaining AutoRater and Critic-and-Revise Pipeline." pith.science (2026). https://pith.science/paper/MQ5DIFWG

@misc{pith2026250607631,
  author       = {Pith},
  title        = {Pith review of: Unblocking Fine-Grained Evaluation of Detailed Captions: An Explaining AutoRater and Critic-and-Revise Pipeline},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MQ5DIFWG}},
  note         = {Machine review of arXiv:2506.07631}
}
read the original abstract

Large Vision-Language Models (VLMs) now generate highly detailed, paragraphlength image captions, yet evaluating their factual accuracy remains challenging. Current methods often miss fine-grained errors, being designed for shorter texts or lacking datasets with verified inaccuracies. We introduce DOCCI-Critique, a benchmark with 1,400 VLM-generated paragraph captions (100 images, 14 VLMs) featuring over 10,216 sentence-level human annotations of factual correctness and explanatory rationales for errors, all within paragraph context. Building on this, we develop VNLI-Critique, a model for automated sentence-level factuality classification and critique generation. We highlight three key applications: (1) VNLI-Critique demonstrates robust generalization, validated by state-of-the-art performance on the M-HalDetect benchmark and strong results in CHOCOLATE claim verification. (2) The VNLI-Critique driven AutoRater for DOCCI-Critique provides reliable VLM rankings, showing excellent alignment with human factuality judgments (e.g., 0.98 Spearman). (3) An innovative Critic-and-Revise pipeline, where critiques from VNLI-Critique guide LLM-based corrections, achieves substantial improvements in caption factuality (e.g., a 46% gain on DetailCaps-4870). Our work offers a crucial benchmark alongside practical tools, designed to significantly elevate the standards for fine-grained evaluation and foster the improvement of VLM image understanding. Project page: https://google.github.io/unblocking-detail-caption

Figures

Figures reproduced from arXiv: 2506.07631 by the authors.

Figure 1
Figure 1. VNLI-Critique in action: operating as a Critic and within the Critic-and-Revise pipeline. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Sentence-level factuality assessment by VNLI-Critique. Figure shows an image, VLM [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Example of the Description Sentences Annotation Interface. Annotators are shown the [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example of the Critique Annotation Interface. Annotators assess if the ’Predicted Explana [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 36 canonical work pages

  1. [1]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  2. [2]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan,

    work page

  3. [3]

    PaLI: A jointly-scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Ca...

    work page 2023

  4. [4]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025

    arXiv 2025

  5. [5]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023

    2023

  6. [6]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tai, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping H...

    work page 2024

  7. [7]

    Instructblip: towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: towards general-purpose vision-language models with instruction tuning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, Red Hook, NY , USA, 2023. Curran Associates Inc

    work page 2023

  8. [8]

    Smith, Hannaneh Hajishirzi, Ross B

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, Yen-Sung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvo...

    arXiv 2024

Show all 54 references
  1. [9]

    Benchmarking and improving detail image caption

    Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405.19092, 2024

    2024 arXiv

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  3. [11]

    ImageInWords: Unlocking hyper-detailed image descriptions

    Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Michael Baldridge, and Radu Soricut. ImageInWords: Unlocking hyper-detailed image descriptions. In Proceedings of the 2024 Conference on Empiri...

    2024

  4. [12]

    Introduction to Cloud TPU

    Google Cloud. Introduction to Cloud TPU. https://cloud.google.com/tpu/docs/intro-to-tpu , 20xx. Accessed: 2024-07-04. 10

    2024

  5. [13]

    Mismatch quest: Visual and textual feedback for image-text misalignment

    Brian Gordon, Yonatan Bitton, Yonatan Shafir, Roopal Garg, Xi Chen, Dani Lischinski, Daniel Cohen- Or, and Idan Szpektor. Mismatch quest: Visual and textual feedback for image-text misalignment. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September ...

    2024

  6. [14]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence an...

    2024

  7. [15]

    CLIPScore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528, Online and Punta Cana, Domi...

    2021

  8. [16]

    Framing image description as a ranking task: data, models and evaluation metrics

    Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as a ranking task: data, models and evaluation metrics. J. Artif. Int. Res., 47(1):853–899, 2013

    2013

  9. [17]

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering . In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 20349...

    2023

  10. [18]

    Do LVLMs understand charts? analyzing and correcting factual errors in chart captioning

    Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. Do LVLMs understand charts? analyzing and correcting factual errors in chart captioning. In Findings of the Association for Computational Linguistics: ACL 2024...

    2024

  11. [19]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015

    2015

  12. [20]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  13. [21]

    Vila: On pre- training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26689–26699, 2024

    2024

  14. [22]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, pages 740–755, Cham, 2014. Springer International Publishing

    2014

  15. [23]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In Computer Vision – ECCV 2024, pages 366–384, Cham, 2025. Springer Nature Switzerland

    2024

  16. [24]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024

    2024

  17. [25]

    Docci: Descriptions of connected and contrasting images

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, and Jason Baldridge. Docci: Descriptions of connected and contrasting images. In Computer Vision – ECCV 2024: 18th Eur...

    2024

  18. [26]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex ...

    2024

  19. [27]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, page 311–318, USA, 2002. Association for Computational Linguistics

    2002

  20. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  21. [29]

    Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jero...

    2024 arXiv

  22. [30]

    Evolution of image caption- ing models: An overview

    Abdelkrim Saouabe, Said Tkatek, Merouane Mazar, and Imad Mourtaji. Evolution of image caption- ing models: An overview. In 2023 10th International Conference on Wireless Networks and Mobile Communications (WINCOM), pages 1–5, 2023

    2023

  23. [31]

    Laion-5b: an open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Lai...

    2022

  24. [32]

    From pixels to prose: A large dataset of dense image captions, 2024

    Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, and Tom Goldstein. From pixels to prose: A large dataset of dense image captions, 2024

    2024

  25. [33]

    From Show to Tell: A Survey on Deep Learning-Based Image Captioning

    Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From Show to Tell: A Survey on Deep Learning-Based Image Captioning . IEEE Transactions on Pattern Analysis & Machine Intelligence, 45(01):539–559, 2023

    2023

  26. [34]

    Paligemma 2: A family of versatile vlms for transfer

    Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lu...

    2024 arXiv

  27. [35]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, A...

    2024

  28. [36]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023

  29. [37]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language enc...

    2025

  30. [38]

    A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions

    Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  31. [39]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, pages 4566–4575. IEEE Computer Society, 2015

    2015

  32. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024 arXiv

  33. [41]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  34. [42]

    Revisiting text-to-image evaluation with gecko: on metrics, prompts, and human rating

    Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajic, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Pinelopi Papalampidi, Ira Ktena, Christopher Knutsen, Cyrus Rashtchian, Anant Nawalgaria, Jordi Pont-Tuset, and Aida Nematzadeh. Revisiting text-to-image evaluation with...

    2025

  35. [43]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024

  36. [44]

    What you see is what you read? improving text-image alignment evaluation

    Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. In Advances in Neural Information Processing Systems, pages 1601–1619. Curran Ass...

    2023

  37. [45]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024

    2024

  38. [46]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (C...

    2024

  39. [47]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014

    2014

  40. [48]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, 2023

    2023

  41. [49]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 17 A DOCCI-Critique AutoRater Leaderboards This appendix presents the complete le...

    2023 arXiv

  42. [51]

    Response-Level Correctness: Percentage of entirely factually accurate paragraphs (Ta- ble 7)

  43. [52]

    Correct Sentences Overall: Total percentage of correct sentences across all descriptions (Table 9)

  44. [53]

    In each leaderboard (Tables 7, 9, and 8), rows list the 14 caption-generating VLMs from DOCCI- Critique (details in Table 2)

    Correct Sentences per Description: Average percentage of correct sentences per descrip- tion (Table 8). In each leaderboard (Tables 7, 9, and 8), rows list the 14 caption-generating VLMs from DOCCI- Critique (details in Table 2). Columns denote automated ranking methods (e.g.,...

  45. [54]

    The pig back hooves are pink, not black. The front hooves are not visible

    was employed with its default hyperparameters, and a constant learning rate of 1 × 10−6 was maintained throughout the fine-tuning process. B.3 Computational Resources The training of the VNLI-Critique model was executed on Google Cloud TPUv5e [12] accelerators. Specifically, a...

  46. [2005]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.