Pith. sign in

REVIEW 4 major objections 4 minor 61 references

CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dense region-level QA shows caption models differ mainly in how much of the scene their captions cover, not in how accurate their committed facts are.

desk verdict CapProbe is a genuine step up in dense, region-anchored caption evaluation, but the headline coverage gaps are plausibly inflated by generator-family overlap and need cross-generator validation before they're trusted. read the letter →

arxiv 2608.11074 v1 pith:ZDK6QXNG submitted 2026-08-11 cs.CV

Mouxiao Huang , Qiangyu Yan , Borui Jiang , Han Shu This is my paper · ORCID
classification cs.CV
keywords detailedimagecaptioningdensequestionansweringregion-alignedevaluationcaptionbenchmarkvision-languagemodelscoveragemetricsinformationdensitymultiple-choiceprobing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CapProbe turns evaluation of detailed image captions from fuzzy scoring into fact-checking: each test image is split into coarse semantic regions, and roughly 74 multiple-choice questions per image probe those regions across ten fact categories, with a language judge answering each question using only the caption under evaluation. The paper's central claim is that this dense, region-aligned questioning is a reliable diagnostic that exposes differences between caption models that sparse or overlap-based metrics miss. Across 13 vision-language models, the decisive result is that all models are about equally accurate once the judge commits to an answer, so the real differentiator is coverage—how much of the probed scene the caption actually lets the judge resolve. The accompanying density metrics show that verbosity does not equal information: the longest captions are often the least dense in verifiable facts. If the method holds up, detailed caption evaluation becomes a reproducible, comparatively cheap multiple-choice reading task instead of an open-ended scoring exercise.

What carries the argument

The load-bearing object is the region-aligned dense QA checklist. Each image is decomposed by YOLOv26-seg and SAM3 into coarse foreground and background regions; Gemini-3.1-Pro writes structured metadata for every region, and Gemini-3.1-Pro together with GPT-5.5 generates multiple-choice questions across ten categories (attributes, recognition, count, OCR/text, camera features, spatial relations, and others), yielding 25,650 human-checked QA pairs over 346 images. At evaluation time, a language judge reads only the caption and answers each question, choosing the Uncertain option (E) when the caption lacks the information; that option drives the analytic identity $\text{Overall Acc} = \text{Effective Acc} \times \text{Coverage}$, which separates whether a caption lets a judge answer from whether the answered facts are correct. The efficiency side counts correct answers per thousand caption tokens ($d_i = (C_i/M_i)/T_i \times 1000$), averaged per image versus pooled over request-token mass, so that verbose but uninformative captions are penalized.

What would settle it

Build a second question set for the same 346 images whose metadata and MCQs come from outside the two generating models—human-written questions or an open-weight VLM with no Gemini or GPT involvement—keep the judge and protocol fixed, and check whether Gemini-3.1-Pro still ranks first on Overall Accuracy and Coverage. A cheaper variant: compute Gemini-3.1-Pro's accuracy separately on its own questions versus GPT-5.5's questions; a systematic gap would show the leaderboard reflects generator alignment rather than caption quality.

Watch

Extended reading notes

Core claim

The authors claim that detailed caption quality decomposes into two judge-measured components with very different behavior across today's models. Using the identity $\text{Overall Acc} = \text{Effective Acc} \times \text{Coverage}$, they report that all 13 evaluated VLMs cluster near 93 percent Effective Accuracy when the judge commits to an answer, while Coverage spans from 36.43 percent (GPT-4o) to 73.68 percent (Gemini-3.1-Pro); the primary differentiator between caption models is therefore coverage, not the correctness of committed facts. The benchmark's second finding is that caption length is a poor proxy for informativeness: Gemini-3.1-Pro writes the longest captions (561 tokens per image) yet sits near the bottom on density, while GPT-5.5's short captions (222 tokens) reach the top densities, and no model simultaneously dominates coverage and efficiency along the reported Pareto frontier.

Load-bearing premise

The load-bearing premise is that the benchmark is neutral enough across model families for its rankings to reflect caption quality, even though Gemini-3.1-Pro—the model that tops the leaderboard—also generated the region metadata and co-wrote the questions, a circularity the paper itself concedes in its limitations section.

Editorial extensions

If this is right

  • Under a fixed judge, model rankings are reproducible even with cheap open-weight readers, so detailed caption evaluation no longer requires a strong proprietary scorer.
  • Because coverage—not committed-answer accuracy—separates today's models, the main bottleneck for detailed captioning is telling the full scene, not stating facts correctly.
  • Length does not buy informativeness: density metrics expose which models pack verifiable facts per token, making them a usable efficiency axis for model comparison.
  • The competency–efficiency Pareto frontier means comprehensive captioners and efficient ones are currently different models, so improving either axis without losing the other is an open gap.
  • Per-category results localize the shared weaknesses of all current models—Count, Camera Features, and Attributes—giving concrete targets for the next generation of caption models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the authors leave implicit: since coverage is the measured bottleneck rather than fact accuracy, it could be used directly as a training reward (for example, via reinforcement learning on CapProbe-style probes) to push caption models toward fuller scene coverage.
  • Because absolute scores are judge-conditioned (the same captions score between 43 and 78 percent across three judges), anchoring every judge with a fixed reference caption would convert judge-dependent numbers into comparable quantities—a calibration step the fixed-judge protocol does not currently provide.
  • Since 'full-scene' is a design goal rather than verified exhaustive coverage, an untested sensitivity remains: models that systematically under-describe background or 'stuff' regions will be penalized most on images with many such regions, so a per-image analysis by region count could reveal whether the coverage gap is concentrated in background description.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CapProbe, a dense, region-aligned QA benchmark for evaluating detailed image captions. Images are segregated into coarse semantic regions; for each region, multiple-choice questions across ten categories are generated by Gemini-3.1-Pro and GPT-5.5, then deduplicated, balanced, and human-reviewed. A language judge answers the questions from the caption alone, with an 'Uncertain' (E) option. The authors define competency metrics (Overall Accuracy, Effective Accuracy, Uncertain Ratio, Coverage) and efficiency metrics (Tokens/Image, Mean Density, Global Density, Density CV), and report results for 13 VLMs. The headline findings are that Effective Accuracy is tightly clustered across models while Coverage varies widely, and that dense probing reveals coverage gaps invisible to sparse or overlap-based metrics. The authors also report a judge-reliability analysis on three top models showing rank stability. The paper explicitly lists limitations in Appendix A, including the generator-benchmark alignment risk, missing region-recall evaluation, and absent inter-annotator agreement metrics.

Significance. If the benchmark is neutral across model families and publicly released, CapProbe would be a useful contribution: it offers higher probe density than CaptionQA (74 vs. 50.3 QA pairs per image), adds explicit region-level anchoring, and proposes a transparent metric framework where Overall Accuracy is exactly the product of Effective Accuracy and Coverage. The derivations in Sec. 4 are sound, and the protocol is cost-effective by design. The authors are also commendably explicit about the judge-conditioned nature of the scores and the distinction between proxies and causal measurements. However, the central claim that the observed Coverage gaps reflect genuine caption-quality differences rather than construction bias is not yet backed by a neutral-QA experiment, and the benchmark itself is unreleased. These issues must be addressed before the paper's central conclusions are convincingly established.

major comments (4)
  1. [Sec. A; Sec. 5.2; Table 3] The headline observation—that models differ mainly in Coverage, not Effective Accuracy—is potentially confounded by benchmark construction bias. The QA set and region metadata are generated by Gemini-3.1-Pro and GPT-5.5, both of which are among the evaluated caption models, and the judge answers from the caption alone. If a caption uses phrasing, fact granularity, or distractor style aligned with the QA generator, the judge will commit to A–D more readily, inflating Coverage and Overall Accuracy without any change in underlying caption factuality. Table 3 shows Effective Accuracy clustering in a narrow range (92.7–94.5%) while Coverage spans 36.4–73.7%, so the coverage gap is doing essentially all the work in separating models—exactly the pattern that a style-overlap artifact would produce. The paper acknowledges this in Sec. A and calls for cross-generator validation, but that validation is essential, not optional, for the paper's central claim. I request a concrete experiment: generate a QA subset with models outside the evaluated set (e.g., open-weight or non-Gemini models), run the same judge and metric pipeline, and show that the ranking and coverage gaps persist. Without this, the claim that CapProbe 'reliably' exposes coverage gaps cannot be distinguished from a construction artifact.
  2. [Abstract; Sec. 1; Sec. 3.8] The main contribution of the paper is the CapProbe benchmark itself, yet the data, annotations, and evaluation code are only described as 'will be released soon' with no URL, repository, or supplementary material. For a benchmark paper, this is a load-bearing issue: none of the experimental results can be reproduced or independently verified, and the proposed evaluation protocol cannot be adopted by the community. To support the claims, the authors should provide either a public release of the benchmark and code or a clear, concrete release plan (e.g., repository URL and estimated date) in addition to a data card detailing the QA pairs, region masks, and human-review information. Without this, the contribution is not yet assessable as a usable artifact.
  3. [Sec. 3.7; Sec. A (fifth limitation)] The human quality assurance protocol is described as three annotators spending over 200 hours reviewing all 26,062 QA pairs, but the paper does not report inter-annotator agreement, the distribution of accept/edit/delete decisions, or the nature of edits made. The paper explicitly states this in Sec. A, but the absence of these numbers matters for the benchmark's reliability: if a large fraction of pairs were edited, or if annotators disagreed substantially, the ground truth labels may contain systematic noise that affects all subsequent metric values. I ask the authors to report at least the per-annotator accept/edit/delete counts, pairwise Cohen's kappa or similar agreement statistics, and a few examples of common edits. This is necessary to substantiate the claim that the benchmark has high-quality ground truth.
  4. [Sec. 3.2; Sec. A (third limitation)] The 'full-scene' claim is explicitly a design goal rather than a measured property, since region recall against human-enumerated salient entities is not reported. This is more than a terminological caveat: if the region set systematically misses certain kinds of content (e.g., small background objects, amorphous regions, or rare categories), then the QA probes derived from those regions will not cover those facts, and the benchmark's ability to expose coverage gaps becomes asymmetric across content types. The paper acknowledges this limitation, but it remains a load-bearing gap for the central claim that CapProbe checks 'full-scene' factual coverage. I request a human-annotated subset (e.g., 50 images) where annotators enumerate salient entities/regions, and a comparison of the YOLOv26-seg/SAM3 region set against that enumeration, reporting recall and missed-region characteristics. This would quantify the degree to which 'full-scene' holds in practice.
minor comments (4)
  1. [Sec. 5.4] The judge-reliability analysis is performed only on the top-3 caption models; the claim that rankings are 'stable' across judges would be stronger if evaluated on a broader subset of the 13 models, given that the Coverage gaps among lower-ranked models are smaller.
  2. [Sec. 3.6] The image-balancing step is described as reducing the dataset from 664 images and 39,127 QA pairs to 346 images, 1,868 regions, and 26,062 QA pairs, while Sec. 3.7 reports the final count as 25,650 after human deletion; the intermediate number 26,062 is correct, but the text could clarify that the 1.6% reduction is from human deletion, not from the balancing step.
  3. [Table 1] In the 'CapProbe (Ours)' row, the 'Probes/Img' column shows 74 and the 'Region' column shows ✓, but the table does not show the number of QA pairs per region (13.7) or the fact that the region count per image is only 5.4; adding these would help readers interpret the density statistics.
  4. [Sec. 4.2, Eq. (6)] The definition of per-image density d_i uses a factor of 1000 to express values in per-mille; the paper states this in prose, but the units ('‰') are only introduced in Table 3, not at the equation site, which could cause initial confusion.

Circularity Check

1 steps flagged · score 5.0 of 10

The headline Coverage ranking is partially self-referential: the QA benchmark is authored by Gemini-3.1-Pro and GPT-5.5, which are also evaluated caption models, and the paper concedes generator-style alignment can inflate Coverage; human fact-checking keeps the answer key image-grounded, so the circularity is partial and disclosed rather than complete.

  1. other [Sec. 5.2 Observation 1; Sec. A Limitations]
    "Both generators are also among the evaluated caption models. We therefore interpret strong Gemini-3.1-Pro Overall Accuracy and Coverage as results under the current Gemini-assisted construction pipeline (Gemini metadata; Gemini/GPT QA), and leave cross-generator validation—e.g., independent non-Gemini QA subsets—to future work. ... wording, fact granularity, category emphasis, and distractor style may still align with the generators’ captioning habits."

    Coverage is defined (Eq. 4) as the judge's non-E answer rate, and the paper's headline separation between models is driven almost entirely by Coverage while Effective Accuracy stays tightly clustered (92.7–94.5%). The questions and options the judge reads were authored by Gemini-3.1-Pro and GPT-5.5, both among the 13 evaluated caption generators, and the paper concedes that question wording and distractor style may align with the generators' habits. A Gemini-style caption therefore has a construction advantage in triggering non-E commitments on Gemini/GPT-written MCQs, inflating its Coverage without demonstrating greater factual coverage.

full rationale

The paper's metric definitions (Eqs. 1–11) are internally consistent and not circular: Overall = Effective Accuracy × Coverage is a designed decomposition, and the density metrics are explicit per-token ratios. There is no load-bearing self-citation: the authors do not cite their own prior work to ground the benchmark, and external references such as CaptionQA are used for comparison rather than to justify CapProbe's validity. There is no ansatz smuggled via citation, no imported uniqueness theorem, and no renaming of a known result. The one genuine self-referential element is that the QA instrument is authored by two of the same models that are then ranked on it; the paper discloses this in Sec. A and explicitly conditions the Gemini-3.1-Pro result on the current Gemini-assisted construction pipeline. This is not full equivalence because each MCQ is human fact-checked against the image, so the answer key is not derived from any caption. However, the central empirical claim—large Coverage gaps with Gemini on top—is not independent of benchmark construction, since Coverage is the judge's willingness to commit and can be inflated by style overlap with the generator-authored questions. Because this affects the paper's main observation that models differ mainly in coverage rather than committed-answer accuracy, the circularity score is moderate rather than zero.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central evaluation rests on several domain assumptions: judge-as-reader validity, adequacy of the automatically segmented region set, human QA correctness, and judge stability under 8-bit quantization. Two manual thresholds and an image cap are free design parameters. The paper introduces two soft constructs, the uncertain-option proxy and the full-scene region set, both explicitly hedged as judge-dependent or design-goal rather than independently validated.

free parameters (3)
  • Deduplication similarity thresholds = global 0.90, local 0.85
    Chosen by sampling a subset with human verification (Sec. 3.5). They control which QA pairs survive and therefore shape the benchmark composition.
  • L2 image cap = 1-5 images per L2 sub-domain, capped at 5
    Manual balancing decision in Sec. 3.6. It makes the suite taxonomy-diverse but prevents stable per-L2 performance estimates.
  • QA generation target per region = 15-20+ QA pairs per region
    Prompt instruction in Sec. B.2 asks for the maximum possible number of QA pairs and aims for 15-20+; this shapes the final density of 13.7 QA pairs per region after deduplication.
assumptions (4)
  • domain assumption A language judge answering MCQs from caption text alone is a valid proxy for the factual content of the caption.
    Core evaluation protocol in Sec. 3.9 and Sec. 4. The paper itself states absolute scores are judge-dependent.
  • domain assumption YOLOv26-seg plus SAM3 produce a region set adequate for full-scene evaluation.
    Region set defines which visual facts are probed (Sec. 3.2). The paper does not measure recall against human-enumerated salient entities.
  • domain assumption Human review of the generated QA pairs ensures ground-truth correctness and option quality.
    Human QC in Sec. 3.7 is the main independent check on automatically generated questions; no full inter-annotator agreement is reported.
  • domain assumption Qwen3-32B with 8-bit quantization is a stable enough reader for comparative ranking.
    Used as the default judge in Sec. 5.1. Judge reliability in Sec. 5.4 shows absolute scores vary with judge, while top-3 ranks remain stable.
invented entities (2)
  • Effective Accuracy and the Uncertain option as a proxy for omission versus hallucination
    purpose: Separate unanswered probes from incorrectly resolved answers
    The paper explicitly labels this a judge-dependent proxy, not a validated causal measure (Sec. 3.9, Sec. 4.1, Sec. A).
  • Full-scene region set
    purpose: Provide coverage of foreground and background evaluation targets
    The paper defines full-scene as a design goal, not verified exhaustive coverage; no region recall is reported (Sec. 3.2, Sec. A).

how reviews work

0 comments
Cite this review

Pith. "Pith review of CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering." pith.science (2026). https://pith.science/paper/ZDK6QXNG

@misc{pith2026260811074,
  author       = {Pith},
  title        = {Pith review of: CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZDK6QXNG}},
  note         = {Machine review of arXiv:2608.11074}
}
read the original abstract

Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity. Reference-based metrics (e.g., CIDEr and SPICE) and LLM-as-scorer protocols struggle to verify dense factual claims, while existing QA-based alternatives generally offer lower probe density, narrower domain coverage, or no explicit alignment between individual questions and segmented image regions. We introduce CapProbe, a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking. Each image is decomposed into coarse semantic regions covering both foreground and background elements; for every retained region, we generate multiple-choice questions spanning 10 semantic categories, forming a dense checklist of probed visual facts. Guided by a two-tier taxonomy of 37 L1 domains and 219 L2 sub-domains, CapProbe comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image. A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones, while density-based metrics penalize verbose yet uninformative captions. The protocol is cost-effective: by converting unconstrained scalar scoring into structured MCQ reading, it reduces open-ended scoring bias while remaining judge-conditioned and yields relatively stable model rankings under a fixed reader. Experiments on 13 VLMs show large Coverage gaps across models, a clear competency-efficiency trade-off, and failure modes that sparse or overlap-based evaluation often misses. The benchmark data, annotations, and evaluation code will be released soon.

Figures

Figures reproduced from arXiv: 2608.11074 by the authors.

Figure 1
Figure 1. Construction and evaluation pipeline of CapProbe. We select images under a hierarchical domain taxonomy, segment semantic regions, annotate structured metadata, generate and deduplicate dense QA pairs, balance domain coverage, and apply human quality assurance; captions are then evaluated via caption-based QA answering. detailed caption may convey, organized into five modules: • Spatial & Pose: bounding box, global … view at source ↗
Figure 2
Figure 2. Hierarchical domain coverage of CapProbe. The inner ring shows 37 L1 domains and the outer ring shows 219 L2 sub-domains; sector area at each level is proportional to the number of benchmark images. The complete hierarchical tag system is provided in Tab. 5. pairwise cosine similarity is computed for every QA pair within the same image. Two-level deduplication. We adopt a hierarchical strategy with two thresholds (w… view at source ↗
Figure 3
Figure 3. Distribution of QA pairs across 10 categories. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Coverage vs. Global Density for each caption model. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Complete prompt for structured metadata generation per region. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Complete prompt for dense QA generation per region. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 9
Figure 9. Figure 9: Per-L1-domain Overall Accuracy (%) for six representative caption models across all 37 L1 domains. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Qualitative sample on CapProbe: one image, captions from three models, and three QA probes (Attributes, Recognition, and Camera Features). 21 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 42 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Nocaps: Novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. InProceedings of the IEEE/CVF international conference on computer vision, pages 8948– 8957, 2019. 2, 8

    work page 2019

  3. [3]

    Introducing Qwen3.5.https:// qwen.ai/blog?id=qwen3.5, 2026

    Alibaba Qwen Team. Introducing Qwen3.5.https:// qwen.ai/blog?id=qwen3.5, 2026. 1, 10, 11, 12

    work page 2026

  4. [4]

    Introducing Qwen3.7-Plus.https: //qwen.ai/blog?id=qwen3.7-plus, 2026

    Alibaba Qwen Team. Introducing Qwen3.7-Plus.https: //qwen.ai/blog?id=qwen3.7-plus, 2026. 1, 10, 11, 12

    work page 2026

  5. [5]

    Spice: Semantic propositional image caption evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. InEuropean conference on computer vision, pages 382–398. Springer, 2016. 1, 3

    work page 2016

  6. [6]

    Introducing Claude 4.8 Opus.https://www

    Anthropic. Introducing Claude 4.8 Opus.https://www. anthropic.com/news/claude- opus- 4- 8, 2026. 1, 10, 11

    work page 2026

  7. [7]

    Introducing Claude 5 Sonnet.https://www

    Anthropic. Introducing Claude 5 Sonnet.https://www. anthropic.com/news/claude- sonnet- 5, 2026. 10, 11

    work page 2026

  8. [8]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025. 1, 4, 10, 11

    arXiv 2025

Show all 61 references
  1. [9]

    METEOR: An auto- matic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An auto- matic metric for MT evaluation with improved correlation with human judgments. InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Ar...

    2005

  2. [10]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Jian, Joon Sung Cho, Clarence Laplace, Aditya Bordes, Ben Zhou, and Aditya Ramesh. Improving image generation with better captions. Technical report, OpenAI, 2023. 1

    2023

  3. [11]

    Sam 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719, 2025

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719, 2025. 2, 4

    2025 arXiv

  4. [12]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Lin Bin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions. InAdvances in Neural...

    2024

  5. [13]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015. 2, 3, 8

    2015 arXiv

  6. [14]

    Caparena: Benchmarking and analyzing detailed image captioning in the llm era

    Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen. Caparena: Benchmarking and analyzing detailed image captioning in the llm era. In Findings of the Association for Computational Linguistics: ACL 202...

    2025

  7. [15]

    Probabilistic embeddings for cross-modal retrieval

    Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8415–8424, 2021. 1

    2021

  8. [16]

    Benchmarking and improv- ing detail image caption.arXiv preprint arXiv:2405.19092,

    Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Benchmarking and improv- ing detail image caption.arXiv preprint arXiv:2405.19092,

    arXiv

  9. [17]

    Caponimage: Context-driven dense-captioning on image

    Yiqi Gao, Xinglin Hou, Yuanmeng Zhang, Tiezheng Ge, Yuning Jiang, and Peng Wang. Caponimage: Context-driven dense-captioning on image. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3449–3465, 2022. 2

    2022

  10. [18]

    Gemini 3.1 pro: A smarter model for your most complex tasks.https://blog.google/ innovation - and - ai / models - and - research / gemini-models/gemini-3-1-pro/, 2026

    Google DeepMind. Gemini 3.1 pro: A smarter model for your most complex tasks.https://blog.google/ innovation - and - ai / models - and - research / gemini-models/gemini-3-1-pro/, 2026. 1, 2, 4, 5, 10, 11, 12, 15

    2026

  11. [19]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 3, 11

    2024 arXiv

  12. [20]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019. 2, 4

    2019

  13. [21]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  14. [22]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 conference on empirical methods in natural language pro- cessing, pages 7514–7528, 2021. 3

    2021

  15. [23]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 3, 10, 11

    2024 arXiv

  16. [24]

    Ultralytics yolo26: Unified real-time end-to-end vision models.arXiv preprint arXiv:2606.03748, 2026

    Glenn Jocher, Jing Qiu, Mengyu Liu, Shuai Lyu, Fatih Ca- gatay Akyon, and Muhammet Esat Kalfaoglu. Ultralytics yolo26: Unified real-time end-to-end vision models.arXiv preprint arXiv:2606.03748, 2026. 2, 4 13

    2026 arXiv

  17. [25]

    Densecap: Fully convolutional localization networks for dense caption- ing

    Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense caption- ing. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4565–4574, 2016. 2, 3

    2016

  18. [26]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection...

    1956

  19. [27]

    Denseworld-1m: Towards detailed dense grounded caption in the real world.arXiv preprint arXiv:2506.24102,

    Xiangtai Li, Tao Zhang, Yanwei Li, Haobo Yuan, Shihao Chen, Yikang Zhou, Jiahao Meng, Yueyi Sun, Shilin Xu, Lu Qi, et al. Denseworld-1m: Towards detailed dense grounded caption in the real world.arXiv preprint arXiv:2506.24102,

    arXiv

  20. [28]

    Evaluating object hallucination in large vision- language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 292–305, 2023. 3, 8

    2023

  21. [29]

    Describe anything: Detailed localized image and video captioning

    Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video captioning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2176...

    2025

  22. [30]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summarization Branches Out, pages 74–81, Barcelona, Spain, 2004. Association for Computa- tional Linguistics. 3

    2004

  23. [31]

    Capability: A comprehensive visual caption benchmark for evaluating both correctness and thoroughness.Advances in Neural Information Processing Systems, 38, 2026

    Zhihang Liu, Chen-Wei Xie, Bin Wen, Feiwu Yu, Pan- deng Li, Boqiang Zhang, Nianzu Yang, Zuan Gao, Yun Zheng, and Hongtao Xie. Capability: A comprehensive visual caption benchmark for evaluating both correctness and thoroughness.Advances in Neural Information Processing Systems...

    2026

  24. [32]

    Benchmarking large vision-language models via directed scene graph for comprehensive image captioning

    Fan Lu, Wei Wu, Kecheng Zheng, Shuailei Ma, Biao Gong, Jiawei Liu, Wei Zhai, Yang Cao, Yujun Shen, and Zheng- Jun Zha. Benchmarking large vision-language models via directed scene graph for comprehensive image captioning. In Proceedings of the Computer Vision and Pattern Recog...

    2025

  25. [33]

    Tiktoken: A fast bpe tokeniser for use with openai’s models, 2025

    OpenAI. Tiktoken: A fast bpe tokeniser for use with openai’s models, 2025. 10

    2025

  26. [34]

    Introducing GPT-5.5.https://openai.com/ index/introducing-gpt-5-5/, 2026

    OpenAI. Introducing GPT-5.5.https://openai.com/ index/introducing-gpt-5-5/, 2026. 1, 2, 5, 10, 11

    2026

  27. [35]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311– 318, Philadelphia, Pennsylvania, USA, 2002. Associ...

    2002

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  29. [37]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026. 1, 10, 11

    2026 arXiv

  30. [38]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 3

    2023 arXiv

  31. [39]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. 1, 3

    2015

  32. [40]

    Mitigating fine-grained hallucination by fine- tuning large vision-language models with caption rewrites

    Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee- Peng Lim. Mitigating fine-grained hallucination by fine- tuning large vision-language models with caption rewrites. InInternational Conference on Multimedia Modeling, pages 32–45. Springer, 2024. 3

    2024

  33. [41]

    Q-bench: A benchmark for general-purpose foundation models on low-level vision

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. InInternational Conference on Learning Representations, pages 1...

    2024

  34. [42]

    Grit: A gener- ative region-to-text transformer for object understanding

    Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A gener- ative region-to-text transformer for object understanding. In European Conference on Computer Vision, pages 207–224. Springer, 2024. 2, 3

    2024

  35. [43]

    Caprl: Stimulating dense image caption capabilities via reinforcement learning.arXiv preprint arXiv:2509.22647,

    Long Xing, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jianze Liang, Qidong Huang, Jiaqi Wang, Feng Wu, and Dahua Lin. Caprl: Stimulating dense image caption capabilities via reinforcement learning.arXiv preprint arXiv:2509.22647,

  36. [44]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 10, 11

    2025 arXiv

  37. [45]

    Cap- tionqa: Is your caption as useful as the image itself? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23741–23750, 2026

    Shijia Yang, Yunong Liu, Bohan Zhai, Ximeng Sun, Zicheng Liu, Emad Barsoum, Manling Li, and Chenfeng Xu. Cap- tionqa: Is your caption as useful as the image itself? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23741–23750, 2026. 1...

    2026

  38. [46]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. InInternational Conference on Learning Representations, pages 57153–57180, 2024. 2, 3

    2024

  39. [47]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions.Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 2 14

    2014

  40. [48]

    Wein- berger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Wein- berger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. 1

    2020

  41. [49]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025. 5

    2025 arXiv

  42. [50]

    Dreamlip: Language- image pre-training with long captions

    Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language- image pre-training with long captions. InEuropean Confer- ence on Computer Vision, pages 73–90. Springer, 2024. 2, 3

    2024

  43. [51]

    full-scene

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition.IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2017. 2, 4 A. Limitations We highlight several limitations ofCapProbe.First...

    2017

  44. [52]

    Masked Image: Used ONLY for your reference to identify which region to focus on

  45. [55]

    Original Image: The full scene context

  46. [63]

    7 8Task: 9Generate the MAXIMUM possible number of QA pairs

    Meta Info: {META_INFO} (Contains the ground-truth attributes and precise global position). 7 8Task: 9Generate the MAXIMUM possible number of QA pairs. These questions will be used to verify if a caption correctly describes the specific region. 10 11Requirements: 12

  47. [74]

    object_id

    Style & Output 8- 4 options (A, B, C, D), one correct answer. 9- Randomize the Position of the Correct Answer: Do NOT always place the correct answer in option A. You MUST randomly distribute the correct answer among A , B, C, and D for each question. 10- Distractors must be p...

  48. [91]

    Do not hallucinate

    Conditional Fields: If a field does not apply to the target (e.g., ’text_and_markings’ for an empty sky , or ’interaction_targets’ for a road), return ‘ null‘ or an empty array ‘[]‘. Do not hallucinate

  49. [102]

    Coordinates: Provide ’bounding_box’ in [ymin, xmin, ymax, xmax] format, normalized (0-1000) based on Image 1

  50. [113]

    spatial_and_localization

    Element & Component Audit: For objects, list sub- parts (e.g., wheels). For background regions, list constituent elements (e.g., tiles on a roof, patches on a road). 12 13Schema Structure: 14{ 15"spatial_and_localization": { 16"bounding_box": [ymin, xmin, ymax, xmax], 17"globa...

  51. [131]

    the dog,

    Absolute Zero Ambiguity (Critical) 14- You MUST assume the Original Image contains multiple similar objects (e.g., multiple dogs, multiple people). 15- Every ’question’ MUST start with or include a unique identifying description based on the ’ global_position’ and ’key_identif...

  52. [212]

    Natural Look

    "Natural Look" Constraint (No Technical Jargon) 22- NEVER mention "masked image," "highlighted region," " target," "masked area," or "bounding box" in the questions or options. 23- The questions must feel like they are asked by a human looking at the ORIGINAL image only. 24- A...

  53. [263]

    How many white patches are on the neck of the Bulldog

    Coverage Requirement (10 Categories) 27Exhaustively cover the target’s details using these categories: 28A. Recognition: Specific breed/type (e.g., English Bulldog, not just dog). 29B. Attributes: Details (head wrinkles, harness color, paw markings). 30C. OCR/Text: Any text/lo...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.