Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read COREVQA is a 5,608-pair true/false visual entailment benchmark for crowded scenes, and the paper shows that current vision-language models, led by GPT-4.1 at 77.57%, fail to reach 80% accuracy.

desk verdict A useful niche dataset whose evaluation section needs a majority-class baseline before its 'VLM struggle' claim can be taken seriously. read the letter →

arxiv 2507.13405 v1 pith:2XQHQCVV submitted 2025-07-17 cs.CV cs.LG

Ishant Chintapatla , Kazuma Choji , Naaisha Agarwal , Andrew Lin , Hannah You , Charles Duong , Kevin Zhu , Sean O'Brien
show 1 more author
Vasu Sharma
This is my paper
classification cs.CVcs.LG
keywords visualentailmentvision-languagemodelsVQAbenchmarkcrowdscenessyntheticquestiongenerationtrue/falsestatementsHumanmodelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

COREVQA is a benchmark of 5,608 image–statement pairs built from CrowdHuman crowd photographs, asking vision-language models to decide whether a synthetically written true/false statement is entailed by the image. The paper's central claim is that even the best tested models, GPT-4.1 at 77.57% accuracy, stay below 80%, with other models ranging from 39.98% to 76.60%, and that this gap exposes systematic weaknesses in visual entailment under crowded, cluttered conditions. The authors attribute the failures to recurring patterns: action recognition, oversight of small details, counting errors, spatial reasoning, and negation. If correct, the benchmark provides a diagnostic tool that existing crowd datasets, which focus on detection and counting, do not offer.

What carries the argument

The central object is the COREVQA benchmark and its synthetic generation pipeline. The pipeline pairs CrowdHuman images with statements crafted by two LLMs under separate true/false prompts that include strategies like quantifier bait, occlusion traps, and hidden contradictions, plus a self-reflection step that forces the generator to justify how the statement might deceive. Statements average 30 words, 57.7% use spatial terms, 39.0% reference clothing, and 35.1% mention color, and the binary true/false format is a key design choice meant to remove multiple-choice artifacts. Hand-labeling of all 5,608 ground truths is what makes the reported accuracy, precision, recall, and F1 scores interpretable as measures of visual entailment ability.

What would settle it

Run an always-false classifier over COREVQA, which would score 72.1% by construction; if top models' advantage over that trivial baseline is small, or if a random sample of statements re-labeled by independent annotators disagrees with the authors' labels, the benchmark's difficulty claim weakens.

Watch

Extended reading notes

Core claim

The paper introduces COREVQA, a benchmark pairing dense crowd images with synthetically generated true/false statements designed to require meticulous visual inspection and multi-step visual entailment. Statements are generated by prompting ChatGPT-4.1 for true claims and Claude 3 Opus for false claims, with prompt strategies such as quantifier bait, occlusion traps, and hidden contradictions, and every ground truth is hand-labeled because an automated solver reached only 89% agreement on a 250-pair sample. On the full set, evaluated models show that even the strongest performer, GPT-4.1, reaches only 77.57% accuracy, while GPT-4o mini scores 76.60%, Janus Pro 72.31%, Qwen2.5 vl 72b 69.95%, and LLaVa-NeXT 39.98%, with LLaVa-NeXT showing near-perfect recall but low precision and Janus Pro showing a strong bias toward answering false. These results support the claim that current vision-language models cannot reliably verify complex claims about dense crowd scenes, with failure patterns concentrated in action recognition, detail oversight, counting, spatial relations, and negation.

Load-bearing premise

The load-bearing premise is that the hand-labeled ground-truth answers are correct, even though no inter-annotator agreement is reported to show that the labels are reliable.

Editorial extensions

If this is right

  • If COREVQA scores reflect capability, current vision-language models are not reliable enough for automated judgment in crowd monitoring or safety applications, where below-80% accuracy on verified statements implies frequent errors.
  • The dataset's failure categories give a concrete diagnostic checklist: action recognition, fine-detail verification, counting, spatial relations, and negation are the specific skills that models need to improve.
  • Because statements are synthetically generated from images, the pipeline can be retargeted to new image collections with relatively little human effort beyond label verification, making new entailment benchmarks cheaper to produce.
  • Other benchmarks can adopt the true/false format to reduce the multiple-choice shortcut documented in LLM evaluation, since models cannot infer the correct answer from the answer choices themselves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper reports no always-false baseline; since 72.1% of statements are false, that trivial rule already scores 72.1%, so the meaningful headroom above chance is narrower than the raw accuracy numbers suggest.
  • A natural follow-up the paper does not run is per-category scoring: computing accuracy separately on counting, spatial, negation, and action subsets could show whether model rankings change by skill.
  • The same pipeline could generate entailment benchmarks for other dense domains, such as animal groups or vehicle traffic, to test whether the observed weaknesses are crowd-specific or general to clutter.
  • Because the labels were hand-labeled by the authors without reported inter-annotator agreement, an independent re-labeling study would be the cheapest way to harden the benchmark and confirm that model rankings reflect visual entailment rather than label noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces COREVQA, a benchmark of 5,608 image-statement pairs for visual entailment in crowded scenes. Images are taken from CrowdHuman, and true/false statements are synthetically generated via LLM prompts; ground truth is hand-labeled. Five vision-language models are evaluated, and the paper reports that even the best model (GPT-4.1) achieves only 77.57% accuracy, with other models ranging from 39.98% to 72.31%, interpreting this as evidence of fundamental limitations in VLM visual reasoning.

Significance. If the benchmark is validated and the empirical claims are properly baselined, COREVQA would be a useful resource for studying fine-grained visual entailment under visual clutter, a relatively under-tested capability. The authors provide a public dataset, code, and detailed generation prompts, and they report evaluations of several current models. However, the significance of the main empirical finding is currently weakened by the absence of a majority-class baseline and the lack of label-reliability evidence, both of which are directly addressable.

major comments (3)
  1. [Section 4.2, Table 3; Table 1] The evaluation does not include a majority-class baseline or per-class accuracy. Because 72.1% of statements are false (Table 1), a trivial always-false classifier achieves 72.1% accuracy. Janus Pro's reported 72.31% accuracy with recall of 1.85% and F1 of 3.60 is behaviorally indistinguishable from such a classifier, so the central claim that models 'struggle' on COREVQA is not established for this model. For GPT-4.1 (77.57%) and GPT-4o mini (76.60%), the margin over the always-false baseline is only about 5 percentage points. The paper should explicitly compare every model against the always-false baseline and report balanced accuracy or per-class accuracy. This is load-bearing for the abstract's claim that results 'reveal key limitations'; the A.7.2 finding that Janus Pro scores 64.0% on 'difficult' cases while GPT-4.1 scores 0.3% further illustrates that aggregate accuracy can mislead when the label prior is ignored.
  2. [Section 3.3] The ground truth is hand-labeled, but the paper reports no number of annotators, no annotation disagreement-resolution protocol, and no inter-annotator agreement measure (e.g., Cohen's kappa) on a sample. Since every model score is computed against these labels, the benchmark's validity rests entirely on their correctness. The sentence 'To ensure complete accuracy of ground truths, the entire dataset was manually labeled' is an assertion, not a demonstration. The authors should provide inter-annotator agreement statistics on a subset or otherwise document a second-pass verification process so that readers can assess label reliability.
  3. [Section 4.3 and A.7.2] The 'difficult cases' used for the failure-pattern analysis are defined as questions where at least two of the five evaluated models answer incorrectly. This makes the reported failure-pattern percentages (81.3% action recognition failures, 78.1% detail oversight, etc.) dependent on the particular model set and, if used to infer general VLM weaknesses, is circular. The percentages could change substantially with a different set of models or with an independent notion of difficulty (e.g., human response time or label agreement). The authors should either define difficulty using an independent criterion or present the failure analysis per model with appropriate caveats about the model-dependent subset.
minor comments (5)
  1. [Section 4.1] The paper does not describe how model outputs were parsed into 'True' or 'False' (e.g., exact-match, case sensitivity, handling of verbose responses). This detail is needed for reproducibility.
  2. [Section 4.2 and Table 3] No confidence intervals or significance tests are reported. For N=5,608, the 0.97-point accuracy difference between GPT-4.1 (77.57%) and GPT-4o mini (76.60%) is within sampling error, so the statement that 'GPT-4.1 achieves the highest overall accuracy' should be qualified or accompanied by statistical support.
  3. [Section 3.2.2 and Table 1] The paper does not report how many true and false statements were generated by ChatGPT-4.1 versus Claude Opus, making it impossible to assess whether the label imbalance (27.9% true) is an artifact of one generator or of the generation prompts.
  4. [Table 2] Table 2 contains a formatting issue ('Y es' appears as two words), and the comparison to SNLI-VE should clarify that SNLI-VE is also a visual entailment benchmark; the current row makes COREVQA appear unique in a way that the text later qualifies.
  5. [Section 5.1] The limitation that 'using a true or false format does not provide insight into key areas where a model went wrong' is contradicted by the failure analysis in A.7.2; either clarify the intended meaning or revise the sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: COREVQA is a hand-labeled benchmark evaluated on external VLMs, with no fitted parameters, self-citation chain, or prediction that reduces to its own inputs.

full rationale

The paper is a benchmark construction and evaluation study, not a derivation or prediction chain. The central claim is that state-of-the-art VLMs score below 80% accuracy on COREVQA, and that this reveals limitations in visual entailment reasoning. Nothing in this claim is equivalent to the paper's inputs by construction. The ground-truth labels were manually labeled (Section 3.3: 'To ensure complete accuracy of ground truths, the entire dataset was manually labeled'), independent of any of the evaluated models, so model scores are not fitted values or self-referential predictions. The true/false statements were generated by ChatGPT and Claude, but the benchmark's evaluation target is model performance on the resulting fixed dataset, not a quantity derived from the generators' own outputs. The paper does not invoke a uniqueness theorem, does not import a load-bearing result from the authors' prior work, and does not rename a known empirical pattern as a new organization. The acknowledged imbalance (72.1% false statements, Section 3.1) and the absence of a majority-class baseline are genuine evaluation-quality limitations, but they are matters of experimental control and statistical interpretation rather than circularity: reporting accuracy against an unbalanced test set does not make the accuracy score an input to itself. Similarly, the low recall of Janus Pro (Section 4.2) is an empirical finding about model bias, not a constructed equivalence. Because there is no fitted parameter renamed as a prediction and no self-citation sustains the central claim, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim (that VLMs struggle on COREVQA) depends on the accuracy of the manual ground-truth labels and on the realism of CrowdHuman images, neither of which is externally verified in the paper. The benchmark design also implicitly assumes that LLM-generated statements provide useful coverage of visual entailment challenges.

assumptions (3)
  • domain assumption CrowdHuman images are representative of real-world crowded scenes.
    The benchmark's generalization claims rest on this assumption; images come only from CrowdHuman train01 and train02 splits (Section 3.4.1).
  • domain assumption Manual ground-truth labels are correct despite no inter-annotator agreement.
    Section 3.3 states the entire dataset was manually labeled after automatic labeling achieved only 89% accuracy, but no reliability statistics are reported.
  • domain assumption LLM-generated statements cover a useful and challenging distribution of visual entailment tasks.
    The generation prompts in Appendix A.1 are designed for trickiness, but the paper does not validate the diversity or difficulty distribution independently of the model evaluations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark." pith.science (2026). https://pith.science/paper/2XQHQCVV

@misc{pith2026250713405,
  author       = {Pith},
  title        = {Pith review of: COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2XQHQCVV}},
  note         = {Machine review of arXiv:2507.13405}
}
read the original abstract

Recently, many benchmarks and datasets have been developed to evaluate Vision-Language Models (VLMs) using visual question answering (VQA) pairs, and models have shown significant accuracy improvements. However, these benchmarks rarely test the model's ability to accurately complete visual entailment, for instance, accepting or refuting a hypothesis based on the image. To address this, we propose COREVQA (Crowd Observations and Reasoning Entailment), a benchmark of 5608 image and synthetically generated true/false statement pairs, with images derived from the CrowdHuman dataset, to provoke visual entailment reasoning on challenging crowded images. Our results show that even the top-performing VLMs achieve accuracy below 80%, with other models performing substantially worse (39.98%-69.95%). This significant performance gap reveals key limitations in VLMs' ability to reason over certain types of image-question pairs in crowded scenes.

Figures

Figures reproduced from arXiv: 2507.13405 by the authors.

Figure 1
Figure 1. showcases a striking example where all tested models unanimously failed on what appears to be a straight￾forward visual assessment. Despite the seemingly precise description, all five models incorrectly classified this state￾ment as true when the ground truth is false. This example illustrates VLMs’ tendency to generate plau￾sible but factually incorrect interpretations when precise action recognition is required. T… view at source ↗
Figure 2
Figure 2. Agreed (Incorrect) Model Answer: TRUE 8 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Agreed (Incorrect) Model Answer: TRUE 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling Mobile Chaos Testing with AI-Driven Test Execution

    cs.SE 2026-02 conditional novelty 6.0 of 10

    An integrated LLM-based mobile testing system and service-level fault injector ran 180,000+ chaos tests at Uber, finding 23 resilience defects that manual and backend-only testing missed.

Reference graph

Works this paper leans on

20 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [4]

    Janus-pro: Unified multimodal understand- ing and generation with data and model scaling

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., and Ruan, C. Janus-pro: Unified multimodal understand- ing and generation with data and model scaling. arXiv preprint arXiv:2501.17811,

    arXiv

  3. [7]

    Li, B., Lin, Z., Peng, W., Nyandwi, J. d. D., Jiang, D., Ma, Z., Khanuja, S., Krishna, R., Neubig, G., and Ra- manan, D. Naturalbench: Evaluating vision-language models on natural adversarial samples. arXiv preprint arXiv:2410.14669,

    arXiv

  4. [8]

    Bench- mark evaluations, applications, and challenges of large vision language models: A survey

    Li, Z., Wu, X., Du, H., Nghiem, H., and Shi, G. Bench- mark evaluations, applications, and challenges of large vision language models: A survey. arXiv preprint arXiv:2501.02189, 1,

    arXiv

  5. [9]

    Liu, H., Li, C., Li, Y ., Li, B., Zhang, Y ., Shen, S., and Lee, Y . J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL https://llava-vl.github.io/blog/ 2024-01-30-llava-next/ . Liu, Y ., Duan, H., Zhang, Y ., Li, B., Zhang, S., Zhao, W., Yuan, Y ., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-...

    work page 2024

  6. [10]

    NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models

    URL https://openai.com/index/gpt-4-1/. An- nouncement of GPT-4.1 model series. Pandya, P., Gupta, V ., Talwarr, A. S., Kataria, T., Roth, D., and Gupta, V . Ntsebench: Cognitive reasoning benchmark for vision language models. arXiv preprint arXiv:2407.10380,

    work page Pith review arXiv

  7. [11]

    Valse: A task-independent bench- mark for vision and language models centered on lin- guistic phenomena

    Parcalabescu, L., Cafagna, M., Muradjan, L., Frank, A., Calixto, I., and Gatt, A. Valse: A task-independent bench- mark for vision and language models centered on lin- guistic phenomena. arXiv preprint arXiv:2112.07566 ,

    arXiv

  8. [13]

    Crowdhuman: A benchmark for detecting human in a crowd

    Shao, S., Zhao, Z., Li, B., Xiao, T., Yu, G., Zhang, X., and Sun, J. Crowdhuman: A benchmark for detecting human in a crowd. arXiv preprint arXiv:1805.00123,

    arXiv

Show all 20 references
  1. [15]

    Nwpu-crowd: A large-scale benchmark for crowd counting and localiza- tion

    Wang, Q., Gao, J., Lin, W., and Li, X. Nwpu-crowd: A large-scale benchmark for crowd counting and localiza- tion. IEEE transactions on pattern analysis and machine intelligence, 43(6):2141–2149, 2020a. Wang, X., Zhang, X., Zhu, Y ., Guo, Y ., Yuan, X., Xiang, L., Wang, Z., Din...

    1901 arXiv

  2. [16]

    Mmt- bench: A comprehensive multimodal benchmark for eval- uating large vision-language models towards multitask agi

    Ying, K., Meng, F., Wang, J., Li, Z., Lin, H., Yang, Y ., Zhang, H., Zhang, W., Lin, Y ., Liu, S., et al. Mmt- bench: A comprehensive multimodal benchmark for eval- uating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006,

    arXiv

  3. [17]

    Mm-vet: Evaluating large multi- modal models for integrated capabilities

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint arXiv:2308.02490,

    arXiv

  4. [18]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Duan, Y ., Tian, H., Su, W., Shao, J., et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479,

    arXiv

  5. [19]

    When in doubt, use more conserva- tive qualifiers

    Avoid statements that: - Are immediately obvious from a glance - Merely describe the presence of common objects - Could apply to many similar images - Require speculation beyond what’s visually evident - Make universal claims about all people or objects in the image IMPORTANT:...

    2024

  6. [20]

    COREVQA requires models to perform multi-step verification by decomposing complex claims and meticu- lously verifying each component against visual evidence

    offers partial fine-grained verification, it uses primar- ily simpler imagery and lacks our systematic adversarial design. COREVQA requires models to perform multi-step verification by decomposing complex claims and meticu- lously verifying each component against visual eviden...

    2019

  7. [2015]

    Ar- tifacts or abduction: How do llms answer multiple- choice questions without the question? arXiv preprint arXiv:2402.12483,

    Balepur, N., Ravichander, A., and Rudinger, R. Ar- tifacts or abduction: How do llms answer multiple- choice questions without the question? arXiv preprint arXiv:2402.12483,

    arXiv

  8. [2019]

    M3gia: A cognition inspired multilingual and multimodal general intelligence ability benchmark

    Song, W., Li, Y ., Xu, J., Wu, G., Ming, L., Yi, K., Luo, W., Li, H., Du, Y ., Guo, F., et al. M3gia: A cognition inspired multilingual and multimodal general intelligence ability benchmark. arXiv preprint arXiv:2406.05343,

    arXiv

  9. [2021]

    R., Bashir, S

    Raza, S., Vayani, A., Jain, A., Narayanan, A., Khazaie, V . R., Bashir, S. R., Dolatabadi, E., Uddin, G., Emmanouilidis, 5 COREVQA C., Qureshi, R., et al. Vldbench: Vision language mod- els disinformation detection benchmark. arXiv preprint arXiv:2502.11361,

  10. [2023]

    Seed-bench: Benchmarking multimodal llms with gener- ative comprehension

    Li, B., Wang, R., Wang, G., Ge, Y ., Ge, Y ., and Shan, Y . Seed-bench: Benchmarking multimodal llms with gener- ative comprehension. arXiv preprint arXiv:2307.16125,

    arXiv

  11. [2024]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y ., Chen, Z., Duan, H., Wang, J., Qiao, Y ., Lin, D., et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    arXiv

  12. [2025]

    Mme-survey: A comprehensive survey on evaluation of multimodal llms

    Fu, C., Zhang, Y .-F., Yin, S., Li, B., Fang, X., Zhao, S., Duan, H., Sun, X., Liu, Z., Wang, L., et al. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.