REVIEW 3 major objections 4 minor 7 cited by
CountQA: How Well Do MLLMs Count in the Wild?
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A new benchmark shows multimodal AI models count objects in real photos correctly less than half the time.
desk verdict CountQA is a plausible new benchmark for dense real-world counting, but the abstract-only evidence leaves the headline 42.9% uninterpretable until annotation and question-generation details are released. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
CountQA itself is the central mechanism: a curated benchmark of real-world images paired with counting questions, designed to test models under high object density, clutter, and occlusion. The benchmark's question-answer pairs serve as the controlled instrument that measures counting accuracy across 15 MLLMs.
What would settle it
Take a random sample of CountQA question-answer pairs and have independent human annotators verify each object count against the source image. If substantial label disagreement or evidence that the questions were generated by an MLLM emerges, the central accuracy claim would need reinterpretation. Alternatively, if human-verified labels match the benchmark and a well-known MLLM scores significantly above 42.9% on a re-split, the reported ceiling would be called into question.
Extended reading notes
Core claim
The central claim is that multimodal large language models are fundamentally unreliable at counting objects in real-world scenes, not just in sparse or synthetic settings. CountQA is presented as a benchmark designed to expose this gap: it contains over 1,500 question-answer pairs on real images with high object density, clutter, and occlusion. Evaluating 15 MLLMs on this benchmark, the authors report that the top-performing model achieves only 42.9% accuracy, and that accuracy declines as object counts rise. The paper argues this shows a severe limitation in numerical grounding and spatial awareness that existing benchmarks fail to capture.
Load-bearing premise
The benchmark's ground-truth object counts and question-answer pairs are accurate and were not generated by the same class of multimodal models being evaluated; if the labels are noisy or the questions were written by an MLLM, the 42.9% accuracy could reflect annotation artifacts rather than genuine counting ability.
Editorial extensions
If this is right
- If the reported accuracy holds, current MLLMs cannot be deployed in real-world tasks that require trustworthy object counts, such as inventory, medical imaging, or autonomous inspection.
- Performance declining with object count implies the failure is not a fixed quirk but scales with task complexity, making dense scenes particularly unreliable.
- A dedicated benchmark like CountQA provides a concrete target for future training and evaluation, pushing models toward numerically grounded rather than purely descriptive visual understanding.
- The 42.9% top accuracy establishes a baseline that future MLLMs must surpass, making counting improvement measurable and comparable across models.
Reading between the lines
- The paper does not describe how ground-truth counts and question-answer pairs were annotated; if those labels were produced by an MLLM or by a noisy automated pipeline, the 42.9% figure could partly reflect annotation artifacts rather than pure counting ability.
- Counting ability is a natural probe for broader numerical and spatial grounding, so CountQA's results may predict weaknesses in other relational tasks such as estimating area, ordering objects, or comparing quantities.
- A testable extension is to vary image clutter and occlusion systematically while holding object count fixed, which would isolate whether the decline stems from numerosity itself or from visual difficulty.
- The benchmark's reliance on real-world images makes it sensitive to domain bias; cross-domain validation would clarify whether the measured gap generalizes beyond the selected image sources.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CountQA, a new benchmark of over 1,500 question-answer pairs built from real-world images with high object density, clutter, and occlusion. The authors evaluate 15 prominent multimodal large language models (MLLMs) on this benchmark and report that the best-performing model reaches only 42.9% accuracy, with accuracy declining as object counts increase. The stated goal is to diagnose a fundamental limitation of MLLMs in object counting and to encourage the development of more numerically grounded and spatially aware models. The paper promises to release the dataset and code upon acceptance.
Significance. If the benchmark's ground truth is valid and the evaluation protocol is sound, CountQA would fill an important gap: existing counting benchmarks tend to use sparse scenes or restricted domains, whereas real-world applications involve clutter and occlusion. The headline result—that 15 current MLLMs plateau below 50% accuracy and degrade with object count—would be practically significant for deployment decisions and for guiding future multimodal research. The explicit commitment to open-source the dataset and code is a strength. However, based on the abstract alone, the measurement's validity is not yet established: no annotation protocol, human baselines, question-generation method, or statistical analysis is reported, so the central numerical claims are currently unverifiable.
major comments (3)
- [Abstract] The central claim that the top model achieves only 42.9% accuracy, with performance declining as object counts rise, depends entirely on the validity of CountQA's ground-truth counts and question-answer pairs. The abstract does not describe how the ground truth was obtained, whether annotators were domain experts or MLLM-assisted, whether there was inter-annotator agreement, or how the questions were authored. If labels are noisy in dense scenes or if questions were auto-generated by an MLLM, the reported numbers could reflect label noise or model self-consistency rather than counting ability. The paper states it will open-source the data only upon acceptance, which further prevents independent verification of this load-bearing premise. This is not an internal inconsistency, but it is an unverified premise on which the paper's main conclusion rests.
- [Abstract] The decline in accuracy with object count is presented as a general result, but the abstract provides no per-count accuracy numbers, no distribution of object counts in the benchmark, and no confidence intervals. Higher-count images are typically harder for any estimator, human or machine; if the benchmark is skewed toward high counts, the declining trend may partly reflect image difficulty or label noise that also increases with count. The authors should provide a stratified analysis by count range, compare against human performance on the same images, and report statistical uncertainty. Without this, the trend claim is not yet established.
- [Abstract] The evaluation protocol is underspecified. The abstract names '15 prominent MLLMs' but does not list them, state the prompting scheme, decoding parameters, number of runs, or how accuracy was computed (exact match vs. tolerance, answer extraction, etc.). The 42.9% figure is a point estimate without variance; there is no way to assess whether differences among models are meaningful. For an empirical benchmark paper, the model roster and evaluation details are essential to interpreting the headline result, and they should at least be summarized in the abstract.
minor comments (4)
- [Abstract] The abstract states 'over 1,500 question-answer pairs' but does not indicate how many unique images this corresponds to. Since multiple questions can be generated from one image, the effective sample size for image-level conclusions may be much smaller. Please clarify.
- [Abstract] The phrase 'mere 42.9%' would benefit from context: what is the chance level for the task, and what is human performance on the same benchmark? A 42.9% accuracy can be difficult or easy to interpret without such a baseline.
- [Abstract] The 'top-performing model' is not named. At least one example or a reference to the full results would help readers calibrate the claim.
- [Abstract] The final sentence about paving the way for models that are 'numerically grounded and spatially aware' is somewhat promotional; consider making the benchmark's diagnostic value more concrete.
Circularity Check
No circularity: CountQA is an evaluation benchmark with no derivation chain that reduces to its inputs.
full rationale
The paper is an abstract-only benchmark introduction and evaluation study. It introduces CountQA, a benchmark of over 1,500 QA pairs on real-world images, and reports accuracy of 15 MLLMs on it (top 42.9%, declining with object count). There is no mathematical derivation, no fitted parameter renamed as a prediction, no self-citation chain invoked as a uniqueness theorem, and no ansatz smuggled in via citation. The central claim is an empirical measurement against a newly introduced benchmark; the benchmark construction is not described in the abstract, but that is a validity concern (possible annotation noise or MLLM-generated questions), not a circularity concern. Under the hard rules, circularity requires exhibiting a specific reduction where an output equals an input by construction. No such reduction appears in the provided text. Therefore the circularity score is 0. The residual risk flagged in the reader's take—undisclosed QA generation—would, if present, potentially undermine external validity but still would not constitute circularity unless the paper explicitly defined the benchmark in terms of the models' outputs or used the models' predictions to construct the ground truth and then called those predictions a finding. No such evidence is available from the abstract.
Assumptions & free parameters
free parameters (3)
- Benchmark size (1,500+ QA pairs) =
1,500+ (stated)
- Model roster (15 MLLMs) =
15 models, unnamed
- Object-count grouping for the trend claim =
Not stated
assumptions (4)
- domain assumption Accuracy on CountQA question-answer pairs measures object-counting ability.
- domain assumption Ground-truth counts and answers in the 1,500 QA pairs are correct and unambiguous.
- domain assumption The selected real-world images are representative of 'the wild' counting conditions.
- domain assumption The 15 evaluated models are prominent and representative of current MLLMs.
invented entities (1)
-
CountQA benchmark
Cite this review
Pith. "Pith review of CountQA: How Well Do MLLMs Count in the Wild?." pith.science (2026). https://pith.science/paper/QHORF3MX
@misc{pith2026250806585,
author = {Pith},
title = {Pith review of: CountQA: How Well Do MLLMs Count in the Wild?},
year = {2026},
howpublished = {\url{https://pith.science/paper/QHORF3MX}},
note = {Machine review of arXiv:2508.06585}
}
read the original abstract
Multimodal Large Language Models (MLLMs) demonstrate remarkable fluency in understanding visual scenes, yet they exhibit a critical lack in a fundamental cognitive skill: object counting. This blind spot severely limits their reliability in real-world applications. To date, this capability has been largely unevaluated in complex scenarios, as existing benchmarks either feature sparse object densities or are confined to specific visual domains, failing to test models under realistic conditions. Addressing this gap, we introduce CountQA, a challenging new benchmark designed to probe this deficiency. Comprising over 1,500 question-answer pairs, CountQA features real-world images with high object density, clutter, and occlusion. We investigate this weakness by evaluating 15 prominent MLLMs on the CountQA benchmark and reveal that the top-performing model achieves a mere 42.9% accuracy, with performance declining as object counts rise. By providing a dedicated benchmark to diagnose and rectify this core weakness, CountQA paves the way for a new generation of MLLMs that are not only descriptively fluent but also numerically grounded and spatially aware. We will open-source the dataset and code upon paper acceptance to foster further research.
Forward citations
Cited by 7 Pith papers
-
When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
SA-OPD filters on-policy distillation tokens that are both weakly input-grounded and highly divergent, improving math reasoning and visual understanding benchmarks over vanilla OPD and selective baselines.
-
Scaling Native Multimodal Pre-Training From Scratch
In models trained from scratch on text plus images, the text-objective scaling law is data-mix-invariant while the image-conditioned objective shifts toward many more tokens relative to parameters as the multimodal sh...
-
Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning
RLVR training on 64,000 procedurally generated Trace instances improves Qwen2.5-VL macro-average on 24 external visual reasoning benchmarks by 3.51 points at 3B and 4.06 points at 7B.
-
HoloCount: A Holistic Visual Counting Benchmark for MLLMs
HoloCount is a three-tier visual counting benchmark showing that MLLMs fail systematically on analytical reasoning, high-density scenes, and linguistic prior conflicts, with even the best models dropping below 50% acc...
-
V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoning
V-Zero trains MLLMs for visual reasoning without answer labels by gating on-policy distillation trajectories using contrastive evidence from relevant versus negative image crops.
-
Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
Thinking LLMs achieve ~10 percentage points higher accuracy than non-thinking ones on RewardBench with under 2x compute overhead, outperforming augmentation strategies that cost over 8x more while also showing better ...
-
ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation
ABACUS adapts a 3B unified foundation model using density-aware zooming, boundary-aware GRPO, and cycle-consistent self-critique to achieve SOTA on seven counting and generation benchmarks without task-specific training.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.