REVIEW 5 major objections 5 minor 37 references
Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A compact vision-language model fine-tuned on teacher-generated quality scores can filter an image-text corpus down to 18%, and a captioning model trained on that subset wins 59.4% of judge comparisons against the full-data model.
desk verdict A plausible compact-filter recipe whose headline result is undermined by using Gemini as both teacher and judge; the released artifacts make it worth a serious look, but the central claim needs an independent evaluation to hold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a two-stage score-distillation pipeline: a strong teacher model (Gemini 2.0-Flash) annotates image-caption pairs with continuous quality scores and explanatory rationales, and supervised fine-tuning transfers that judgment into a 2-billion-parameter compact VLM that can score new pairs without external modules or API calls. The resulting scalar score acts as a high-precision filter, with a retention threshold of 9 or above, and its quality is validated through CLIP cosine-similarity alignment, caption perplexity, and an LLM-as-a-judge downstream comparison.
What would settle it
Train the same compact scorer on labels from a different strong teacher, or replace the Gemini evaluation judge with human raters on the same 500 test images; if human preference for the filtered-model captions does not come out near 59.4%—or if the CLIP alignment gain vanishes under the re-trained filter—the reported advantage is an artifact of the teacher's own scoring bias rather than a property of filtered data.
Extended reading notes
Core claim
The central claim is that a compact vision-language model (Qwen2-VL-2B) can be fine-tuned to reproduce the quality-judgment behavior of a stronger teacher model and thereby act as a standalone scoring function for image-text pairs. The filter is trained on 5,000 balanced samples from Recap-COCO and CC12M, each annotated with a 1–10 score and a textual rationale; after supervised fine-tuning, it scores new pairs and the authors retain only those scoring at least 9. On a 20K-pair evaluation set this preserves 3.5K pairs, or 18% of the data. Compared with the full set and a same-size random subset, the filtered set shows higher CLIP-based image-text cosine similarity (0.313 versus 0.298 and 0.297, with a two-sample $t$-test $p = 4.27 \times 10^{-56}$) and lower caption perplexity (137.2 versus 170.2 and 168.6). In the downstream test, two identical ViT-GPT2 captioning models are trained on filtered versus full data, and a Gemini judge prefers the filtered model's captions for 297 of 500 test images, i.e. 59.4% of cases.
Load-bearing premise
The load-bearing premise is that Gemini 2.0-Flash's 1–10 quality scores are a fair measure of true image-text alignment, yet the same model family is also the judge in the headline 59.4% comparison, so any bias in the teacher would be baked into both the filter's training signal and the evaluation.
Editorial extensions
If this is right
- Filtering a web-scale image-text corpus with the compact scorer can cut training data to roughly 18% while keeping or improving downstream captioning quality, meaning the same caption quality can be reached with far less data and compute.
- Because the filter runs on a 2B-parameter model with no auxiliary scoring modules and no API calls, it can be deployed on-premises to screen large datasets before they enter a VLM pre-training pipeline.
- Lower caption perplexity and higher CLIP cosine similarity on the filtered set imply that the retained pairs are both more fluent and more semantically aligned, so models trained on them should generalize better and hallucinate less.
- The persistence of a 40.6% preference for the full-data model exposes a diversity cost: aggressive filtering removes some informative variation, so the optimal retention rate is not necessarily the 18% threshold studied here.
Reading between the lines
- Because the teacher's scores are both the training signal and the evaluation signal, the 59.4% preference result is best read as an upper bound on the filter's true benefit; an independent human or a differently trained judge could plausibly give a lower, though probably still positive, preference rate.
- The monotone increase in CLIP alignment across score buckets in the ablation suggests the scorer carries calibrated signal well beyond the binary 9-or-10 threshold, so a natural extension is to sweep the threshold and map the quality-diversity frontier rather than assume 18% is optimal.
- The same distillation recipe could be transferred to other domains: any strong teacher's scoring behavior could be compressed into a compact model, turning expensive API-based data auditing into a cheap on-premises screening step for other large multimodal corpora.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a compact VLM-based data filtration framework for image-text pairs. A Qwen2-VL-2B model is fine-tuned on 5,000 image-caption pairs annotated by Gemini 2.0-Flash with 1-10 quality scores and rationales. The resulting scorer is used to filter 20K pairs from CC3M and CC12M down to 3.5K pairs (18%) using a threshold of 9. The authors compare filtered, full, and random subsets using CLIP cosine similarity and caption perplexity, and they evaluate downstream captioning by fine-tuning identical ViT-GPT2 models on full versus filtered data, with Gemini 2.0-Flash as the judge, reporting a 59.4% preference rate for the filtered model. The paper claims that datasets filtered with this compact VLM perform on par with or surpass larger and noisier web-crawled datasets.
Significance. If the central claims hold, the work provides a practical, low-cost, on-prem data curation tool for VLM pre-training, and the released model, data, and scripts are a useful community resource. The idea of distilling a large teacher model's quality judgments into a compact scoring VLM is sensible, and the authors do report statistical testing and a bucket-level ablation in Section 5. However, the significance of the contribution is currently limited because the headline downstream result relies on the same model family as both teacher and judge, and because the abstract's claim about outperforming larger web-crawled datasets is not directly tested in the experiments.
major comments (5)
- [Sections 3.1 and 3.4/4.3] The teacher model Gemini 2.0-Flash generates the training labels for the filter (Section 3.1) and also serves as the judge for the headline captioning result (Sections 3.4 and 4.3), so the 59.4% preference rate may partly reflect Gemini's stylistic preferences rather than human-relevant image-text alignment. The manual review mentioned in Section 3.1 is asserted without a protocol, sample size, or inter-annotator agreement, so it does not establish that Gemini's scores are a valid ground-truth proxy. I recommend replacing or supplementing the LLM-as-a-judge evaluation with human preference judgments, a judge from a different model family, and standard captioning metrics such as CIDEr, BLEU, and METEOR.
- [Abstract and Section 3.3] The abstract claims that filtered datasets 'perform on par with, or even surpass, larger and noisier datasets gathered through high-volume web crawling,' but the experiments only compare a 3.5K filtered subset against the same 20K full set and a 3.5K random subset. No training run or evaluation against an actual larger web-scale dataset is reported, so this central claim is not supported by the evidence in the manuscript.
- [Section 4.2] The lower perplexity of the filtered captions is interpreted as evidence of higher linguistic quality, but perplexity is strongly affected by caption length and lexical predictability, and shorter or more generic captions may achieve lower perplexity without being better. The paper does not report caption length distributions, lexical diversity, or n-gram statistics for the three splits, so the conclusion that filtering improves 'linguistic fluency' is not yet established. I suggest length-matched controls and additional diversity metrics.
- [Section 4.3] The 59.4% preference rate is reported as 297/500 without a confidence interval or significance test, and the judge prompt is not provided, so the robustness of the result to prompt variation cannot be assessed. Standard captioning metrics are absent, leaving the magnitude of downstream improvement relative to common benchmarks unknown. A binomial confidence interval and results from at least one non-Gemini judge or human evaluation are needed before this can be taken as evidence of genuine quality gains.
- [Sections 4.1 and 5, Table 2] The CLIP cosine-similarity gains are statistically significant but small in absolute terms: the mean increases from 0.298 to 0.313 in Section 4.1, and the bucket averages in Table 2 range only from 0.27 to 0.32. The paper does not establish that differences of this size are practically meaningful for downstream VLM training. Reporting an effect size such as Cohen's d and linking filtration scores to downstream task performance would strengthen the interpretation.
minor comments (5)
- [Section 1] There are grammatical and stylistic errors, including 'significantly broadens' in the abstract and 'Our approach is cost- and resource-effective than previous approaches' in Section 1; these should be corrected.
- [Section 3.2] The training details are incomplete: the optimizer, warmup schedule, and whether the model was fully fine-tuned or LoRA-adapted are not specified, and no evaluation of the fine-tuned model's score distribution is provided.
- [Section 3.3] The filtration threshold of 9 is a free parameter, and no sensitivity analysis is reported. The paper should show how the downstream metrics vary with thresholds such as 8, 9, and 10.
- [Section 3.4] The composition of the 500-pair test set is not described: it is unclear where the images and ground-truth captions come from and whether they overlap with the training data of the captioning model.
- [References] Reference [1] is cited for Gemini 2.0-Flash, but the reference describes Gemini 1.5; the correct model or version should be cited. Also, Table 1 states 2.5K selected samples per dataset while the text says 5,000 total; please clarify the exact counts.
Circularity Check
Teacher/judge circularity: Gemini 2.0-Flash supplies the filter's training labels (Sec. 3.1) and the 59.4% caption-preference evaluation (Secs. 3.4 and 4.3), so the headline result partly measures Gemini's own stylistic preferences.
-
fitted input called prediction
[Section 3.1 (data annotation) with Section 3.4 and Section 4.3 (captioning evaluation)]
"Each data point received a numerical quality score (1–10) and a detailed textual explanation, allowing the downstream filtration model to learn both how to score and explain image-caption quality. ... To assess caption quality, we used an "LLM-as-a-judge" strategy, employing Gemini 2.0 Flash [1] as the evaluator. ... Strikingly, in 59.4% of cases (297/500), the judge favored the filtered model's caption over that of the full model."
The compact VLM is supervised to approximate Gemini 2.0-Flash's scoring behavior, so the filtered subset is selected according to Gemini's notion of image-text quality. The headline evaluation then uses the same Gemini model as the judge to compare captions from models trained on filtered vs. full data. Any systematic stylistic bias in Gemini (e.g., toward Recap-COCO-style longer captions) is therefore baked into the filter and rewarded by the judge. The 59.4% preference is not an independent measure of data quality; it partly measures how well the filtered model reproduces the teacher's taste.
full rationale
The paper's central headline number, the 59.4% judge preference in Section 4.3, is structurally dependent on Gemini 2.0-Flash because the same model family generated the filter's training labels (Section 3.1) and serves as the evaluation judge (Sections 3.4 and 4.3). The compact filter is explicitly trained to approximate the teacher's scoring, and the downstream judge is the same teacher; thus the reported 'quality' gain is partly a measure of agreement with Gemini's own preferences rather than an external ground truth. This is a genuine partial circularity, not merely a self-citation concern: the prediction (Gemini prefers filtered captions) is closely related by construction to the fitted input (Gemini scores). The paper does provide independent evidence—CLIP cosine similarity and perplexity—which is non-circular and prevents the entire derivation from collapsing, but those metrics are coarse and do not rescue the headline captioning claim. The asserted manual review in Section 3.1 lacks a protocol or inter-annotator agreement, so it does not break the teacher/judge coupling. No load-bearing self-citation or ansatz-smuggling is present. Overall, one central prediction is partially forced by the shared oracle, so the circularity score is 6.
Assumptions & free parameters
free parameters (1)
- filtration threshold =
score >= 9
assumptions (5)
- domain assumption Gemini 2.0-Flash's quality scores reflect ground-truth image-text alignment
- domain assumption CLIP cosine similarity is a valid proxy for image-text semantic alignment
- domain assumption Perplexity is a valid proxy for caption linguistic quality
- domain assumption Gemini-as-judge preference is a valid measure of captioning quality
- domain assumption Fine-tuning a 2B model on 5,000 examples can distill the teacher's scoring behavior
Cite this review
Pith. "Pith review of Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality." pith.science (2026). https://pith.science/paper/SR6W2XOH
@misc{pith2026250720156,
author = {Pith},
title = {Pith review of: Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality},
year = {2026},
howpublished = {\url{https://pith.science/paper/SR6W2XOH}},
note = {Machine review of arXiv:2507.20156}
}
read the original abstract
Vision-language models (VLMs) extend the conventional large language models by integrating visual data, enabling richer multimodal reasoning and significantly broadens the practical applications of AI. However, including visual inputs also brings new challenges in maintaining data quality. Empirical evidence consistently shows that carefully curated and representative training examples often yield superior results compared to simply increasing the quantity of data. Inspired by this observation, we introduce a streamlined data filtration framework that employs a compact VLM, fine-tuned on a high-quality image-caption annotated dataset. This model effectively evaluates and filters potential training samples based on caption and image quality and alignment. Unlike previous approaches, which typically add auxiliary filtration modules on top of existing full-scale VLMs, our method exclusively utilizes the inherent evaluative capability of a purpose-built small VLM. This strategy eliminates the need for extra modules and reduces training overhead. Our lightweight model efficiently filters out inaccurate, noisy web data, improving image-text alignment and caption linguistic fluency. Experimental results show that datasets underwent high-precision filtration using our compact VLM perform on par with, or even surpass, larger and noisier datasets gathered through high-volume web crawling. Thus, our method provides a lightweight yet robust solution for building high-quality vision-language training corpora. \\ \textbf{Availability and implementation:} Our compact VLM filtration model, training data, utility scripts, and Supplementary data (Appendices) are freely available at https://github.com/daulettoibazar/Compact_VLM_Filter.
Reference graph
Works this paper leans on
-
[19]
H. Liu et al., “Visual instruction tuning,” in Advances in Neural Information Processing Systems , A. Oh et al. , Eds., vol. 36, Curran Associates, Inc., 2023, pp. 34 892–34 916. [Online]. Available: https : / / proceedings.neurips.cc/paper_files/ paper/2023/file/6dcf277ea32ce3288914faf369fe6de0- Paper-Conference.pdf
work page 2023
-
[1]
, Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of con- text, 2024
Gemini Team et al. , Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of con- text, 2024. DOI: 10.48550/ARXIV.2403.05530. [Online]. Available: https://arxiv.org/abs/ 2403.05530
-
[2]
Wang et al., Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution ,
P. Wang et al., Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution ,
-
[3]
OpenAI et al., Gpt-4 technical report, 2023. DOI: 10. 48550/ARXIV.2303.08774. [Online]. Available: https://arxiv.org/abs/2303.08774
-
[4]
Bai et al., Qwen2.5-vl technical report, 2025
S. Bai et al., Qwen2.5-vl technical report, 2025. DOI: 10.48550/ARXIV.2502.13923. [Online]. Avail- able: https://arxiv.org/abs/2502.13923
-
[5]
LLaV A-onevision: Easy visual task transfer,
B. Li et al. , “LLaV A-onevision: Easy visual task transfer,” Transactions on Machine Learning Re- search, 2025, ISSN : 2835-8856. [Online]. Available: https : / / openreview . net / forum ? id = zKv8qULV6n
work page 2025
-
[6]
Lu et al., Deepseek-vl: Towards real-world vision- language understanding , 2024
H. Lu et al., Deepseek-vl: Towards real-world vision- language understanding , 2024. DOI: 10 . 48550 / ARXIV.2403.05525. [Online]. Available:https: //arxiv.org/abs/2403.05525
-
[7]
A. Masry et al. , Alignvlm: Bridging vision and lan- guage latent spaces for multimodal understanding ,
Show all 37 references
- [8]
-
[9]
Mplug-owi2: Revolutionizing multi- modal large language model with modality collab- oration,
Q. Ye et al. , “Mplug-owi2: Revolutionizing multi- modal large language model with modality collab- oration,” in 2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , 2024, pp. 13 040–13 051. DOI: 10 . 1109 / CVPR52733 . 2024.01239
2024
-
[10]
Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts,
S. Changpinyo et al., “Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021, pp. 3557–3567. DOI: 10.1109/CVPR46437. 2021.00356
2021
- [11]
-
[12]
Obelics: An open web-scale fil- tered dataset of interleaved image-text documents,
H. Laurenc ¸on et al., “Obelics: An open web-scale fil- tered dataset of interleaved image-text documents,” in Advances in Neural Information Processing Sys- tems, A. Oh et al. , Eds., vol. 36, Curran Associates, Inc., 2023, pp. 71 683–71 702. [Online]. Available: https : / / ...
2023
-
[13]
Bai et al
Y . Bai et al. , Let’s go shopping (lgs) – web-scale image-text dataset for visual concept understanding ,
-
[14]
Towards efficient visual-language align- ment of the q-former for visual reasoning tasks,
S. Kim et al., “Towards efficient visual-language align- ment of the q-former for visual reasoning tasks,” in Findings of the Association for Computational Linguistics: EMNLP 2024 , Association for Compu- tational Linguistics, 2024, pp. 15 155–15 165. DOI: 10.18653/v1/2024.fin...
2024 doi
-
[15]
Training language models to fol- low instructions with human feedback,
L. Ouyang et al. , “Training language models to fol- low instructions with human feedback,” inProceedings of the 36th International Conference on Neural Infor- mation Processing Systems , ser. NIPS ’22, New Or- leans, LA, USA: Curran Associates Inc., 2022, ISBN : 9781713871088
2022
- [16]
-
[17]
Lima: Less is more for alignment,
C. Zhou et al., “Lima: Less is more for alignment,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS ’23, New Orleans, LA, USA: Curran Associates Inc., 2023
2023
-
[18]
Agarwal and D
I. Agarwal and D. Hakkani-T ¨ur, Data valuation using neural networks for efficient instruction fine-tuning ,
- [20]
-
[21]
On the origin of hallucinations in con- versational models: Is it the datasets or the models?
N. Dziri et al., “On the origin of hallucinations in con- versational models: Is it the datasets or the models?” In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies , M. Carpuat et ...
2022
-
[22]
48550 / ARXIV
DOI: 10 . 48550 / ARXIV . 2502 . 09969. [Online]. Available: https://arxiv.org/abs/ 2502.09969
- [23]
-
[24]
MiniGPT-4: Enhancing vision-language understanding with advanced large language mod- els,
D. Zhu et al., “MiniGPT-4: Enhancing vision-language understanding with advanced large language mod- els,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https : / / openreview . net / forum ? id = 1tZbq88f27
2024
-
[25]
Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning,
P. Sharma et al., “Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning,” in Proceedings of ACL, 2018
2018
- [26]
-
[27]
Alpagasus: Training a better alpaca with fewer data,
L. Chen et al. , “Alpagasus: Training a better alpaca with fewer data,” in The Twelfth International Con- ference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum? id=FdVXgSJhvz
2024
-
[28]
Language models are unsupervised multitask learners,
A. Radford et al., “Language models are unsupervised multitask learners,”OpenAI, 2019, Accessed: 2024-11-
2019
-
[29]
A frustratingly simple approach for end-to-end image captioning,
Z. Luo et al. , “A frustratingly simple approach for end-to-end image captioning,” 2022. [Online]. Avail- able: https://api.semanticscholar.org/ CorpusID:248178267
2022
-
[30]
Learning transferable visual mod- els from natural language supervision,
A. Radford et al., “Learning transferable visual mod- els from natural language supervision,” in Proceed- ings of the 38th International Conference on Ma- chine Learning, ICML 2021, 18-24 July 2021, Vir- tual Event , M. Meila and T. Zhang, Eds., ser. Pro- ceedings of Machine L...
2021
-
[31]
Dosovitskiy et al., An image is worth 16x16 words: Transformers for image recognition at scale , 2020
A. Dosovitskiy et al., An image is worth 16x16 words: Transformers for image recognition at scale , 2020. DOI: 10.48550/ARXIV.2010.11929 . [Online]. Available: https : / / arxiv . org / abs / 2010 . 11929
-
[33]
Available: https://cdn.openai
[Online]. Available: https://cdn.openai. com/better-language-models/language_ models _ are _ unsupervised _ multitask _ learners.pdf
-
[35]
A survey on enhancing image caption- ing with advanced strategies and techniques,
B. Zou et al., “A survey on enhancing image caption- ing with advanced strategies and techniques,” Com- puter Modeling in Engineering & amp; Sciences , vol. 142, no. 3, pp. 2247–2280, 2025, ISSN : 1526-
2025
-
[37]
SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,
P. Virtanen et al., “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Meth- ods, vol. 17, pp. 261–272, 2020. DOI: 10 . 1038 / s41592-019-0686-2
2020
-
[1506]
32604 / cmes
DOI: 10 . 32604 / cmes . 2025 . 059192. [Online]. Available: http://dx.doi.org/10. 32604/cmes.2025.059192
2025
- [2024]
-
[2025]
48550 / ARXIV
DOI: 10 . 48550 / ARXIV . 2502 . 01341. [Online]. Available: https://arxiv.org/abs/ 2502.01341
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.