{"id":"ba6e42a7-3422-4986-8aa6-a9fdd1098e89","arxiv_id":"2507.20156","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A compact VLM fine-tuned on Gemini-generated quality scores can filter image-caption pairs, yielding data with higher CLIP alignment, lower perplexity, and higher Gemini-judged captioning preference.","lead":"The authors fine-tuned a small vision-language model to score how well images match their captions, using a larger AI model's ratings as training labels. They found that filtering web-scraped image-text data with this small judge improved caption quality scores and a captioning model's outputs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Teacher/judge circularity: Gemini 2.0-Flash supplies both the filter's training labels (Sec. 3.1) and the 59.4% caption-preference evaluation (Sec. 4.3), so the headline result may reflect Gemini's stylistic preferences rather than genuine data-quality gains.","rationale":"Reader's weakest assumption is exactly the concern I consider load-bearing. The paper contains non-circular support: the t-test on CLIP similarity and the perplexity differences show the filter changes the distribution, and Table 2 shows monotonic alignment by score bucket. However, those analyses do not validate the central performance claim; they only show the filtered set is different in ways measurable by CLIP and perplexity. The only direct evidence for 'on par or better' is the Gemini judge result. Because Gemini is also the teacher that defines what the filter optimizes, the evaluation is confounded: a filtered model is more likely to produce captions the teacher likes. This is distinct from a general concern about LLM judges; the specific issue is that the same model family defines the optimization target and the evaluation metric. A blind independent judge (human or different model family) would settle it. If the preference is confirmed, the claim is credible despite the circular setup; if not, the paper needs substantial revision. I therefore do not move the reader's verdict: unconditional rejection would be too harsh given the addressable nature of the concern and the non-circular auxiliary analyses, but the current conditional framing is appropriate.","tokens_in":7408,"tokens_out":4368,"duration_ms":47834,"concrete_test":"Conduct a blind head-to-head preference study on the same 500 test pairs with an independent oracle: human raters (e.g., 3 annotators per pair) or a different model family (e.g., GPT-4o or Claude) that was not involved in label generation, with generated captions presented in randomized order and no model identity revealed. Compute the preference rate for the filtered-model captions with confidence intervals. If the independent rate is within noise of 50% or is reversed, the 59.4% result is an artifact of Gemini self-preference and the central claim fails without revision. As a complementary check, evaluate the same two ViT-GPT2 models on standard reference-based captioning metrics (CIDEr, BLEU-4, METEOR) and on a benchmark such as NoCaps; a discrepancy between Gemini preference and these metrics would confirm the bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that an 18% filtered subset performs on par with or better than the full 20K set—rests on the validity of Gemini 2.0-Flash's quality assessments. Section 3.1 uses Gemini to score the 5K training pairs that supervise the compact filter; Section 3.4 and Section 4.3 use Gemini 2.0-Flash to judge which of two captioning models better matches a reference caption. If Gemini's scores encode any systematic preference (e.g., for longer, Recap-COCO-style recaptioned sentences, for certain lexical patterns, or for captions similar to its own outputs), then the filter is trained to select according to that preference and the evaluator rewards the same preference. The 59.4% preference is then a measure of agreement with Gemini's taste, not of human-relevant image-text alignment. The non-circular evidence (CLIP cosine similarity, perplexity, Table 2) is suggestive but does not establish downstream superiority: CLIP similarity is a coarse proxy, and lower perplexity can merely indicate more generic or shorter captions. The manual review in Section 3.1 is asserted without protocol or inter-annotator agreement, so it does not break the circularity. Unless Gemini's scores are independently validated as ground truth for image-caption alignment, the headline result is confounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a compact VLM-based data filtration framework for image-text pairs. A Qwen2-VL-2B model is fine-tuned on 5,000 image-caption pairs annotated by Gemini 2.0-Flash with 1-10 quality scores and rationales. The resulting scorer is used to filter 20K pairs from CC3M and CC12M down to 3.5K pairs (18%) using a threshold of 9. The authors compare filtered, full, and random subsets using CLIP cosine similarity and caption perplexity, and they evaluate downstream captioning by fine-tuning identical ViT-GPT2 models on full versus filtered data, with Gemini 2.0-Flash as the judge, reporting a 59.4% preference rate for the filtered model. The paper claims that datasets filtered with this compact VLM perform on par with or surpass larger and noisier web-crawled datasets.","tokens_in":7815,"tokens_out":4963,"duration_ms":50024,"significance":"If the central claims hold, the work provides a practical, low-cost, on-prem data curation tool for VLM pre-training, and the released model, data, and scripts are a useful community resource. The idea of distilling a large teacher model's quality judgments into a compact scoring VLM is sensible, and the authors do report statistical testing and a bucket-level ablation in Section 5. However, the significance of the contribution is currently limited because the headline downstream result relies on the same model family as both teacher and judge, and because the abstract's claim about outperforming larger web-crawled datasets is not directly tested in the experiments.","major_comments":[{"comment":"The teacher model Gemini 2.0-Flash generates the training labels for the filter (Section 3.1) and also serves as the judge for the headline captioning result (Sections 3.4 and 4.3), so the 59.4% preference rate may partly reflect Gemini's stylistic preferences rather than human-relevant image-text alignment. The manual review mentioned in Section 3.1 is asserted without a protocol, sample size, or inter-annotator agreement, so it does not establish that Gemini's scores are a valid ground-truth proxy. I recommend replacing or supplementing the LLM-as-a-judge evaluation with human preference judgments, a judge from a different model family, and standard captioning metrics such as CIDEr, BLEU, and METEOR.","section":"Sections 3.1 and 3.4/4.3"},{"comment":"The abstract claims that filtered datasets 'perform on par with, or even surpass, larger and noisier datasets gathered through high-volume web crawling,' but the experiments only compare a 3.5K filtered subset against the same 20K full set and a 3.5K random subset. No training run or evaluation against an actual larger web-scale dataset is reported, so this central claim is not supported by the evidence in the manuscript.","section":"Abstract and Section 3.3"},{"comment":"The lower perplexity of the filtered captions is interpreted as evidence of higher linguistic quality, but perplexity is strongly affected by caption length and lexical predictability, and shorter or more generic captions may achieve lower perplexity without being better. The paper does not report caption length distributions, lexical diversity, or n-gram statistics for the three splits, so the conclusion that filtering improves 'linguistic fluency' is not yet established. I suggest length-matched controls and additional diversity metrics.","section":"Section 4.2"},{"comment":"The 59.4% preference rate is reported as 297/500 without a confidence interval or significance test, and the judge prompt is not provided, so the robustness of the result to prompt variation cannot be assessed. Standard captioning metrics are absent, leaving the magnitude of downstream improvement relative to common benchmarks unknown. A binomial confidence interval and results from at least one non-Gemini judge or human evaluation are needed before this can be taken as evidence of genuine quality gains.","section":"Section 4.3"},{"comment":"The CLIP cosine-similarity gains are statistically significant but small in absolute terms: the mean increases from 0.298 to 0.313 in Section 4.1, and the bucket averages in Table 2 range only from 0.27 to 0.32. The paper does not establish that differences of this size are practically meaningful for downstream VLM training. Reporting an effect size such as Cohen's d and linking filtration scores to downstream task performance would strengthen the interpretation.","section":"Sections 4.1 and 5, Table 2"}],"minor_comments":[{"comment":"There are grammatical and stylistic errors, including 'significantly broadens' in the abstract and 'Our approach is cost- and resource-effective than previous approaches' in Section 1; these should be corrected.","section":"Section 1"},{"comment":"The training details are incomplete: the optimizer, warmup schedule, and whether the model was fully fine-tuned or LoRA-adapted are not specified, and no evaluation of the fine-tuned model's score distribution is provided.","section":"Section 3.2"},{"comment":"The filtration threshold of 9 is a free parameter, and no sensitivity analysis is reported. The paper should show how the downstream metrics vary with thresholds such as 8, 9, and 10.","section":"Section 3.3"},{"comment":"The composition of the 500-pair test set is not described: it is unclear where the images and ground-truth captions come from and whether they overlap with the training data of the captioning model.","section":"Section 3.4"},{"comment":"Reference [1] is cited for Gemini 2.0-Flash, but the reference describes Gemini 1.5; the correct model or version should be cited. Also, Table 1 states 2.5K selected samples per dataset while the text says 5,000 total; please clarify the exact counts.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern raised in the reader's report is well-founded and should be the central focus of revision. The manuscript fits the workshop's applied ML scope, but the gap between the abstract's claims and the experimental evidence is significant; I would not recommend acceptance without an independent or human evaluation of the captioning results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this paper is a clean, honest write-up of a compact VLM data-filtering recipe, but its headline number (59.4% caption preference) is confounded because Gemini 2.0-Flash supplies both the filter's training labels and the evaluation judge. The stress-test note lands. Section 3.1 has Gemini score the 5K training pairs; Section 3.4 and 4.3 have Gemini pick the better caption. If Gemini has taste preferences, both the filter and the judge share them, so the 59.4% partly measures self-agreement. That is a real flaw, not a manufactured one.\n\nWhat's genuinely good: they ship the fine-tuned 2B model, training data, scripts, and appendices. They compare against a same-size random subset, which is the right control at this scale. They report a t-test on the CLIP cosine similarity gain (p ≈ 4e-56), and the ablation in Table 2 shows a monotonic relation between their filter's score buckets and CLIP alignment. That is credible evidence the filter is doing something.\n\nThe soft spots, in proportion: the circularity is the big one, and it's not addressed. The manual review mentioned in Sec. 3.1 has no protocol or agreement numbers, so it doesn't break the loop. The effect sizes are modest—CLIP similarity goes from 0.297 to 0.313, and perplexity drops from ~170 to ~137—suggestive but not dramatic. The downstream captioning evaluation uses only the Gemini judge; there is no CIDEr, BLEU, or human eval, and no comparison to existing filters like Self-Filter (Ref [19]) or InstructionGPT-4. The novelty is incremental: the idea that a VLM itself can be a filter is already in the cited Self-Filter paper; the delta here is compact size and distilled supervision from a proprietary teacher. The abstract overclaims scale benefits that the 20K-pair test can't support.\n\nWho is this for? Someone building cheap pre-training curation pipelines would find the released model and recipe useful as a starting point. A researcher working on data quality would read it as a workshop-level contribution with a clear validation gap.\n\nRecommendation: it deserves a serious referee, not a desk reject. The artifacts are real and the method is plausible. But the paper needs major revision—an independent judge (human or different model family), standard captioning metrics, and a comparison against at least one existing filter. I would not cite it yet in my own work until the circularity is resolved.","headline":"A plausible compact-filter recipe whose headline result is undermined by using Gemini as both teacher and judge; the released artifacts make it worth a serious look, but the central claim needs an independent evaluation to hold.","tokens_in":8262,"tokens_out":1170,"would_cite":false,"duration_ms":13738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A compact vision-language model fine-tuned on teacher-generated quality scores can filter an image-text corpus down to 18%, and a captioning model trained on that subset wins 59.4% of judge comparisons against the full-data model.","keywords":["vision-language models","data filtration","training corpora","image-text alignment","model-based data selection","LLM-as-a-judge","supervised fine-tuning","captioning"],"falsifier":"Train the same compact scorer on labels from a different strong teacher, or replace the Gemini evaluation judge with human raters on the same 500 test images; if human preference for the filtered-model captions does not come out near 59.4%—or if the CLIP alignment gain vanishes under the re-trained filter—the reported advantage is an artifact of the teacher's own scoring bias rather than a property of filtered data.","tokens_in":7217,"feed_emoji":"🧹","tokens_out":6763,"duration_ms":62053,"temperature":0.7,"pith_summary":"The paper attempts to show that a small, purpose-built vision-language model—not a giant model or separate filter modules—can judge whether an image-caption pair is aligned well enough for training data. Fine-tuned on scores and rationales produced by a stronger teacher model, this compact judge assigns each pair a score from 1 to 10. Keeping only pairs scored 9 or higher leaves 18% of the original corpus, and captioning models trained on that filtered subset produce captions that the teacher-judge prefers over those from the full dataset in 59.4% of comparisons. The result matters because it suggests high-quality training data can be produced cheaply and on-premises, without expensive API calls, and that quality filtering can rival simply collecting more web-scale data. The paper nonetheless frames the outcome as a tradeoff, since the full-data model still wins more than 40% of comparisons, indicating some informative diversity is lost.","feed_headline":"A tiny VLM judge filters data so well it beats the full set","feed_subtitle":"Filtered to 18% of the corpus, the small model's captions were preferred in 59.4% of head-to-head judge comparisons.","key_machinery":"The central mechanism is a two-stage score-distillation pipeline: a strong teacher model (Gemini 2.0-Flash) annotates image-caption pairs with continuous quality scores and explanatory rationales, and supervised fine-tuning transfers that judgment into a 2-billion-parameter compact VLM that can score new pairs without external modules or API calls. The resulting scalar score acts as a high-precision filter, with a retention threshold of 9 or above, and its quality is validated through CLIP cosine-similarity alignment, caption perplexity, and an LLM-as-a-judge downstream comparison.","core_discovery":"The central claim is that a compact vision-language model (Qwen2-VL-2B) can be fine-tuned to reproduce the quality-judgment behavior of a stronger teacher model and thereby act as a standalone scoring function for image-text pairs. The filter is trained on 5,000 balanced samples from Recap-COCO and CC12M, each annotated with a 1–10 score and a textual rationale; after supervised fine-tuning, it scores new pairs and the authors retain only those scoring at least 9. On a 20K-pair evaluation set this preserves 3.5K pairs, or 18% of the data. Compared with the full set and a same-size random subset, the filtered set shows higher CLIP-based image-text cosine similarity (0.313 versus 0.298 and 0.297, with a two-sample $t$-test $p = 4.27 \\times 10^{-56}$) and lower caption perplexity (137.2 versus 170.2 and 168.6). In the downstream test, two identical ViT-GPT2 captioning models are trained on filtered versus full data, and a Gemini judge prefers the filtered model's captions for 297 of 500 test images, i.e. 59.4% of cases.","pith_inferences":["Because the teacher's scores are both the training signal and the evaluation signal, the 59.4% preference result is best read as an upper bound on the filter's true benefit; an independent human or a differently trained judge could plausibly give a lower, though probably still positive, preference rate.","The monotone increase in CLIP alignment across score buckets in the ablation suggests the scorer carries calibrated signal well beyond the binary 9-or-10 threshold, so a natural extension is to sweep the threshold and map the quality-diversity frontier rather than assume 18% is optimal.","The same distillation recipe could be transferred to other domains: any strong teacher's scoring behavior could be compressed into a compact model, turning expensive API-based data auditing into a cheap on-premises screening step for other large multimodal corpora."],"forward_implications":["Filtering a web-scale image-text corpus with the compact scorer can cut training data to roughly 18% while keeping or improving downstream captioning quality, meaning the same caption quality can be reached with far less data and compute.","Because the filter runs on a 2B-parameter model with no auxiliary scoring modules and no API calls, it can be deployed on-premises to screen large datasets before they enter a VLM pre-training pipeline.","Lower caption perplexity and higher CLIP cosine similarity on the filtered set imply that the retained pairs are both more fluent and more semantically aligned, so models trained on them should generalize better and hallucinate less.","The persistence of a 40.6% preference for the full-data model exposes a diversity cost: aggressive filtering removes some informative variation, so the optimal retention rate is not necessarily the 18% threshold studied here."],"supporting_citations":[{"why":"Supplies the teacher-model scores and rationales that supervise the compact filter, and later the judge model behind the 59.4% preference rate.","marker":"[1]"},{"why":"Provides the compact Qwen2-VL base model that is fine-tuned into the standalone scoring function.","marker":"[2]"},{"why":"Contributes web-scale noisy caption pairs used for training the filter and for the held-out 20K-pair evaluation.","marker":"[10]"},{"why":"Supplies the high-quality re-captioned pairs used to balance the filter's training set.","marker":"[11]"},{"why":"Provides the Conceptual Captions source for 10K of the held-out image-caption pairs filtered in the evaluation.","marker":"[25]"},{"why":"Gives the CLIP embedding space used to compute cosine-similarity alignment for both the main result and the score-bucket ablation.","marker":"[26]"},{"why":"Implements the two-sample t-test that reports the p-value for the filtered set's alignment gain.","marker":"[31]"}],"fun_headline_variants":["Compact VLM judge beats full dataset with 18% of data","Tiny VLM as data judge: 18% data outperforms full set","Small VLM filters to 18%, beats larger noisier dataset","A 2B VLM judge trims data to 18% and wins","Compact VLM judge: 18% data wins 59.4% of comparisons"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Gemini 2.0-Flash's 1–10 quality scores are a fair measure of true image-text alignment, yet the same model family is also the judge in the headline 59.4% comparison, so any bias in the teacher would be baked into both the filter's training signal and the evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Compact VLM judge beats full dataset with 18% of data","Tiny VLM as data judge: 18% data outperforms full set","Small VLM filters to 18%, beats larger noisier dataset","A 2B VLM judge trims data to 18% and wins","Compact VLM judge: 18% data wins 59.4% of comparisons"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2968,"prompt_tokens":1092,"completion_tokens":1876,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":708,"completion_tokens_details":{"reasoning_tokens":1774}},"tokens_in":708,"tokens_out":1876,"duration_ms":14755,"temperature":1.0,"reasoning_tokens":1774,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:47:35.814398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same compact scorer on labels from a different strong teacher, or replace the Gemini evaluation judge with human raters on the same 500 test images; if human preference for the filtered-model captions does not come out near 59.4%—or if the CLIP alignment gain vanishes under the re-trained filter—the reported advantage is an artifact of the teacher's own scoring bias rather than a property of filtered data.","supporting_citations":[{"cited_title":"Wang et al., Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution ,","cited_arxiv_id":null,"evidence_quote":"Provides the compact Qwen2-VL base model that is fine-tuned into the standalone scoring function."},{"cited_title":"Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts,","cited_arxiv_id":null,"evidence_quote":"Contributes web-scale noisy caption pairs used for training the filter and for the held-out 20K-pair evaluation."},{"cited_title":"Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning,","cited_arxiv_id":null,"evidence_quote":"Provides the Conceptual Captions source for 10K of the held-out image-caption pairs filtered in the evaluation."}],"review_version":1}