{"id":"d4ac561b-0eee-49eb-9139-a6126f473230","arxiv_id":"2506.05673","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning LLaVA-NeXT and BLIP2 on the DataSeeds.AI sample dataset improves caption similarity to its own annotations, but only on that dataset's validation split.","lead":"This paper introduces DSD, a dataset of about 10,600 peer-ranked photos with layered human annotations, and fine-tunes two vision-language models on it. The reported gains are measured only on a held-out slice of the same dataset, so they do not yet show general improvement on independent benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of improvements 'against known benchmarks' rests entirely on metrics computed on the DSD validation split used for checkpoint selection, so the claim is not empirically supported as stated.","rationale":"The reader's weakest assumption correctly identifies the failure mode: the evaluation assumes that metrics on the DSD validation split, which was also used for checkpoint selection, are a valid proxy for 'known benchmarks.' My independent reading of Sections 4.3 and 4.5 confirms that every reported fine-tuning gain is computed on this same validation split, with no external test set, no held-out DSD test split, and no comparison against an alternative training corpus. The BLIP2 results also show a large BERTScore decline and a CLIPScore decline, so the evidence is internally mixed. The central claim is therefore unsupported. I find no reason to adjust the reader's REJECT verdict, though the paper does offer a potentially useful dataset and a clear description of its annotation pipeline. The proposed concrete test directly settles whether the claimed benchmark improvements exist outside the validation split used for model selection.","tokens_in":13630,"tokens_out":1653,"duration_ms":17675,"concrete_test":"Run the fine-tuned and base LLaVA-NeXT and BLIP2 checkpoints on COCO Karpathy test, NoCaps validation, and Flickr30k test, computing BLEU-4, ROUGE-L, BERTScore, and CLIPScore with identical decoding settings. Also fine-tune each model on a randomly sampled 9,549-pair subset of COCO train and evaluate on the same external benchmarks. If DSD fine-tuning fails to beat both the base model and the COCO-subset fine-tuned model on external splits, the 'known benchmarks' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract promises 'quantitative improvements generated by the DSD on specific models against known benchmarks,' but Sections 4.5.2 and 4.5.3 report all fine-tuning gains on the DSD validation split (1,061 image-caption pairs), which is the same split used for checkpoint selection in Sections 4.3.1 and 4.3.2 (lowest validation loss for LLaVA-NeXT; peak CIDEr+Bleu-4 for BLIP2). Reporting final metrics on the split that determined the model choice introduces selection bias and does not constitute evaluation on a 'known benchmark.' No external test set (e.g., COCO Karpathy test, NoCaps, Flickr30k) is used, no comparison to fine-tuning on an equally sized generic caption dataset is provided, and no variance or confidence intervals are reported. Additionally, BLIP2 degrades on BERTScore and CLIPScore after fine-tuning, so even the internal evidence does not support a blanket claim of improvement. The dataset may be useful, but the paper's central quantitative claim is not established by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the DataSeeds.AI Sample Dataset (DSD), approximately 10,610 human peer-ranked photographs with multi-tier human annotations, semantic segmentation masks, and EXIF metadata. The authors report an evaluation of AWS Rekognition label detection against human annotations, then fine-tune two vision-language models (LLaVA-NeXT and BLIP2) on DSD caption pairs and compare base versus fine-tuned models with BLEU-4, ROUGE-L, BERTScore, and CLIPScore. The abstract claims that the paper documents quantitative improvements 'against known benchmarks' and releases code and trained models.","tokens_in":13936,"tokens_out":7433,"duration_ms":66987,"significance":"If the quantitative claims were valid, a licensed, human-annotated, peer-ranked dataset with segmentation and metadata would be a useful contribution to data-centric vision-language research, and the public release of code and models is commendable. However, the evaluation as presented does not establish the paper's central claim: the fine-tuning metrics are computed on the same DSD validation split used for checkpoint selection, no external benchmark or independent test set is used, and the BLIP2 results show degradation on semantic metrics. The dataset may still have value, but the evidence for 'improvements against known benchmarks' is not supportable from the experiments reported.","major_comments":[{"comment":"The final metrics are computed on the DSD validation split that was also used to select checkpoints: Section 4.3.1 chooses the LLaVA-NeXT checkpoint with lowest validation loss, and Section 4.3.2 chooses the BLIP2 checkpoint with peak CIDEr+Bleu-4 on the validation set. Reporting improvements on the same split after model selection introduces selection bias and measures fit to the validation distribution, not generalization to unseen data. An independent test split or external benchmarks such as COCO Karpathy test, NoCaps, or Flickr30k are needed to support the abstract's claim of improvements 'against known benchmarks'.","section":"Sections 4.3.1, 4.3.2, 4.5.2, 4.5.3"},{"comment":"The abstract and the experiment design state that the paper documents 'quantitative improvements generated by the DSD on specific models against known benchmarks,' but no known external benchmark is used anywhere in the evaluation. The only comparisons in Tables 3 and 4 are between base and fine-tuned models on the DSD validation split. This is a mismatch between the central claim and the evidence presented.","section":"Abstract and Section 4.1"},{"comment":"The BLIP2 results show substantial degradation after fine-tuning: BERTScore drops from 0.0545 to -0.0537 (a -198.53% relative change) and CLIPScore drops from 0.2854 to 0.2583 (-9.49%). Section 4.5.5 nevertheless says the fine-tuned model achieved 'substantial improvements across all evaluation metrics,' and the conclusion claims 'compelling evidence' that the DSD 'significantly enhance[s] the performance and reliability of modern vision-language models.' These statements are directly contradicted by the paper's own Table 4.","section":"Table 4 and Section 4.5.3"},{"comment":"The experiments were run on 10,610 images, but the released dataset contains only 7,772 images after removal of sensitive content, and the training/validation split was created from the original 10,610 images. The exact experimental data is therefore not available to readers, and the released 7,772-image subset cannot reproduce the reported training and evaluation. The claim that code and trained models are publicly available does not address this reproducibility gap.","section":"Footnotes 1 and 2, Section 4.2"}],"minor_comments":[{"comment":"There are several typographical errors: 'followed byportraits' is missing a space, 'See' in the top-20 label list is likely a typo for 'Sea,' and the model name is rendered inconsistently as 'LLaVV A-NeXT,' 'LLaV A-NeXT,' and 'LLaVA-NeXT' across the text.","section":"Sections 3.2.1 and 3.2.2"},{"comment":"The claim that the annotation framework was 'independently developed prior to the publication and our awareness of' NVIDIA's DAM framework is unverifiable and irrelevant to the technical content; it should be removed or substantiated with dated evidence.","section":"Section 3"},{"comment":"No confidence intervals, standard errors, or multiple-seed results are reported for any of the fine-tuning metrics, despite the validation set containing only 1,061 pairs; the differences in Tables 3 and 4 may not be statistically distinguishable.","section":"Section 4.5"},{"comment":"The Rekognition comparison uses exact label matching at a fixed 50% confidence threshold without sensitivity analysis or a human-agreement baseline, so the reported precision/recall/F1 values are hard to interpret as a measure of 'semantic gap'; some discussion of threshold dependence would strengthen this section.","section":"Sections 4.4.1 and Table 2"}],"recommendation":"reject","confidential_remarks":"The paper sits in a gray zone between major revision and rejection. The main reason for rejection is not the dataset itself but the evaluation design: the central claim of benchmark improvements is unsupported because all fine-tuning metrics come from the validation split used for checkpoint selection, and no external benchmark appears. The additional mismatch between the 10,610-image experimental dataset and the 7,772-image released subset makes the reported results irreproducible from the public artifact. A resubmission with an independent test split or external benchmarks, a clear mapping between released images and experimental splits, and a toned-down abstract could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The DSD is a real new resource: peer-ranked photography with three-tier human annotations, segmentation masks, and released code and fine-tuned models. The paper is also honest in places, especially about BLIP2's semantic degradation. But the central claim of \"quantitative improvements ... against known benchmarks\" is not supported: every fine-tuning metric is computed on the DSD validation split, the same split used for checkpoint selection, and no external benchmark (COCO Karpathy test, NoCaps, Flickr30k) appears anywhere. The Rekognition comparison is under-specified too, with a single confidence threshold and no error bars.\n\nWhat actually works: the peer-ranking signal is genuinely uncommon and the multi-tier annotation structure is thoughtful. The qualitative example (Sony WH-1000XM3) shows real gains in technical accuracy and compositional awareness. The authors also openly disclose that BLIP2 loses BERTScore and CLIPScore, and they explain the \"4600% BLEU improvement\" as a near-zero baseline artifact. That is the kind of candor you do not always see.\n\nThe soft spot is load-bearing. The abstract promises improvements against known benchmarks, but the tables report improvements relative to the DSD's own annotations on the validation split. Checkpoint selection on that same split inflates the result, and even then the LLaVA gains are tiny (BLEU-4 from 0.0199 to 0.0246, CLIPScore +0.0013). Without a held-out external set or a comparison to fine-tuning on an equally sized generic caption dataset, the quantitative evidence cannot carry the claim. A minor point: the abstract says ~10,610 images, while the released dataset has 7,772 after face removal. That should be stated clearly up front.\n\nWho is this for? Researchers working on data-centric AI or commercial image datasets will find the DSD itself worth a look. The paper's evaluation section is a useful cautionary example of in-domain validation being mistaken for external benchmarking. As a scientific claim about model improvement, it needs either external evaluation or a substantially resealed conclusion.\n\nIt deserves a serious referee because the dataset release and the shipped code/models are real contributions. The conversation about what counts as a benchmark is worth having, and the authors have shown they can engage with criticism. Send it out, but expect the referees to demand external benchmarks or a modest, honest claim.\n\nBottom line: read the data release, skip the \"improvements against known benchmarks\" wording.","headline":"Useful dataset resource, but the fine-tuning evaluation is in-domain only, so the 'known benchmarks' claim does not hold as stated.","tokens_in":14337,"tokens_out":1916,"would_cite":false,"duration_ms":21345,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on a 10,610-image human-annotated, peer-ranked dataset shifts vision-language models toward human-style scene descriptions.","keywords":["DataSeeds DSD","data-centric AI","vision-language models","fine-tuning","human annotation","peer-ranked photography","semantic segmentation","scene description"],"falsifier":"Take the released fine-tuned checkpoints and run them on an independent image-captioning test set whose human references were not written by DataSeeds annotators, comparing BLEU-4, ROUGE-L, BERTScore, and CLIPScore against the base models. If the fine-tuned models do not beat the base models on that external set, the paper's central claim that DSD fine-tuning improves scene-description performance is falsified.","tokens_in":13416,"feed_emoji":"📷","tokens_out":8735,"duration_ms":76276,"temperature":0.7,"pith_summary":"This paper introduces the DataSeeds Sample Dataset (DSD): roughly 10,610 human peer-ranked photographs, each carrying three tiers of handwritten English captions and a full semantic segmentation mask. The goal is to show that a small, carefully annotated dataset can act as a data-centric lever, improving how vision-language models describe scenes without changing their architecture. The paper documents two kinds of evidence: AWS Rekognition captures only a fraction of the human label vocabulary on the same images, and fine-tuning LLaVA-NeXT and BLIP2 on the DSD changes their captioning behavior. On the dataset's own validation split, LLaVA-NeXT improves across BLEU-4, ROUGE-L, BERTScore, and CLIPScore, while BLIP2 gains lexical overlap but loses semantic alignment. If the data-centric claim holds, then high-quality human annotation, not just model scaling, is a practical route toward more precise scene understanding.","feed_headline":"Peer-ranked photos lift vision-model scene descriptions","feed_subtitle":"Fine-tuning on the DataSeeds DSD improves LLaVA-NeXT's captioning metrics and exposes a human-vs-machine labeling gap.","key_machinery":"The load-bearing object is the DSD annotation package: for each image, a short title, a 15+ word narrative description, a 20-30 word technical scene analysis, and a pixel-level semantic segmentation mask, all produced by human annotators and attached to photographs that carry an organic peer-ranking signal from a gamified photography community. The machinery is supervised fine-tuning: each image is paired with a prompt asking for a technical scene analysis, and the human annotation is the target output. LLaVA-NeXT is adapted with LoRA on top of a 0.5B Qwen2 language model with a SigLIP vision encoder, while BLIP2 is fine-tuned end-to-end with the Vision Transformer unfrozen; both are evaluated against the held-out human annotations using BLEU-4, ROUGE-L, BERTScore, and Long-CLIP CLIPScore.","core_discovery":"The paper sets out to establish that a human peer-ranked, multi-tier-annotated image dataset is a sufficient lever for improving scene-description performance in vision-language models. Concretely, it reports that after LoRA fine-tuning of LLaVA-NeXT on 9,549 image-description pairs, BLEU-4 rose from 0.0199 to 0.0246, a 24.09% relative gain, while ROUGE-L, BERTScore, and CLIPScore also rose by smaller amounts. For BLIP2, BLEU-4 rose from 0.001 to 0.047 and ROUGE-L from 0.126 to 0.242, while BERTScore fell from 0.0545 to -0.0537 and CLIPScore from 0.2854 to 0.2583. The paper reads these contrasting results as evidence that data quality matters, but architecture determines how the data's value is realized.","pith_inferences":["The paper does not evaluate on an independently annotated benchmark; running the released checkpoints on one would settle whether the gains generalize beyond the DSD validation split.","The peer-ranking scores could be repurposed as a reward signal for preference-based tuning of image-generation or captioning models; the paper flags this direction but does not train such a model.","BLIP2's degraded BERTScore and CLIPScore after fine-tuning suggests that training only on technical scene descriptions may overfit annotation style; mixing generic captions into the fine-tuning set is a plausible fix the paper does not test.","Because the dataset includes EXIF fields such as camera, lens, focal length, and GPS, a natural extension is camera-aware or geography-aware captioning and generation, which the paper leaves unexplored."],"forward_implications":["A relatively small fine-tuning set of about 10,000 images can measurably change a vision-language model's descriptive output toward human technical style, at least on data drawn from the same distribution.","Commercial label APIs should not be trusted as training-grade ground truth: Rekognition's 0.19 F1 and 18.96% label overlap with human annotations suggest uncritical use of such tags would inject noise.","Lexical and semantic metrics can move in opposite directions after fine-tuning, so evaluations that rely on a single metric can misrepresent whether a model became better or worse.","The DSD's EXIF metadata and peer-ranking scores make additional supervision channels available for future models, even though this paper only uses the captions for fine-tuning.","If the pipeline scales to the larger licensed catalog, custom data-centric fine-tuning becomes feasible for commercial applications without relying on noisy public labels."],"supporting_citations":[{"why":"Supplies the data-centric framing and the two hypotheses about semantic gaps and data-driven gains that the experiments test.","marker":"Bhatt et al. [2024]"},{"why":"Provides the LLaVA-NeXT architecture and pretrained model family used for the first fine-tuning arm.","marker":"Sun et al. [2024]"},{"why":"Provides the BLIP2 architecture and pretrained weights that form the second fine-tuning arm.","marker":"Li et al. [2023]"},{"why":"Supplies the LoRA low-rank adaptation method that defines which parameters are trainable in the LLaVA-NeXT run.","marker":"Hu et al. [2022]"},{"why":"Supplies the specific pretrained LLaVA-OneVision-Qwen2-0.5B checkpoint that is then adapted with LoRA.","marker":"LMMS Lab [2024]"},{"why":"Supplies the SigLIP vision encoder used in the LLaVA-NeXT fine-tuning configuration.","marker":"Zhai et al. [2023]"},{"why":"Defines BLEU-4, the primary lexical precision metric used to compare generated and reference descriptions.","marker":"Papineni et al. [2002]"},{"why":"Defines ROUGE-L, the structural recall metric used for sequence overlap.","marker":"Lin [2004]"},{"why":"Defines BERTScore, the contextual-embedding semantic similarity metric whose decline for BLIP2 is a central result.","marker":"Zhang et al. [2019]"},{"why":"Provides Long-CLIP, used to compute CLIPScore on longer generated descriptions.","marker":"Zhang et al. [2024]"}],"fun_headline_variants":["Peer-ranked dataset boosts LLaVA-NeXT captions, BLIP2 mixed","Human-ranked photo data improves vision captioning, with trade-offs","DataSeeds' peer-ranked set lifts LLaVA-NeXT, splits BLIP2","Peer-ranked image dataset: fine-tuning wins for LLaVA, mixed for BLIP2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire quantitative case rests on the assumption that high scores on the DSD's own validation set, after choosing the checkpoint with lowest validation loss, measure a model's general ability to describe scenes well; if that assumption fails, the reported gains show only that the model learned to imitate the DSD annotation style.","fun_headline_variants_meta":{"raw":{"variants":["Peer-ranked dataset boosts LLaVA-NeXT captions, BLIP2 mixed","Human-ranked photo data improves vision captioning, with trade-offs","DataSeeds' peer-ranked set lifts LLaVA-NeXT, splits BLIP2","Peer-ranked image dataset: fine-tuning wins for LLaVA, mixed for BLIP2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000747,"raw_usage":{"total_tokens":3342,"prompt_tokens":974,"completion_tokens":2368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":2279}},"tokens_in":590,"tokens_out":2368,"duration_ms":20512,"temperature":1.0,"reasoning_tokens":2279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:12:51.760674+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released fine-tuned checkpoints and run them on an independent image-captioning test set whose human references were not written by DataSeeds annotators, comparing BLEU-4, ROUGE-L, BERTScore, and CLIPScore against the base models. If the fine-tuned models do not beat the base models on that external set, the paper's central claim that DSD fine-tuning improves scene-description performance is falsified.","supporting_citations":[{"cited_title":"A data-centric approach to improve performance of deep learning models","cited_arxiv_id":null,"evidence_quote":"Supplies the data-centric framing and the two hypotheses about semantic gaps and data-driven gains that the experiments test."},{"cited_title":"Introducing LLaVA - NeXT , 2024","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-NeXT architecture and pretrained model family used for the first fine-tuning arm."},{"cited_title":"Llava-onevision-qwen2-0.5b-ov","cited_arxiv_id":null,"evidence_quote":"Supplies the specific pretrained LLaVA-OneVision-Qwen2-0.5B checkpoint that is then adapted with LoRA."}],"review_version":1}