{"id":"023c2b32-9b3a-4c84-a3ed-467b5bb4d291","arxiv_id":"2505.13788","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Ground-V is a 500K-sample dataset of instruction-segmentation pairs that improves VLM grounding on complex instructions and claims new state-of-the-art results on referring segmentation benchmarks.","lead":"This paper presents a workflow that uses a teacher vision-language model to generate 500,000 instruction-segmentation pairs for visual grounding, covering five challenge types. The resulting dataset, Ground-V, is shown to improve two existing segmentation models when added to their training data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-V is built from COCO 2017 images while RefCOCO/+/g and gRefCOCO use COCO 2014 images; COCO 2017 train contains most 2014 images, so train/eval image overlap is likely, and no deduplication is reported. This can inflate the headline SOTA and average gIoU gains.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing risk. Section 3.2 says Ground-V uses COCO 2017 images; Section 4.1 evaluates on RefCOCO/+/g and gRefCOCO built from COCO 2014. COCO 2017 train is largely a re-partition of COCO 2014, so without deduplication the same images appear in training and test. Since Ground-V is a mask-supervision dataset, image overlap lets the model memorize scene layout and object locations; even novel referring expressions are then easier. The supplement's human annotation and the Table 7 rephrasing check address label quality and language overfitting, not image leakage. The internal Ground-V test set (COCO 2017 val) and D3 results give some independent signal, but they do not support the specific SOTA claims on RefCOCO/+/g and gRefCOCO, which are central to the abstract. I therefore agree with the REJECT verdict; no adjustment is needed.","tokens_in":21090,"tokens_out":9624,"duration_ms":85785,"concrete_test":"Compute image-level overlap between the 50,000 COCO 2017 Ground-V training images and the evaluation image sets of RefCOCO/RefCOCO+/RefCOCOg (UNC/UMD splits) and gRefCOCO, using COCO image IDs. Retrain LISA-G5 and PSALM-G5 with all overlapping images removed from Ground-V, keeping hyperparameters fixed, and re-report Tables 3 and 4. If the gains over the corresponding baselines shrink materially, the headline numbers are contamination artifacts; if they persist, the contamination concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that adding Ground-V during training yields large gains on RefCOCO/+/g, gRefCOCO, and D3. Section 3.2 states 'We use COCO 2017 train and validation split as our image source,' while Section 4.1 evaluates on RefCOCO/+/g and gRefCOCO, which are built on COCO 2014 images. COCO 2017 train contains most of COCO 2014 val (and much of 2014 train), so the same photographs can easily appear in both Ground-V training and the evaluation splits. The paper and supplement never describe image-level deduplication against these benchmarks. This matters because Ground-V training provides mask supervision for the exact scenes that later appear at test time: even with different referring expressions, the model can exploit familiarity with object layout and masks, inflating cIoU/gIoU/N-Acc. Since the headline SOTA (Table 3, Table 4) and the 4.4%/7.9% boosts depend on these benchmarks, the central claim is not supported as presented. The internal Ground-V test set and D3 results are less affected but do not rescue the benchmark claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Ground-V, a dataset of roughly 481K automatically generated instruction-segmentation pairs covering five challenges: multi-granularity, multi-object scenes, hallucinated references, part-level references, and reasoning-driven segmentation. The data is produced by prompting Claude 3 Sonnet to link natural language instructions to existing COCO annotations, with a smaller human-annotated test partition. The authors train two existing VLM segmentation models, LISA and PSALM, with Ground-V added to their standard training data, and report consistent gains on RefCOCO/+/g, gRefCOCO, D3, and the internal Ground-V test set, including new state-of-the-art results on gRefCOCO. The central claim is that this data-centric workflow substantially improves pixel-level grounding under complex instructions, with average gIoU gains of 4.4% for LISA and 7.9% for PSALM across six benchmarks.","tokens_in":21274,"tokens_out":7084,"duration_ms":63546,"significance":"If the empirical results are valid, the paper makes a useful data-centric contribution: the Ground-V dataset expands beyond simple referring expressions, and the evaluation across two model families with human-annotated test data is a strength. The authors also include a rephrasing robustness check and a general-VQA transfer experiment, which are thoughtful additions. However, the significance is contingent on resolving a serious train/eval image-overlap risk between the Ground-V training images (COCO 2017 train+val) and the public benchmark test images (built from COCO 2014), which the paper does not address. The internal test set also shares its generation pipeline with the training set, raising a circularity concern that is only partially mitigated. Because the headline SOTA and improvement claims depend on these evaluations, the current version does not fully support its central conclusions.","major_comments":[{"comment":"The paper states in Section 3.2 that Ground-V uses COCO 2017 train and validation splits as its image source, while the public benchmarks evaluated in Section 4.1 (RefCOCO/+/g and gRefCOCO) are built from COCO 2014 images. Since COCO 2017 train contains most COCO 2014 images, including much of the COCO 2014 validation set, the same photographs can appear in both Ground-V training and the benchmark test sets. The paper never reports any image-level deduplication against RefCOCO/+/g, gRefCOCO, or D3. With mask supervision on the exact test images, a model could exploit familiarity with object layouts and masks, inflating the cIoU/gIoU/N-Acc numbers in Tables 3, 4, and 6. Please provide a concrete overlap analysis (e.g., image hash or file-name comparison) and, if any overlap exists, retrain on a deduplicated Ground-V training set and re-report all benchmark results. This is necessary to support the headline claims of 4.4%/7.9% average improvements and state-of-the-art results.","section":"Section 3.2 and Section 4.1"},{"comment":"The Ground-V test set is generated using the same Claude-based workflow as the training set, so large gains on that test set (Table 5) may partly reflect the model learning the teacher's instruction style and answer distribution rather than independent capability. The rephrasing experiment in Table 7, which is meant to address this, covers only the reasoning subset; the multi-granular, multi-object, hallucination, and part-reference subsets are not tested. Please extend the rephrasing robustness check to all five subsets, or provide a separate externally written evaluation set of comparable complexity, so the internal test results can be interpreted as evidence of general task ability rather than style matching.","section":"Section 3.2 and Section 4.2, Table 7"},{"comment":"The scaling analysis in Figure 5 demonstrates that increasing the amount of Ground-V data improves performance on the Ground-V test set, but it does not isolate whether the gains are due to the content of the five addressed challenges or simply to adding a large volume of segmentation pairs. Please include a control experiment that trains on an equivalent volume of generic referring-expression data (for example, a randomly selected set of existing RefCOCO/RefCOCO+ training pairs of the same size) and report the resulting gains on the same benchmarks. Without such a control, the attribution of the improvements to the complex-instruction workflow is not fully supported.","section":"Section 4.2, Figure 5"}],"minor_comments":[{"comment":"The abstract and introduction describe Ground-V as a 500K-pair dataset, while Table 1 reports 481K pairs and Table 2 totals 481,406 instruction pairs. Please reconcile these numbers by stating the exact count and explaining the rounding.","section":"Abstract and Tables 1-2"},{"comment":"The row for “PSALM (ft)” in Table 4 appears to have a formatting error: the first three entries read “-69.378.3 -”, which is ambiguous. Please clarify the intended values (or missing entries) for gIoU, cIoU, and N-Acc on the validation split.","section":"Table 4"},{"comment":"The paper states that the training set contains 50,000 images and the test set contains 5,000 images, but it does not explicitly describe how the test images are guaranteed to be disjoint from the training images. The Table 5 caption calls the test set “disjoint”; please specify the selection protocol that ensures no image-level overlap between the two partitions.","section":"Section 3.2 and Table 5"},{"comment":"Footnote 4 contains the typo “incorprated”; it should be “incorporated.”","section":"Section 4.2, footnote 4"},{"comment":"The rephrasing experiment uses Qwen2-VL to paraphrase instructions, but no details are given about how the paraphrases were validated to preserve the original meaning and expected segmentation targets. Please describe the validation process for the rephrased instructions.","section":"Section 4.2, Table 7"},{"comment":"The paper does not state whether the Ground-V dataset and any associated code will be made publicly available. A data and code availability statement would be important for reproducibility and for the community to use the dataset.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The train/eval image-overlap concern is the central issue. The authors must report whether any Ground-V training images coincide with the test images of RefCOCO/+/g or gRefCOCO; if they do, the benchmark tables must be regenerated after removing those images from training. I do not recommend outright rejection because this is addressable in a revision, but the current headline results are not reliable without this analysis. The internal test-set circularity is a second concern that also needs a concrete response. If, after a revision, the authors cannot demonstrate disjointness, the benchmark claims should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe short version: Ground-V is a large, thoughtfully assembled dataset for VLM-based grounding, and the paper's internal evaluation is fairly careful, but the headline gains on RefCOCO/+/g and gRefCOCO are not trustworthy as reported. The likely cause is that Ground-V was built from COCO 2017 images while those benchmarks use COCO 2014 images, and COCO 2017 train contains most of COCO 2014; the paper never reports deduplicating training images against evaluation splits. That is exactly the kind of overlap that can inflate segmentation numbers, since the model gets mask supervision on the very photographs it is later tested on.\n\nWhat is genuinely good: the five-challenge taxonomy is a useful organizing frame, and the 500K-pair dataset is a real scaling effort that goes beyond earlier works like MUSE. The human-annotated test set with two rounds of validation is a strong point, and the ablations (data scaling, subset removal) are informative. The rephrasing experiment on the reasoning subset is a nice check that the model is not just memorizing language templates. On the internal Ground-V test set, which is image-disjoint, the gains are substantial, and even if those numbers partly reflect distribution matching, they show the dataset is teaching something.\n\nThe soft spots, in order: (1) the benchmark contamination, which is load-bearing; (2) the lack of a data-scale control, so we don't know if a similar volume of generic referring pairs would yield the same gains; (3) no data/code release, making the claims hard to verify.\n\nThe internal test set and D3 results are less affected by the concern, but they don't rescue the benchmark claims. If the authors can show that their 50K image sample happens to avoid all RefCOCO/gRefCOCO test images, the story would be different; as is, the burden is on them.\n\nWho is this for: researchers building instruction-following data for segmentation, and anyone who should be reminded how easily image-level contamination sneaks into benchmark evaluation. I think the paper deserves a serious referee — the dataset is potentially valuable and the flaw is fixable — but the authors should be required to re-run their experiments on deduplicated splits and release the data with explicit overlap checks.\n\nBest.","headline":"Large, well-curated grounding dataset, but the headline SOTA results rest on likely train/eval image overlap between Ground-V (COCO 2017) and RefCOCO/gRefCOCO (COCO 2014) — the paper needs deduplication and re-evaluation before the benchmark claims can be trusted.","tokens_in":21889,"tokens_out":4961,"would_cite":false,"duration_ms":42405,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training with Ground-V, a 500K-pair dataset of automatically generated instruction-segmentation examples spanning five challenge dimensions, substantially improves pixel-level grounding of vision-language models…","keywords":["visual grounding","referring expression segmentation","instruction following","dataset generation","vision-language models","hallucination mitigation","reasoning segmentation","multi-granularity"],"falsifier":"Compute the image-ID intersection between the Ground-V training set (together with the COCO-based data it augments) and the test and validation splits of RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO; then retrain the same LISA and PSALM configurations on the deduplicated version of Ground-V and compare the gIoU and N-Acc numbers with the reported ones. If the gains shrink substantially after removing overlapping images, the headline improvements are at least partly due to training/test image leakage.","tokens_in":20833,"feed_emoji":"🎯","tokens_out":6703,"duration_ms":56211,"temperature":0.7,"pith_summary":"The paper claims that a single large, automatically generated dataset can teach vision-language models to turn complex text instructions into accurate pixel-level segmentations. The dataset, Ground-V, contains roughly 500K instruction-segmentation pairs built by prompting a teacher model with few-shot examples that cover five real-world difficulties: multi-object references, multi-granular naming, hallucinated or absent objects, part-level queries, and reasoning. Training existing segmentation VLMs (LISA and PSALM) on this data improves their average generalized IoU by 4.4% and 7.9% across six benchmarks and yields state-of-the-art results on RefCOCO/+/g and gRefCOCO, including an 83.3% N-Acc on gRefCOCO that beats the prior state of the art by more than 20 points. If the result holds, it gives practitioners a low-annotation pathway to substantially better instruction-following segmentation.","feed_headline":"Training on Ground-V lifts VLM grounding up to 7.9%","feed_subtitle":"Auto-generated instruction-segmentation pairs push LISA and PSALM to state-of-the-art on six benchmarks.","key_machinery":"The machinery is the Ground-V data-generation pipeline. For each of five challenge dimensions, the authors hand-craft a three-shot prompt (image, instruction, answer) and query a teacher vision-language model to generate new instruction-response pairs that are then linked to existing pixel-level annotations, mainly from COCO 2017 and the PACO part annotations, preserving the original segmentation masks as supervision. Hallucination examples are deliberately assigned empty masks and text-only explanations so a correct model should abstain from producing a mask. The evaluation set is generated with the same workflow but then filtered through a judge model and two independent human annotators, retaining only pairs both annotators confirm. This pipeline is what carries the argument: it converts abundant existing masks into complex, nuanced instruction data at scale with minimal human cost.","core_discovery":"On the paper's own terms, the central discovery is that scaling instruction-following data with a teacher-driven generation workflow elicits pixel-level grounding that existing training sets fail to provide. Concretely, the authors claim that augmenting the standard training mixtures of LISA and PSALM with Ground-V immediately improves grounding accuracy across six benchmarks, and that the resulting models surpass previous state of the art on RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO without requiring architectural changes. The most striking single number is a gRefCOCO N-Acc of 83.3%, exceeding the previous state of the art by more than 20 percentage points, which the paper attributes to the hallucination-mitigation portion of Ground-V.","pith_inferences":["The caption-level overlap between COCO 2017 training images and COCO 2014 benchmark images is unresolved, so the headline gains should be treated as an upper bound until the authors report deduplication statistics.","Because the hallucination subset is text-only, it could be mixed into any visual instruction-tuning pipeline; the paper tests only one generalist VLM, but the abstention behavior may transfer to other models with similar tuning.","The ablation finding that hallucination training slightly reduces fine-grained recall suggests a tunable trade-off between abstention and sensitivity, so application-specific mixes of the five subsets could yield a family of models.","The zero-shot gain on MUSE hints that the multi-object and reasoning subsets encode transferable skills; testing PSALM-G5 on additional unseen referring benchmarks would clarify how far the dataset generalizes."],"forward_implications":["Training on Ground-V directly improves two differently architected segmentation VLMs, LISA and PSALM, with average gIoU gains of 4.4% and 7.9% across six benchmarks.","Ground-V-trained PSALM reaches an N-Acc of 83.3% on gRefCOCO, exceeding the previous state of the art by more than 20 points, indicating substantially better abstention when instructed objects are absent.","The hallucination subset, being text-only, can be dropped into general visual instruction tuning: the reported results on POPE, Vizwiz, MMBench, and SciQA show consistent improvements for a generalist VLM.","The model's performance is robust to rephrasing of the reasoning instructions, suggesting the gains reflect understanding rather than overfitting to one language style.","Data scaling within Ground-V shows monotonic improvement from 25% to 100% of the dataset, supporting the paper's data-centric scaling hypothesis."],"supporting_citations":[{"why":"Defines LISA, the first VLM-segmentation baseline whose training protocol and hyperparameters are reused to test Ground-V.","marker":"[13]"},{"why":"Defines PSALM, the second baseline; Ground-V training on it produces the largest gains and the reported 83.3% gRefCOCO N-Acc.","marker":"[53]"},{"why":"Introduces gRefCOCO and the gIoU and N-Acc metrics used for multi-object and no-target grounding evaluation.","marker":"[19]"},{"why":"Defines RefCOCO+, one of the standard referring-expression-segmentation benchmarks used in the evaluation.","marker":"[29]"},{"why":"Defines RefCOCO and RefCOCOg, the other two standard benchmarks against which the models are compared.","marker":"[50]"},{"why":"PACO supplies the part-level annotations that the part-reference subset of Ground-V rephrases into natural language instructions.","marker":"[34]"},{"why":"COCO is the image, segmentation-mask, and bounding-box source for the Ground-V training and evaluation sets.","marker":"[18]"},{"why":"D3 provides the described-object-segmentation benchmark used to test transfer of Ground-V training to unseen language.","marker":"[45]"}],"fun_headline_variants":["Ground-V: 83.3% gRefCOCO, +7.9% accuracy","Auto-data pipeline lifts VLM grounding to SOTA","Teacher-generated data grounds VLMs in pixels","Data scaling without human labels boosts VLM grounding","Ground-V: new SOTA on pixel grounding tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation results assume that the images used to build Ground-V (COCO 2017 train and val) are disjoint from the images used in the RefCOCO/+/g and gRefCOCO evaluation splits, and the paper does not report any deduplication between them; if those photographs overlap, the reported gains could partly reflect test images already seen during training.","fun_headline_variants_meta":{"raw":{"variants":["Ground-V: 83.3% gRefCOCO, +7.9% accuracy","Auto-data pipeline lifts VLM grounding to SOTA","Teacher-generated data grounds VLMs in pixels","Data scaling without human labels boosts VLM grounding","Ground-V: new SOTA on pixel grounding tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1498,"prompt_tokens":903,"completion_tokens":595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":513}},"tokens_in":519,"tokens_out":595,"duration_ms":5467,"temperature":1.0,"reasoning_tokens":513,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:10:58.001929+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the image-ID intersection between the Ground-V training set (together with the COCO-based data it augments) and the test and validation splits of RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO; then retrain the same LISA and PSALM configurations on the deduplicated version of Ground-V and compare the gIoU and N-Acc numbers with the reported ones. If the gains shrink substantially after removing overlapping images, the headline improvements are at least partly due to training/test image leakage.","supporting_citations":[{"cited_title":"Lisa: Reasoning segmentation via large language model","cited_arxiv_id":null,"evidence_quote":"Defines LISA, the first VLM-segmentation baseline whose training protocol and hyperparameters are reused to test Ground-V."},{"cited_title":"Psalm: Pixelwise segmentation with large multi-modal model.ECCV, 2024","cited_arxiv_id":null,"evidence_quote":"Defines PSALM, the second baseline; Ground-V training on it produces the largest gains and the reported 83.3% gRefCOCO N-Acc."},{"cited_title":"Gres: Gen- eralized referring expression segmentation","cited_arxiv_id":null,"evidence_quote":"Introduces gRefCOCO and the gIoU and N-Acc metrics used for multi-object and no-target grounding evaluation."},{"cited_title":"Mod- eling context between objects for referring expression under- standing","cited_arxiv_id":null,"evidence_quote":"Defines RefCOCO+, one of the standard referring-expression-segmentation benchmarks used in the evaluation."},{"cited_title":"Modeling context in referring ex- pressions","cited_arxiv_id":null,"evidence_quote":"Defines RefCOCO and RefCOCOg, the other two standard benchmarks against which the models are compared."},{"cited_title":"Paco: Parts and attributes of common objects","cited_arxiv_id":null,"evidence_quote":"PACO supplies the part-level annotations that the part-reference subset of Ground-V rephrases into natural language instructions."},{"cited_title":"Microsoft coco: Common objects in context","cited_arxiv_id":null,"evidence_quote":"COCO is the image, segmentation-mask, and bounding-box source for the Ground-V training and evaluation sets."},{"cited_title":"Described object detection: Liberating ob- ject detection with flexible expressions.Advances in Neural Information Processing Systems, 36, 2023","cited_arxiv_id":null,"evidence_quote":"D3 provides the described-object-segmentation benchmark used to test transfer of Ground-V training to unseen language."}],"review_version":1}